RapidMiner

Heating a Vermont house with Pi and Pellets - Part 2
Community Manager
‎09-01-2017 09:36 AM
304 Views
1 Comment

 

Greetings Community,

 

It is a crisp morning here in Vermont - 44°F / 7°C - so my quest to get optimal heating efficiency has a renewed sense of urgency.  Snow in September is not unheard of around here.

 

I am still in the data collection phase of this project, although some data are already coming in nicely for analysis (see later on in this post).  Bought several more sensors that I plan on adding to this project in addition to room temp and infrared flame...

Read more...

Greetings Community,

 

It is a crisp morning here in Vermont - 44°F / 7°C - so my quest to get optimal heating efficiency has a renewed sense of urgency.  Snow in September is not unheard of around here.

 

I am still in the data collection phase of this project, although some data are already coming in nicely for analysis (see later on in this post).  Bought several more sensors that I plan on adding to this project in addition to room temp and infrared flame:

- 20A A/C current sensor (a Hall effect transistor IC chip) for measuring total current into stove (and of course easily converted to kW later on...)

- 433MHz superheterodyne receiver (this one) for getting the outside temperature measurements from my Acu-Rite Temperature Sensor hanging on a tree outside, inspired by this clever post.

- A bunch of A/D converter chips to convert some of my analog signals to the Pi GPIO ports

- A bunch of 5A current transformers to monitor the current of individual power-consuming components (later converted to kW) on the stove: the convection blower, the ignitor, the combustion blower, and the auger motor

- A USB microphone to record audio of the various actions of the stove as a verification of readings elsewhere

 

I had to move my flame sensor MUCH closer to the stove in order to get more reliable data; the IR spectrum of this cheap "flame sensor" is picking up wavelengths from sunlight as well as flame during the daytime.  Once I did that, the Pi was recording great data:

overnight data collection (x-axis unit ≈ 5 sec)overnight data collection (x-axis unit ≈ 5 sec)So I am immediately thinking this is time to master the RapidMiner Time Series Extension! I have played around with this before but never done FFTs or other wave analysis with it.  Last time I did an FFT was in college and I did it long-hand with an HP 42S RPN handheld calculator.  Good times.  Anyone want to give this a crack?  I am attaching the data to this post and will give you kudos in my next post if you can show me!

 

Lastly, I want to join these data with those gathered by the U.S. National Weather Service via their REST API.  They have stations everywhere and all their data are easily accessible for free.  It took some doing to decipher what my "gridpoint" was and to deal with the JSON mess I received, but it was well worth it:

Screen Shot 2017-09-01 at 9.27.34 AM.png

Here's my XML for anyone who wants to use this API for retrieving weather data in the U.S.:

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.6.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.6.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="generate_data_user_specification" compatibility="7.6.001" expanded="true" height="68" name="Generate Data by User Specification" width="90" x="45" y="34">
        <list key="attribute_values"/>
        <list key="set_additional_roles"/>
      </operator>
      <operator activated="true" class="web:enrich_data_by_webservice" compatibility="7.3.000" expanded="true" height="68" name="Enrich Data by Webservice" width="90" x="179" y="34">
        <parameter key="query_type" value="Regular Expression"/>
        <list key="string_machting_queries"/>
        <list key="regular_expression_queries">
          <parameter key="foo" value=".*"/>
        </list>
        <list key="regular_region_queries"/>
        <list key="xpath_queries"/>
        <list key="namespaces"/>
        <list key="index_queries"/>
        <list key="jsonpath_queries"/>
        <parameter key="url" value="https://api.weather.gov/gridpoints/BTV/124,29"/>
        <list key="request_properties"/>
      </operator>
      <operator activated="true" class="subprocess" compatibility="7.6.001" expanded="true" height="82" name="Subprocess (3)" width="90" x="313" y="34">
        <process expanded="true">
          <operator activated="true" class="nominal_to_text" compatibility="7.6.001" expanded="true" height="82" name="Nominal to Text" width="90" x="45" y="34">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="foo"/>
          </operator>
          <operator activated="true" class="text:data_to_documents" compatibility="7.5.000" expanded="true" height="68" name="Data to Documents" width="90" x="179" y="34">
            <parameter key="select_attributes_and_weights" value="true"/>
            <list key="specify_weights">
              <parameter key="foo" value="1.0"/>
            </list>
          </operator>
          <operator activated="true" class="text:combine_documents" compatibility="7.5.000" expanded="true" height="82" name="Combine Documents" width="90" x="313" y="34"/>
          <operator activated="true" class="text:json_to_data" compatibility="7.5.000" expanded="true" height="82" name="JSON To Data" width="90" x="447" y="34"/>
          <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (2)" width="90" x="581" y="34">
            <parameter key="attribute_filter_type" value="regular_expression"/>
            <parameter key="regular_expression" value=".*values\[[0-9]+\].*"/>
          </operator>
          <operator activated="true" class="transpose" compatibility="7.6.001" expanded="true" height="82" name="Transpose" width="90" x="715" y="34"/>
          <operator activated="true" class="replace" compatibility="7.6.001" expanded="true" height="82" name="Replace" width="90" x="849" y="34">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="id"/>
            <parameter key="include_special_attributes" value="true"/>
            <parameter key="replace_what" value="properties[.]|values|valid|apparent"/>
          </operator>
          <operator activated="true" class="split" compatibility="7.6.001" expanded="true" height="82" name="Split" width="90" x="983" y="34">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="id"/>
            <parameter key="include_special_attributes" value="true"/>
            <parameter key="split_pattern" value="[.]+"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (3)" width="90" x="1117" y="34">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="id_3"/>
            <parameter key="regular_expression" value=".*values\[[0-9]+\].*"/>
            <parameter key="invert_selection" value="true"/>
          </operator>
          <operator activated="true" class="trim" compatibility="7.6.001" expanded="true" height="82" name="Trim" width="90" x="1251" y="34"/>
          <operator activated="true" class="generate_attributes" compatibility="7.6.001" expanded="true" height="82" name="Generate Attributes (2)" width="90" x="1385" y="34">
            <list key="function_descriptions">
              <parameter key="type" value="if(contains(att_1,&quot;+&quot;),&quot;datetime&quot;,&quot;value&quot;)"/>
            </list>
          </operator>
          <operator activated="true" class="concurrency:loop_values" compatibility="7.6.001" expanded="true" height="82" name="Loop Values" width="90" x="1519" y="34">
            <parameter key="attribute" value="id_1"/>
            <parameter key="enable_parallel_execution" value="false"/>
            <process expanded="true">
              <operator activated="false" class="de_pivot" compatibility="7.6.001" expanded="true" height="82" name="De-Pivot" width="90" x="45" y="187">
                <list key="attribute_name">
                  <parameter key="datetime" value="sss"/>
                </list>
              </operator>
              <operator activated="true" class="filter_examples" compatibility="7.6.001" expanded="true" height="103" name="Filter Examples (2)" width="90" x="45" y="34">
                <list key="filters_list">
                  <parameter key="filters_entry_key" value="id_1.equals.%{loop_value}"/>
                </list>
              </operator>
              <operator activated="true" class="extract_macro" compatibility="7.6.001" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="136">
                <parameter key="macro" value="value"/>
                <parameter key="macro_type" value="data_value"/>
                <parameter key="attribute_name" value="id_1"/>
                <parameter key="example_index" value="1"/>
                <list key="additional_macros"/>
              </operator>
              <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (4)" width="90" x="179" y="34">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="id_4"/>
                <parameter key="invert_selection" value="true"/>
              </operator>
              <operator activated="true" class="pivot" compatibility="7.6.001" expanded="true" height="82" name="Pivot" width="90" x="313" y="34">
                <parameter key="group_attribute" value="id_2"/>
                <parameter key="index_attribute" value="type"/>
                <parameter key="consider_weights" value="false"/>
                <parameter key="skip_constant_attributes" value="false"/>
              </operator>
              <operator activated="true" class="rename" compatibility="7.6.001" expanded="true" height="82" name="Rename" width="90" x="447" y="34">
                <parameter key="old_name" value="att_1_value"/>
                <parameter key="new_name" value="value"/>
                <list key="rename_additional_attributes">
                  <parameter key="att_1_datetime" value="datetime"/>
                  <parameter key="id_1_datetime" value="measurement"/>
                </list>
              </operator>
              <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (5)" width="90" x="581" y="34">
                <parameter key="attribute_filter_type" value="regular_expression"/>
                <parameter key="attribute" value="id_2"/>
                <parameter key="attributes" value="id_1_datetime|id_2"/>
                <parameter key="regular_expression" value="id.*"/>
                <parameter key="invert_selection" value="true"/>
              </operator>
              <operator activated="true" class="replace" compatibility="7.6.001" expanded="true" height="82" name="Replace (2)" width="90" x="715" y="34">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="datetime"/>
                <parameter key="include_special_attributes" value="true"/>
                <parameter key="replace_what" value="T"/>
                <parameter key="replace_by" value=" "/>
              </operator>
              <operator activated="true" class="replace" compatibility="7.6.001" expanded="true" height="82" name="Replace (3)" width="90" x="849" y="34">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="datetime"/>
                <parameter key="include_special_attributes" value="true"/>
                <parameter key="replace_what" value="\/.*"/>
              </operator>
              <operator activated="true" class="replace" compatibility="7.6.001" expanded="true" height="82" name="Replace (4)" width="90" x="983" y="34">
                <parameter key="attribute_filter_type" value="single"/>
                <parameter key="attribute" value="datetime"/>
                <parameter key="include_special_attributes" value="true"/>
                <parameter key="replace_what" value="[+]00[:]00"/>
                <parameter key="replace_by" value=" +0000"/>
              </operator>
              <operator activated="true" class="nominal_to_date" compatibility="7.6.001" expanded="true" height="82" name="Nominal to Date (2)" width="90" x="1117" y="34">
                <parameter key="attribute_name" value="datetime"/>
                <parameter key="date_type" value="date_time"/>
                <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss Z"/>
                <parameter key="time_zone" value="SYSTEM"/>
              </operator>
              <connect from_port="input 1" to_op="Filter Examples (2)" to_port="example set input"/>
              <connect from_op="Filter Examples (2)" from_port="example set output" to_op="Extract Macro" to_port="example set"/>
              <connect from_op="Extract Macro" from_port="example set" to_op="Select Attributes (4)" to_port="example set input"/>
              <connect from_op="Select Attributes (4)" from_port="example set output" to_op="Pivot" to_port="example set input"/>
              <connect from_op="Pivot" from_port="example set output" to_op="Rename" to_port="example set input"/>
              <connect from_op="Rename" from_port="example set output" to_op="Select Attributes (5)" to_port="example set input"/>
              <connect from_op="Select Attributes (5)" from_port="example set output" to_op="Replace (2)" to_port="example set input"/>
              <connect from_op="Replace (2)" from_port="example set output" to_op="Replace (3)" to_port="example set input"/>
              <connect from_op="Replace (3)" from_port="example set output" to_op="Replace (4)" to_port="example set input"/>
              <connect from_op="Replace (4)" from_port="example set output" to_op="Nominal to Date (2)" to_port="example set input"/>
              <connect from_op="Nominal to Date (2)" from_port="example set output" to_port="output 1"/>
              <portSpacing port="source_input 1" spacing="0"/>
              <portSpacing port="source_input 2" spacing="0"/>
              <portSpacing port="sink_output 1" spacing="0"/>
              <portSpacing port="sink_output 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="append" compatibility="7.6.001" expanded="true" height="82" name="Append" width="90" x="1653" y="34"/>
          <operator activated="true" class="pivot" compatibility="7.6.001" expanded="true" height="82" name="Pivot (2)" width="90" x="1787" y="34">
            <parameter key="group_attribute" value="datetime"/>
            <parameter key="index_attribute" value="measurement"/>
            <parameter key="consider_weights" value="false"/>
          </operator>
          <operator activated="true" class="set_role" compatibility="7.6.001" expanded="true" height="82" name="Set Role (2)" width="90" x="1921" y="34">
            <parameter key="attribute_name" value="datetime"/>
            <parameter key="target_role" value="id"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="rename_by_replacing" compatibility="7.6.001" expanded="true" height="82" name="Rename by Replacing" width="90" x="2055" y="34">
            <parameter key="replace_what" value="value_"/>
          </operator>
          <operator activated="true" class="generate_attributes" compatibility="7.6.001" expanded="true" height="82" name="Generate Attributes (3)" width="90" x="2189" y="34">
            <list key="function_descriptions">
              <parameter key="temperatureNWS" value="(9/5)*parse(temperature)+32"/>
            </list>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="7.6.001" expanded="true" height="82" name="Select Attributes (6)" width="90" x="2323" y="34">
            <parameter key="attribute_filter_type" value="subset"/>
            <parameter key="attribute" value="Temperature"/>
            <parameter key="attributes" value="Temperature|temperature"/>
            <parameter key="invert_selection" value="true"/>
          </operator>
          <operator activated="true" class="parse_numbers" compatibility="7.6.001" expanded="true" height="82" name="Parse Numbers" width="90" x="2457" y="34">
            <parameter key="unparsable_value_handling" value="skip attribute"/>
          </operator>
          <connect from_port="in 1" to_op="Nominal to Text" to_port="example set input"/>
          <connect from_op="Nominal to Text" from_port="example set output" to_op="Data to Documents" to_port="example set"/>
          <connect from_op="Data to Documents" from_port="documents" to_op="Combine Documents" to_port="documents 1"/>
          <connect from_op="Combine Documents" from_port="document" to_op="JSON To Data" to_port="documents 1"/>
          <connect from_op="JSON To Data" from_port="example set" to_op="Select Attributes (2)" to_port="example set input"/>
          <connect from_op="Select Attributes (2)" from_port="example set output" to_op="Transpose" to_port="example set input"/>
          <connect from_op="Transpose" from_port="example set output" to_op="Replace" to_port="example set input"/>
          <connect from_op="Replace" from_port="example set output" to_op="Split" to_port="example set input"/>
          <connect from_op="Split" from_port="example set output" to_op="Select Attributes (3)" to_port="example set input"/>
          <connect from_op="Select Attributes (3)" from_port="example set output" to_op="Trim" to_port="example set input"/>
          <connect from_op="Trim" from_port="example set output" to_op="Generate Attributes (2)" to_port="example set input"/>
          <connect from_op="Generate Attributes (2)" from_port="example set output" to_op="Loop Values" to_port="input 1"/>
          <connect from_op="Loop Values" from_port="output 1" to_op="Append" to_port="example set 1"/>
          <connect from_op="Append" from_port="merged set" to_op="Pivot (2)" to_port="example set input"/>
          <connect from_op="Pivot (2)" from_port="example set output" to_op="Set Role (2)" to_port="example set input"/>
          <connect from_op="Set Role (2)" from_port="example set output" to_op="Rename by Replacing" to_port="example set input"/>
          <connect from_op="Rename by Replacing" from_port="example set output" to_op="Generate Attributes (3)" to_port="example set input"/>
          <connect from_op="Generate Attributes (3)" from_port="example set output" to_op="Select Attributes (6)" to_port="example set input"/>
          <connect from_op="Select Attributes (6)" from_port="example set output" to_op="Parse Numbers" to_port="example set input"/>
          <connect from_op="Parse Numbers" from_port="example set output" to_port="out 1"/>
          <portSpacing port="source_in 1" spacing="0"/>
          <portSpacing port="source_in 2" spacing="0"/>
          <portSpacing port="sink_out 1" spacing="0"/>
          <portSpacing port="sink_out 2" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Generate Data by User Specification" from_port="output" to_op="Enrich Data by Webservice" to_port="Example Set"/>
      <connect from_op="Enrich Data by Webservice" from_port="ExampleSet" to_op="Subprocess (3)" to_port="in 1"/>
      <connect from_op="Subprocess (3)" from_port="out 1" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
    </process>
  </operator>
</process>

Note there is no API key/token needed for this service which is an added bonus.

 

That's it for this post.  Stay warm and enjoy!

 

 

 

 

 

 

Scott Genzer
Senior Community Manager
RapidMiner, Inc.
1 Comment
RM Certified Expert

The series extension is nice, but it doesn't actually have everything one might want for analyzing time series data.  In my prior experience we got good results by using the R scripting extension to look at the data with the "forecast" package, which allows for a Holt-Winters model for extracting trend/ moving average and seasonal/periodic variations.  Unfortunately I don't have any code to share but it wasn't that complicated to work with, so it may be something you want to check out!