Time Series & Windowing

MaerkliMaerkli Member Posts: 84 Guru
edited March 2019 in Help
Hallo team, I am lost!
Here my XML file. I suppose that some parameters are not correctly set. Can someone help, please?
In Attachment the Excel with the data. Thanks in advance.
Maerkli

<?xml version="1.0" encoding="UTF-8"?><process version="9.2.000">
  <context>
    <input/>
    <output/>
    <macros>
      <macro>
        <key>futureMonths</key>
        <value>15</value>
      </macro>
      <macro>
        <key>horizon</key>
        <value>1</value>
      </macro>
      <macro>
        <key>windowSize</key>
        <value>6</value>
      </macro>
    </macros>
  </context>
  <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
    <parameter key="logverbosity" value="init"/>
    <parameter key="random_seed" value="2001"/>
    <parameter key="send_mail" value="never"/>
    <parameter key="notification_email" value=""/>
    <parameter key="process_duration_for_mail" value="30"/>
    <parameter key="encoding" value="SYSTEM"/>
    <process expanded="true">
      <operator activated="true" class="retrieve" compatibility="9.2.000" expanded="true" height="68" name="Retrieve 10_TimeS_10.2.1_simpleTS" width="90" x="45" y="34">
        <parameter key="repository_entry" value="10_TimeS_10.2.1_simpleTS"/>
      </operator>
      <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role" width="90" x="179" y="30">
        <parameter key="attribute_name" value="Normalized Profits"/>
        <parameter key="target_role" value="label"/>
        <list key="set_additional_roles">
          <parameter key="Normalized Profits" value="label"/>
        </list>
      </operator>
      <operator activated="true" class="select_attributes" compatibility="9.2.000" expanded="true" height="82" name="Select Attributes" width="90" x="179" y="136">
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="inputYt|Normalized Profits|Date"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="attribute_value"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="time"/>
        <parameter key="block_type" value="attribute_block"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_matrix_row_start"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="false"/>
      </operator>
      <operator activated="true" class="filter_examples" compatibility="6.4.000" expanded="true" height="103" name="Filter Examples" width="90" x="179" y="238">
        <parameter key="parameter_expression" value=""/>
        <parameter key="condition_class" value="no_missing_attributes"/>
        <parameter key="invert_filter" value="false"/>
        <list key="filters_list"/>
        <parameter key="filters_logic_and" value="true"/>
        <parameter key="filters_check_metadata" value="true"/>
      </operator>
      <operator activated="true" class="series:windowing" compatibility="7.4.000" expanded="true" height="82" name="Windowing for Training" width="90" x="380" y="34">
        <parameter key="series_representation" value="encode_series_by_examples"/>
        <parameter key="window_size" value="%{windowSize}"/>
        <parameter key="step_size" value="1"/>
        <parameter key="create_single_attributes" value="true"/>
        <parameter key="create_label" value="true"/>
        <parameter key="select_label_by_dimension" value="false"/>
        <parameter key="label_attribute" value="Normalized Profits"/>
        <parameter key="horizon" value="%{horizon}"/>
        <parameter key="add_incomplete_windows" value="false"/>
        <parameter key="stop_on_too_small_dataset" value="true"/>
      </operator>
      <operator activated="true" class="series:windowing" compatibility="5.2.000" expanded="true" height="82" name="Windowing for Application" width="90" x="380" y="136">
        <parameter key="series_representation" value="encode_series_by_examples"/>
        <parameter key="window_size" value="%{windowSize}"/>
        <parameter key="step_size" value="1"/>
        <parameter key="create_single_attributes" value="true"/>
        <parameter key="create_label" value="false"/>
        <parameter key="select_label_by_dimension" value="false"/>
        <parameter key="label_attribute" value="inputYt"/>
        <parameter key="horizon" value="1"/>
        <parameter key="add_incomplete_windows" value="false"/>
        <parameter key="stop_on_too_small_dataset" value="true"/>
      </operator>
      <operator activated="false" class="vector_linear_regression" compatibility="9.2.000" expanded="true" height="82" name="Vector Linear Regression" width="90" x="514" y="187">
        <parameter key="use_bias" value="true"/>
        <parameter key="ridge" value="1.0E-8"/>
      </operator>
      <operator activated="true" class="extract_macro" compatibility="9.2.000" expanded="true" height="68" name="Extract Example Count" width="90" x="380" y="238">
        <parameter key="macro" value="exampleCount"/>
        <parameter key="macro_type" value="number_of_examples"/>
        <parameter key="statistics" value="average"/>
        <parameter key="attribute_name" value=""/>
        <list key="additional_macros"/>
        <description align="center" color="yellow" colored="true" width="126">&lt;br&gt; &lt;br&gt; Before we start the looping, we need to store the last forecasted row in&lt;br&gt;a separate data structure&lt;br&gt;This is accomplished by the macro titled Extract&lt;br&gt;Example SetT</description>
      </operator>
      <operator activated="true" class="filter_example_range" compatibility="9.2.000" expanded="true" height="82" name="Filter Example Range" width="90" x="514" y="340">
        <parameter key="first_example" value="%{exampleCount}"/>
        <parameter key="last_example" value="%{exampleCount}"/>
        <parameter key="invert_filter" value="false"/>
      </operator>
      <operator activated="true" class="remember" compatibility="9.2.000" expanded="true" height="68" name="Remember" width="90" x="648" y="340">
        <parameter key="name" value="data"/>
        <parameter key="io_object" value="ExampleSet"/>
        <parameter key="store_which" value="1"/>
        <parameter key="remove_from_process" value="true"/>
      </operator>
      <operator activated="true" class="discretize_by_user_specification" compatibility="9.2.000" expanded="true" height="103" name="Discretize" width="90" x="514" y="34">
        <parameter key="return_preprocessing_model" value="false"/>
        <parameter key="create_view" value="false"/>
        <parameter key="attribute_filter_type" value="subset"/>
        <parameter key="attribute" value=""/>
        <parameter key="attributes" value="label"/>
        <parameter key="regular_expression" value="label"/>
        <parameter key="use_except_expression" value="false"/>
        <parameter key="value_type" value="numeric"/>
        <parameter key="use_value_type_exception" value="false"/>
        <parameter key="except_value_type" value="real"/>
        <parameter key="block_type" value="value_series"/>
        <parameter key="use_block_type_exception" value="false"/>
        <parameter key="except_block_type" value="value_series_end"/>
        <parameter key="invert_selection" value="false"/>
        <parameter key="include_special_attributes" value="true"/>
        <list key="classes">
          <parameter key="first" value="-Infinity"/>
          <parameter key="last" value="Infinity"/>
        </list>
      </operator>
      <operator activated="true" class="classification_by_regression" compatibility="9.2.000" expanded="true" height="82" name="Classification by Regression" width="90" x="648" y="34">
        <process expanded="true">
          <operator activated="true" class="nominal_to_numerical" compatibility="9.2.000" expanded="true" height="103" name="Nominal to Numerical (2)" width="90" x="111" y="59">
            <parameter key="return_preprocessing_model" value="false"/>
            <parameter key="create_view" value="false"/>
            <parameter key="attribute_filter_type" value="all"/>
            <parameter key="attribute" value=""/>
            <parameter key="attributes" value=""/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="nominal"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="file_path"/>
            <parameter key="block_type" value="single_value"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="single_value"/>
            <parameter key="invert_selection" value="false"/>
            <parameter key="include_special_attributes" value="false"/>
            <parameter key="coding_type" value="dummy coding"/>
            <parameter key="use_comparison_groups" value="false"/>
            <list key="comparison_groups"/>
            <parameter key="unexpected_value_handling" value="all 0 and warning"/>
            <parameter key="use_underscore_in_name" value="false"/>
          </operator>
          <operator activated="true" class="vector_linear_regression" compatibility="9.2.000" expanded="true" height="82" name="Vector Linear Regression (2)" width="90" x="313" y="85">
            <parameter key="use_bias" value="true"/>
            <parameter key="ridge" value="1.0E-8"/>
          </operator>
          <connect from_port="training set" to_op="Nominal to Numerical (2)" to_port="example set input"/>
          <connect from_op="Nominal to Numerical (2)" from_port="example set output" to_op="Vector Linear Regression (2)" to_port="training set"/>
          <connect from_op="Vector Linear Regression (2)" from_port="model" to_port="model"/>
          <portSpacing port="source_training set" spacing="0"/>
          <portSpacing port="sink_model" spacing="0"/>
          <description align="center" color="yellow" colored="false" height="105" resized="false" width="180" x="682" y="98">Type your comment</description>
        </process>
      </operator>
      <operator activated="true" class="loop" compatibility="9.2.000" expanded="true" height="82" name="Loop" width="90" x="648" y="136">
        <parameter key="set_iteration_macro" value="true"/>
        <parameter key="macro_name" value="iteration"/>
        <parameter key="macro_start_value" value="1"/>
        <parameter key="iterations" value="%{futureMonths}"/>
        <parameter key="limit_time" value="false"/>
        <parameter key="timeout" value="1"/>
        <process expanded="true">
          <operator activated="true" class="recall" compatibility="9.2.000" expanded="true" height="68" name="Recall" width="90" x="45" y="85">
            <parameter key="name" value="data"/>
            <parameter key="io_object" value="ExampleSet"/>
            <parameter key="remove_from_store" value="true"/>
          </operator>
          <operator activated="true" class="apply_model" compatibility="7.1.001" expanded="true" height="82" name="Apply Model" width="90" x="179" y="34">
            <list key="application_parameters"/>
            <parameter key="create_view" value="false"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="9.2.000" expanded="true" height="103" name="Multiply" width="90" x="447" y="34"/>
          <operator activated="true" class="materialize_data" compatibility="9.2.000" expanded="true" height="82" name="Materialize Data (2)" width="90" x="112" y="187">
            <parameter key="datamanagement" value="double_array"/>
            <parameter key="data_management" value="auto"/>
          </operator>
          <operator activated="true" class="generate_attributes" compatibility="9.2.000" expanded="true" height="82" name="Increase Date (2)" width="90" x="313" y="165">
            <list key="function_descriptions">
              <parameter key="Date" value="date_add(Date, 1, DATE_UNIT_MONTH)"/>
            </list>
            <parameter key="keep_all" value="true"/>
          </operator>
          <operator activated="true" class="set_role" compatibility="5.3.013" expanded="true" height="82" name="Set Role (2)" width="90" x="581" y="136">
            <parameter key="attribute_name" value="prediction(label)"/>
            <parameter key="target_role" value="label"/>
            <list key="set_additional_roles"/>
          </operator>
          <operator activated="true" class="select_attributes" compatibility="9.2.000" expanded="true" height="82" name="Select Attributes (3)" width="90" x="179" y="391">
            <parameter key="attribute_filter_type" value="single"/>
            <parameter key="attribute" value="inputYt-5"/>
            <parameter key="attributes" value=""/>
            <parameter key="use_except_expression" value="false"/>
            <parameter key="value_type" value="attribute_value"/>
            <parameter key="use_value_type_exception" value="false"/>
            <parameter key="except_value_type" value="time"/>
            <parameter key="block_type" value="attribute_block"/>
            <parameter key="use_block_type_exception" value="false"/>
            <parameter key="except_block_type" value="value_matrix_row_start"/>
            <parameter key="invert_selection" value="true"/>
            <parameter key="include_special_attributes" value="false"/>
          </operator>
          <operator activated="true" class="rename" compatibility="9.2.000" expanded="true" height="82" name="Rename" width="90" x="313" y="391">
            <parameter key="old_name" value="inputYt-4"/>
            <parameter key="new_name" value="inputYt-5"/>
            <list key="rename_additional_attributes">
              <parameter key="inputYt-3" value="inputYt-4"/>
              <parameter key="inputYt-2" value="inputYt-3"/>
              <parameter key="inputYt-1" value="inputYt-2"/>
              <parameter key="inputYt-0" value="inputYt-1"/>
              <parameter key="prediction(label)" value="inputYt-0"/>
            </list>
          </operator>
          <operator activated="true" class="remember" compatibility="9.2.000" expanded="true" height="68" name="Remember (2)" width="90" x="514" y="391">
            <parameter key="name" value="data"/>
            <parameter key="io_object" value="ExampleSet"/>
            <parameter key="store_which" value="1"/>
            <parameter key="remove_from_process" value="true"/>
          </operator>
          <connect from_port="input 1" to_op="Apply Model" to_port="model"/>
          <connect from_op="Recall" from_port="result" to_op="Apply Model" to_port="unlabelled data"/>
          <connect from_op="Apply Model" from_port="labelled data" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_port="output 1"/>
          <connect from_op="Multiply" from_port="output 2" to_op="Materialize Data (2)" to_port="example set input"/>
          <connect from_op="Materialize Data (2)" from_port="example set output" to_op="Increase Date (2)" to_port="example set input"/>
          <connect from_op="Increase Date (2)" from_port="example set output" to_op="Set Role (2)" to_port="example set input"/>
          <connect from_op="Set Role (2)" from_port="example set output" to_op="Select Attributes (3)" to_port="example set input"/>
          <connect from_op="Select Attributes (3)" from_port="example set output" to_op="Rename" to_port="example set input"/>
          <connect from_op="Rename" from_port="example set output" to_op="Remember (2)" to_port="store"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_output 1" spacing="0"/>
          <portSpacing port="sink_output 2" spacing="0"/>
          <description align="center" color="yellow" colored="false" height="105" resized="false" width="180" x="673" y="462">Inner level loop process for generating forecasts.</description>
          <description align="center" color="yellow" colored="false" height="105" resized="false" width="180" x="83" y="279">This operator creates a fresh and clean copy of the data in the memory.</description>
          <description align="center" color="yellow" colored="false" height="105" resized="false" width="180" x="289" y="273">Generate Attributes.</description>
        </process>
      </operator>
      <operator activated="true" class="append" compatibility="9.2.000" expanded="true" height="82" name="Append" width="90" x="782" y="85">
        <parameter key="datamanagement" value="double_array"/>
        <parameter key="data_management" value="auto"/>
        <parameter key="merge_type" value="all"/>
      </operator>
      <connect from_op="Retrieve 10_TimeS_10.2.1_simpleTS" from_port="output" to_op="Set Role" to_port="example set input"/>
      <connect from_op="Set Role" from_port="example set output" to_op="Select Attributes" to_port="example set input"/>
      <connect from_op="Select Attributes" from_port="example set output" to_op="Filter Examples" to_port="example set input"/>
      <connect from_op="Filter Examples" from_port="example set output" to_op="Windowing for Training" to_port="example set input"/>
      <connect from_op="Windowing for Training" from_port="example set output" to_op="Discretize" to_port="example set input"/>
      <connect from_op="Windowing for Training" from_port="original" to_op="Windowing for Application" to_port="example set input"/>
      <connect from_op="Windowing for Application" from_port="example set output" to_op="Extract Example Count" to_port="example set"/>
      <connect from_op="Extract Example Count" from_port="example set" to_op="Filter Example Range" to_port="example set input"/>
      <connect from_op="Filter Example Range" from_port="example set output" to_op="Remember" to_port="store"/>
      <connect from_op="Discretize" from_port="example set output" to_op="Classification by Regression" to_port="training set"/>
      <connect from_op="Classification by Regression" from_port="model" to_op="Loop" to_port="input 1"/>
      <connect from_op="Loop" from_port="output 1" to_op="Append" to_port="example set 1"/>
      <connect from_op="Append" from_port="merged set" to_port="result 1"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <description align="center" color="yellow" colored="false" height="121" resized="false" width="180" x="648" y="419">Finally the Remember operator&lt;br&gt;stores this in memory and allows us to &amp;#8220;recall&amp;#8221; the stored value once inside&lt;br&gt;the loop</description>
    </process>
  </operator>
</process>


Tagged:

Best Answers

  • MaerkliMaerkli Member Posts: 84 Guru
    Solution Accepted
    Rebonjour Hughes,
    Yes, it works! Please, explain me the correction you made. I tried before your proposed correction but issue was not solved.
    Maerkli

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    Solution Accepted
    Hi Maerkil,

    I made two corrections to your process. One was to fix the 'date increase' operator by using Date-[0] rather than Date. This is due to the shift of the windowing operator. The second correction was to the order of the main part of the process in front of the loop. If you use show order on the fixed process you will see the change.

    regards,

    Alex

Answers

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    edited February 2019
    Hi @Maerkli,

    The error is inside your loop. You have a windowing operator creating Date-0, Date-1 etc.  You are then generating an attribute which references Date and not your windowed Date. Date is not in your example set. Look at your 'Increase Date(2)' operator. 

    Your rename operator inside the loop is also looking for an attribute that is not inside your example set.

    regards,

    Alex
  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    edited February 2019
    If you take out the rename operator in your loop, your process runs. Does this help? This just fixes some things I noticed although I would have to look at your processes more closely to see if it really fixes everything.


  • MaerkliMaerkli Member Posts: 84 Guru
    Bonjour Hughes,
    Thanks for your rapid answer. Issue shows up inside the inner Loop: "Attributes do not match" with Operator Apply Model, even with your proposed correction. "You are then generating an attribute which references Date and not your windowed Date"  could a good path to follow.
    Maerkli



  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    edited February 2019
    @Maerkil, let me know if you got it  to work the way you want. If not, I have time tomorrow morning to look at it carefully.
  • MaerkliMaerkli Member Posts: 84 Guru
    Hallo Hughes, Hallo Scott,
    Sorry for my late answer and thanks for the support.
    I have tried your suggestion.  At Increase Date operator, I still have the warning "Expresssion Evaluation Error". "The variable Date is unknown."
    Maerkli
  • MaerkliMaerkli Member Posts: 84 Guru
    When you fly to Europe, Hughes, we are going to have a beer together!
  • MaerkliMaerkli Member Posts: 84 Guru
    Alex, Something is strange! I run your process right now with th same dataset and I get the message at Operator Apply inside the Loop: " Attributes do not match. The Input ExampleSet does not match the training ExampleSet. Missing Attribute: Date-5=1/2/2009". Although it worked some hours ago.
  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    edited March 2019
    Try this version. I noticed the same thing once and then it started to work. Let me know if this fixes it. Both this and the first are working for me. What is happening is that the recall inside the loop tries to read the remember inside the loop before it has been stored. I am trying to reproduce your error but can't at the moment.

  • MaerkliMaerkli Member Posts: 84 Guru
    Encore merci. I will try as soon as possible your new version. Maerkli
  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    Everything is working for me here. PM me if there is still a problem.
Sign In or Register to comment.