Does Windowing operator accept a macro for "window size"?

vgpowered

Hello,
Is it possible to use a macro in the Windowing  for "window size" parameter?

I have read the thread "Forecasting process - doing it right?" and it seems the operator can do, but I have the error "Required parameter missing"

What happens?

ceaperez

Hi @vgpowered, 

here is the XML:

<?xml version="1.0" encoding="UTF-8"?><process version="10.3.001">

  <context>

    <input/>

    <output/>

    <macros/>

  </context>

  <operator activated="true" class="process" compatibility="10.3.001" expanded="true" name="Process" origin="GENERATED_TUTORIAL">

    <parameter key="logverbosity" value="init"/>

    <parameter key="random_seed" value="2001"/>

    <parameter key="send_mail" value="never"/>

    <parameter key="notification_email" value=""/>

    <parameter key="process_duration_for_mail" value="30"/>

    <parameter key="encoding" value="SYSTEM"/>

    <process expanded="true">

      <operator activated="true" class="retrieve" compatibility="10.3.001" expanded="true" height="68" name="Retrieve Lake Huron" origin="GENERATED_TUTORIAL" width="90" x="45" y="34">

        <parameter key="repository_entry" value="//Samples/Time Series/data sets/Lake Huron"/>

      </operator>

      <operator activated="true" class="concurrency:optimize_parameters_grid" compatibility="10.3.001" expanded="true" height="124" name="Optimize Parameters (Grid)" width="90" x="246" y="34">

        <list key="parameters">

          <parameter key="Set Macro (Real).value" value="[2;10;5;linear]"/>

        </list>

        <parameter key="error_handling" value="fail on error"/>

        <parameter key="log_performance" value="true"/>

        <parameter key="log_all_criteria" value="false"/>

        <parameter key="synchronize" value="false"/>

        <parameter key="enable_parallel_execution" value="true"/>

        <process expanded="true">

          <operator activated="true" class="operator_toolbox:set_macro_real" compatibility="2.17.000" expanded="true" height="82" name="Set Macro (Real)" width="90" x="45" y="187">

            <parameter key="macro" value="uno"/>

            <parameter key="value" value="3.5"/>

          </operator>

          <operator activated="true" class="generate_macro" compatibility="10.3.001" expanded="true" height="82" name="Generate Macro" width="90" x="179" y="238">

            <list key="function_descriptions">

              <parameter key="dos" value="floor(eval(%{uno}))"/>

            </list>

          </operator>

          <operator activated="true" class="time_series:windowing" compatibility="10.3.001" expanded="true" height="103" name="Windowing (Series)" origin="GENERATED_TUTORIAL" width="90" x="179" y="34">

            <parameter key="attribute_filter_type" value="single"/>

            <parameter key="attribute" value="Lake surface level / feet"/>

            <parameter key="attributes" value=""/>

            <parameter key="use_except_expression" value="false"/>

            <parameter key="value_type" value="numeric"/>

            <parameter key="use_value_type_exception" value="false"/>

            <parameter key="except_value_type" value="real"/>

            <parameter key="block_type" value="value_series"/>

            <parameter key="use_block_type_exception" value="false"/>

            <parameter key="except_block_type" value="value_series_end"/>

            <parameter key="invert_selection" value="false"/>

            <parameter key="include_special_attributes" value="false"/>

            <parameter key="has_indices" value="true"/>

            <parameter key="indices_attribute" value="Date"/>

            <parameter key="sort_time_series" value="false"/>

            <parameter key="expert_settings" value="false"/>

            <parameter key="unit" value="example based"/>

            <parameter key="windows_defined" value="from start"/>

            <parameter key="custom_start_point" value="5"/>

            <parameter key="custom_end_point" value="100"/>

            <parameter key="window_size" value="%{dos}"/>

            <parameter key="custom_start_time" value="2000-01-01 00:00:00"/>

            <parameter key="custom_end_time" value="2030-01-01 00:00:00"/>

            <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>

            <parameter key="window_size_time" value="1.Hours"/>

            <parameter key="windows_stop_definition" value="from next window start"/>

            <parameter key="window_start_attribute" value=""/>

            <parameter key="window_stop_attribute" value=""/>

            <parameter key="no_overlapping_windows" value="false"/>

            <parameter key="step_size" value="1"/>

            <parameter key="step_size_time" value="1.Minutes"/>

            <parameter key="create_horizon_(labels)" value="true"/>

            <parameter key="horizon_attribute" value="Lake surface level / feet"/>

            <parameter key="horizon_size" value="1"/>

            <parameter key="horizon_offset" value="0"/>

            <parameter key="horizon_size_time" value="1.Hours"/>

            <parameter key="horizon_offset_time" value="0.Minutes"/>

            <parameter key="horizon_start_attribute" value=""/>

            <parameter key="horizon_stop_attribute" value=""/>

            <parameter key="empty_window_handling" value="add empty exampleset"/>

          </operator>

          <operator activated="true" class="h2o:deep_learning" compatibility="10.3.001" expanded="true" height="103" name="Deep Learning" width="90" x="313" y="34">

            <parameter key="activation" value="Rectifier"/>

            <enumeration key="hidden_layer_sizes">

              <parameter key="hidden_layer_sizes" value="50"/>

              <parameter key="hidden_layer_sizes" value="50"/>

            </enumeration>

            <enumeration key="hidden_dropout_ratios"/>

            <parameter key="reproducible_(uses_1_thread)" value="false"/>

            <parameter key="use_local_random_seed" value="false"/>

            <parameter key="local_random_seed" value="1992"/>

            <parameter key="epochs" value="10.0"/>

            <parameter key="compute_variable_importances" value="false"/>

            <parameter key="train_samples_per_iteration" value="-2"/>

            <parameter key="adaptive_rate" value="true"/>

            <parameter key="epsilon" value="1.0E-8"/>

            <parameter key="rho" value="0.99"/>

            <parameter key="learning_rate" value="0.005"/>

            <parameter key="learning_rate_annealing" value="1.0E-6"/>

            <parameter key="learning_rate_decay" value="1.0"/>

            <parameter key="momentum_start" value="0.0"/>

            <parameter key="momentum_ramp" value="1000000.0"/>

            <parameter key="momentum_stable" value="0.0"/>

            <parameter key="nesterov_accelerated_gradient" value="true"/>

            <parameter key="standardize" value="true"/>

            <parameter key="L1" value="1.0E-5"/>

            <parameter key="L2" value="0.0"/>

            <parameter key="max_w2" value="10.0"/>

            <parameter key="loss_function" value="Automatic"/>

            <parameter key="distribution_function" value="AUTO"/>

            <parameter key="early_stopping" value="false"/>

            <parameter key="stopping_rounds" value="1"/>

            <parameter key="stopping_metric" value="AUTO"/>

            <parameter key="stopping_tolerance" value="0.001"/>

            <parameter key="missing_values_handling" value="MeanImputation"/>

            <parameter key="max_runtime_seconds" value="0"/>

            <list key="expert_parameters"/>

            <list key="expert_parameters_"/>

          </operator>

          <operator activated="true" class="apply_model" compatibility="10.3.001" expanded="true" height="82" name="Apply Model" width="90" x="447" y="34">

            <list key="application_parameters"/>

          </operator>

          <operator activated="true" class="performance_regression" compatibility="10.3.001" expanded="true" height="82" name="Performance (2)" width="90" x="581" y="34">

            <parameter key="main_criterion" value="first"/>

            <parameter key="root_mean_squared_error" value="true"/>

            <parameter key="absolute_error" value="false"/>

            <parameter key="relative_error" value="false"/>

            <parameter key="relative_error_lenient" value="false"/>

            <parameter key="relative_error_strict" value="false"/>

            <parameter key="normalized_absolute_error" value="false"/>

            <parameter key="root_relative_squared_error" value="false"/>

            <parameter key="squared_error" value="false"/>

            <parameter key="correlation" value="false"/>

            <parameter key="squared_correlation" value="false"/>

            <parameter key="prediction_average" value="false"/>

            <parameter key="spearman_rho" value="false"/>

            <parameter key="kendall_tau" value="false"/>

            <parameter key="skip_undefined_labels" value="true"/>

            <parameter key="use_example_weights" value="true"/>

          </operator>

          <connect from_port="input 1" to_op="Windowing (Series)" to_port="example set"/>

          <connect from_op="Set Macro (Real)" from_port="through 1" to_op="Generate Macro" to_port="through 1"/>

          <connect from_op="Windowing (Series)" from_port="windowed example set" to_op="Deep Learning" to_port="training set"/>

          <connect from_op="Deep Learning" from_port="model" to_op="Apply Model" to_port="model"/>

          <connect from_op="Deep Learning" from_port="exampleSet" to_op="Apply Model" to_port="unlabelled data"/>

          <connect from_op="Apply Model" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>

          <connect from_op="Performance (2)" from_port="performance" to_port="performance"/>

          <portSpacing port="source_input 1" spacing="0"/>

          <portSpacing port="source_input 2" spacing="0"/>

          <portSpacing port="sink_performance" spacing="0"/>

          <portSpacing port="sink_model" spacing="0"/>

          <portSpacing port="sink_output 1" spacing="0"/>

        </process>

      </operator>

      <connect from_op="Retrieve Lake Huron" from_port="output" to_op="Optimize Parameters (Grid)" to_port="input 1"/>

      <connect from_op="Optimize Parameters (Grid)" from_port="performance" to_port="result 1"/>

      <portSpacing port="source_input 1" spacing="0"/>

      <portSpacing port="sink_result 1" spacing="0"/>

      <portSpacing port="sink_result 2" spacing="0"/>

    </process>

  </operator>

</process>

ceaperez

Hi @vgpowered, 
yes, is possible, please find attached a simple sample using a modification of the windows size operator tutorial.

best, 

Cesar

vgpowered

Hi Cesar,
Sorry, Where is the attachment?
By the way I'm using Windowing operator inside a Optimize parameter (Grid)...

ceaperez

Hi @vgpowered, 
The attached file is in the bottom of my message.

Can you share your process? 

Best, 

Cesar 

vgpowered

Hi Cesar,
I can only see an icon (it is a page with the upper right corner folded). When I move the mouse over this icon the "Macrowinsize.rmp" message appear, but nothing else. I think you have attached an image, not a file.

My process is very complex, but only a draft (Accepts my apology)

Víctor

vgpowered

Hi Cesar
Maybe the Rapidminer Web is not working. I'm testing my own message and it shows somthing similar like yours (about attached files)...

ceaperez

Hi @vgpowered, 
I can download your process. Anyway this is the xml 

<?xml version="1.0" encoding="UTF-8"?><process version="10.3.001">

  <context>

    <input/>

    <output/>

    <macros/>

  </context>

  <operator activated="true" class="process" compatibility="10.3.001" expanded="true" name="Process" origin="GENERATED_TUTORIAL">

    <parameter key="logverbosity" value="init"/>

    <parameter key="random_seed" value="2001"/>

    <parameter key="send_mail" value="never"/>

    <parameter key="notification_email" value=""/>

    <parameter key="process_duration_for_mail" value="30"/>

    <parameter key="encoding" value="SYSTEM"/>

    <process expanded="true">

      <operator activated="true" class="retrieve" compatibility="10.3.001" expanded="true" height="68" name="Retrieve Lake Huron" origin="GENERATED_TUTORIAL" width="90" x="112" y="34">

        <parameter key="repository_entry" value="//Samples/Time Series/data sets/Lake Huron"/>

      </operator>

      <operator activated="true" class="set_macro" compatibility="10.3.001" expanded="true" height="82" name="Set Macro" width="90" x="112" y="238">

        <parameter key="macro" value="Winsiz"/>

        <parameter key="value" value="4"/>

      </operator>

      <operator activated="true" class="time_series:process_windows" compatibility="10.3.001" expanded="true" height="103" name="Process Windows" origin="GENERATED_TUTORIAL" width="90" x="313" y="34">

        <parameter key="attribute_filter_type" value="all"/>

        <parameter key="attribute" value=""/>

        <parameter key="attributes" value=""/>

        <parameter key="use_except_expression" value="false"/>

        <parameter key="value_type" value="numeric"/>

        <parameter key="use_value_type_exception" value="false"/>

        <parameter key="except_value_type" value="real"/>

        <parameter key="block_type" value="value_series"/>

        <parameter key="use_block_type_exception" value="false"/>

        <parameter key="except_block_type" value="value_series_end"/>

        <parameter key="invert_selection" value="false"/>

        <parameter key="include_special_attributes" value="false"/>

        <parameter key="has_indices" value="true"/>

        <parameter key="indices_attribute" value="Date"/>

        <parameter key="sort_time_series" value="false"/>

        <parameter key="expert_settings" value="false"/>

        <parameter key="unit" value="example based"/>

        <parameter key="windows_defined" value="from start"/>

        <parameter key="custom_start_point" value="5"/>

        <parameter key="custom_end_point" value="100"/>

        <parameter key="window_size" value="%{Winsiz}"/>

        <parameter key="custom_start_time" value="2000-01-01 00:00:00"/>

        <parameter key="custom_end_time" value="2030-01-01 00:00:00"/>

        <parameter key="date_format" value="yyyy-MM-dd HH:mm:ss"/>

        <parameter key="window_size_time" value="1.Hours"/>

        <parameter key="windows_stop_definition" value="from next window start"/>

        <parameter key="window_start_attribute" value=""/>

        <parameter key="window_stop_attribute" value=""/>

        <parameter key="no_overlapping_windows" value="false"/>

        <parameter key="step_size" value="1"/>

        <parameter key="step_size_time" value="1.Minutes"/>

        <parameter key="create_horizon_(labels)" value="true"/>

        <parameter key="horizon_attribute" value="Lake surface level / feet"/>

        <parameter key="horizon_size" value="1"/>

        <parameter key="horizon_offset" value="0"/>

        <parameter key="horizon_size_time" value="1.Hours"/>

        <parameter key="horizon_offset_time" value="0.Minutes"/>

        <parameter key="horizon_start_attribute" value=""/>

        <parameter key="horizon_stop_attribute" value=""/>

        <parameter key="empty_window_handling" value="add empty exampleset"/>

        <parameter key="add_last_index_in_window_attribute" value="true"/>

        <parameter key="enable_parallel_execution" value="true"/>

        <process expanded="true">

          <operator activated="true" class="time_series:extract_std_descriptive_features" compatibility="10.3.001" expanded="true" height="82" name="Extract Aggregates" origin="GENERATED_TUTORIAL" width="90" x="313" y="34">

            <parameter key="attribute_filter_type" value="all"/>

            <parameter key="attribute" value=""/>

            <parameter key="attributes" value=""/>

            <parameter key="use_except_expression" value="false"/>

            <parameter key="value_type" value="numeric"/>

            <parameter key="use_value_type_exception" value="false"/>

            <parameter key="except_value_type" value="real"/>

            <parameter key="block_type" value="value_series"/>

            <parameter key="use_block_type_exception" value="false"/>

            <parameter key="except_block_type" value="value_series_end"/>

            <parameter key="invert_selection" value="false"/>

            <parameter key="include_special_attributes" value="false"/>

            <parameter key="sum" value="true"/>

            <parameter key="mean" value="true"/>

            <parameter key="geometric_mean" value="true"/>

            <parameter key="first_quartile" value="true"/>

            <parameter key="median" value="true"/>

            <parameter key="third_quartile" value="true"/>

            <parameter key="min" value="true"/>

            <parameter key="max" value="true"/>

            <parameter key="std_deviation" value="true"/>

            <parameter key="kurtosis" value="true"/>

            <parameter key="skewness" value="true"/>

            <parameter key="add_time_series_name" value="false"/>

            <parameter key="ignore_invalid_values" value="false"/>

          </operator>

          <connect from_port="windowed example set" to_op="Extract Aggregates" to_port="example set"/>

          <connect from_op="Extract Aggregates" from_port="features" to_port="output 1"/>

          <portSpacing port="source_windowed example set" spacing="0"/>

          <portSpacing port="source_input 1" spacing="0"/>

          <portSpacing port="sink_output 1" spacing="0"/>

          <portSpacing port="sink_output 2" spacing="0"/>

        </process>

      </operator>

      <operator activated="true" class="append" compatibility="10.3.001" expanded="true" height="82" name="Append" origin="GENERATED_TUTORIAL" width="90" x="514" y="34">

        <parameter key="datamanagement" value="double_array"/>

        <parameter key="data_management" value="auto"/>

        <parameter key="merge_type" value="all"/>

      </operator>

      <connect from_op="Retrieve Lake Huron" from_port="output" to_op="Process Windows" to_port="example set"/>

      <connect from_op="Set Macro" from_port="through 1" to_op="Process Windows" to_port="custom windows"/>

      <connect from_op="Process Windows" from_port="output 1" to_op="Append" to_port="example set 1"/>

      <connect from_op="Append" from_port="merged set" to_port="result 1"/>

      <portSpacing port="source_input 1" spacing="0"/>

      <portSpacing port="sink_result 1" spacing="0"/>

      <portSpacing port="sink_result 2" spacing="0"/>

    </process>

  </operator>

</process>

vgpowered

Thank Cesar,
I think the problem is the Optimization operator.
I have build a little example that demostrates there is a limitation.
Do not lose your time with the previous complex procces, simply test the next example.

Víctor

ceaperez

Hola Victor,
Just put the macro outside of the optimization operator.

Saludos, 

Cesar

vgpowered

Hi Cesar,
But in this case you cannot optimize its value. You could say, you can optimize it directly, but in my real process there are several windowing operators (inside the Optimize process) that share this same value but not others. This is the issue.

Víctor

ceaperez

Hi @vgpowered, 
I have a possible solution:

Best,

César

vgpowered

Hi Cesar,
The attachment does not work.

Regards
Víctor

vgpowered

Hi Cesar,
Í cannot download the "windowing with optimiza.rmp" . Icon doesn't work,

Víctor

vgpowered

Hello César,
Yes, this is the solution. I did'n have installed the Operator Toolbox set of operators. In all cases this is a real case of trick solutión. I think Ripidminer has a problem if we have to use this approach.

It should be more efficient to create a new versión of the optimize operator.

Thank you very much.

Víctor

ceaperez

Hi @vgpowered,
You are right, this is a tricky solution, but these kinds of issues help the RM team improve the product. I'm not part of the RM team so we need help from someone there to review this issue or find other way, maybe @rjones13 can follow up this issue. 

best 

Cesar