RapidMiner

Optimize Deep Learning's network structure and parameters

Contributor II cyborghijacker
Contributor II

Re: Optimize Deep Learning's network structure and parameters

No problem, and thank you for the clarification.

 

In fact, in Thomas' earlier video (https://www.youtube.com/watch?v=R5vPrTLMzng&list=PLLdyjsPklEJvpxLHZM-llt40FLw-qEiZO&index=1), around the 8:40 min mark, we also see that hidden_layers for Neural network is greyed out, although it was not highlighted.

 

I am still unsure how to use the Set Macros using the script Thomas posted. Where should it be placed? Nested within the Optimize Parameters operater before Cross Validation?

 

Regards

Ben

Maven
Maven

Re: Optimize Deep Learning's network structure and parameters

You can just copy the XML of Thomas' process and paste it into your XML panel in RapidMiner. Then press the green checkmark and the process will appear in the process panel.

Contributor II cyborghijacker
Contributor II

Re: Optimize Deep Learning's network structure and parameters

Hi FBT

 

I see. In essence, its just creating the no. of Set Macro operators to be the no. of hidden layers? I'm assuming the Set Macros can be used instead of defining one Set Macro for one hidden layer. For the optimizing, I will still have to change the macro value ( = no. of nodes) manually correct?

 

Regards

Ben

Highlighted
Maven
Maven

Re: Optimize Deep Learning's network structure and parameters

Hi,

not really. The whole point about using "Set Macro" operators is to make the parameters available within the "Optimize" operator in which you can set the values you would like to iterate through. Take a look at the screenshot:

HiddenLayerOptimization.png

I have renamed the two "Set Macro" operators from Thomas' process to make it a bit clearer in the optimization pane. You can now either use the grid selection by setting a range of values and the distance between values for the layer, or you directly define your desired values with a list (you would just need to check the "List" radio button).

 

Using the "Set Macros" operator with which several macros can be defined at once, brings you back to the problem of the selection being "greyed out" in the optimization pane. Hence you would need to use one "Set Macro" operator for every hidden layer you want to iterate through.

 

Contributor II cyborghijacker
Contributor II

Re: Optimize Deep Learning's network structure and parameters

Hi FBT

 

I see, I can get a clearer picture now, your first statement was key. Now I face another strange Problem Message. I defined 3 layers as per Set Macros connected to Cross-Validation with a Deep Learning operator within it.

 

 

Any chance this error could be diagnosed? Has it got to do with strings vs. integers in Rapidminer?

Maven
Maven

Re: Optimize Deep Learning's network structure and parameters

Ok, you are right. Values set in the "Set Macro" operator are strings. This is usually not a problem in other contexts, as you can use eval() or parse() to convert it to a number. In your context, however, this does not work and we need to get a bit more creative. What seems to work is the following:

 

<?xml version="1.0" encoding="UTF-8"?><process version="7.5.003">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.5.003" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="optimize_parameters_grid" compatibility="7.5.003" expanded="true" height="166" name="Optimize Parameters (Grid)" width="90" x="380" y="34">
        <list key="parameters">
          <parameter key="Extract Macro - Layer 1.example_index" value="[1;6;6;linear]"/>
          <parameter key="Extract Macro - Layer 2.example_index" value="[1;6;6;linear]"/>
        </list>
        <process expanded="true">
          <operator activated="true" class="read_excel" compatibility="7.5.003" expanded="true" height="68" name="Read Excel" width="90" x="45" y="136">
            <parameter key="first_row_as_names" value="false"/>
            <list key="annotations">
              <parameter key="0" value="Name"/>
            </list>
            <list key="data_set_meta_data_information"/>
          </operator>
          <operator activated="true" class="extract_macro" compatibility="7.5.003" expanded="true" height="68" name="Extract Macro - Layer 1" width="90" x="179" y="136">
            <parameter key="macro" value="layer1"/>
            <parameter key="macro_type" value="data_value"/>
            <parameter key="attribute_name" value="LayerSize"/>
            <parameter key="example_index" value="6"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="extract_macro" compatibility="7.5.003" expanded="true" height="68" name="Extract Macro - Layer 2" width="90" x="313" y="136">
            <parameter key="macro" value="layer2"/>
            <parameter key="macro_type" value="data_value"/>
            <parameter key="attribute_name" value="LayerSize"/>
            <parameter key="example_index" value="6"/>
            <list key="additional_macros"/>
          </operator>
          <operator activated="true" class="concurrency:cross_validation" compatibility="7.5.003" expanded="true" height="145" name="Cross Validation" width="90" x="447" y="34">
            <process expanded="true">
              <operator activated="true" class="h2o:deep_learning" compatibility="7.5.000" expanded="true" height="82" name="Deep Learning" width="90" x="179" y="34">
                <enumeration key="hidden_layer_sizes">
                  <parameter key="hidden_layer_sizes" value="%{layer1}"/>
                  <parameter key="hidden_layer_sizes" value="%{layer2}"/>
                </enumeration>
                <enumeration key="hidden_dropout_ratios"/>
                <list key="expert_parameters"/>
                <list key="expert_parameters_"/>
              </operator>
              <connect from_port="training set" to_op="Deep Learning" to_port="training set"/>
              <connect from_op="Deep Learning" from_port="model" to_port="model"/>
              <portSpacing port="source_training set" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="7.5.003" expanded="true" height="82" name="Apply Model (2)" width="90" x="112" y="34">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="7.5.003" expanded="true" height="82" name="Performance" width="90" x="246" y="34">
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance" from_port="performance" to_port="performance 1"/>
              <connect from_op="Performance" from_port="example set" to_port="test set results"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_test set results" spacing="0"/>
              <portSpacing port="sink_performance 1" spacing="0"/>
              <portSpacing port="sink_performance 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="log" compatibility="7.5.003" expanded="true" height="82" name="Log" width="90" x="648" y="136">
            <list key="log">
              <parameter key="Count" value="operator.SVM.value.applycount"/>
              <parameter key="SVM C" value="operator.SVM.parameter.C"/>
              <parameter key="SVM gamma" value="operator.SVM.parameter.gamma"/>
              <parameter key="Testing Accuracy" value="operator.Validation.value.performance"/>
            </list>
          </operator>
          <connect from_port="input 1" to_op="Cross Validation" to_port="example set"/>
          <connect from_op="Read Excel" from_port="output" to_op="Extract Macro - Layer 1" to_port="example set"/>
          <connect from_op="Extract Macro - Layer 1" from_port="example set" to_op="Extract Macro - Layer 2" to_port="example set"/>
          <connect from_op="Cross Validation" from_port="model" to_port="result 1"/>
          <connect from_op="Cross Validation" from_port="example set" to_port="result 2"/>
          <connect from_op="Cross Validation" from_port="test result set" to_port="result 3"/>
          <connect from_op="Cross Validation" from_port="performance 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="performance"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
          <portSpacing port="sink_result 4" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Optimize Parameters (Grid)" from_port="parameter" to_port="result 1"/>
      <connect from_op="Optimize Parameters (Grid)" from_port="result 1" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>

This process is very similar to the one before, however, instead of using "Set Macro" operators, you start by reading in an Excel or CSV File that consists of one column --> "LayerSize". In this column you have the hidden layer sizes you want to iterate through. It is important that you read the values in as numerical types, otherwise you will run into the same issue as before. If in doubt, use the import wizard.

 

Now, instead of setting the macro values directly, you can extract them from the read-in file, which contains your layer sizes, with the "Extract Macro" operator. By setting the macro type to "data value" you get a new parameter called "example index". This is basically just the row number in your Excel sheet and luckily you can iterate through it within the Optimize operator.

 

OptimizeParameter_ExtractMacro.png 

My Excel sheet had six values for hidden layers, hence I am iterating through from 1 to 6. You will need to adapt accordingly, depending on how many values for hidden layers you have. You need to use one "Extract Macro" operator for each hidden layer you would like to iterate through. If the values you would like to iterate through for different hidden layers are not identical, you would need to create more than one Excel/CSV file and read them in, as shown in the process above.

 

Contributor II cyborghijacker
Contributor II

Re: Optimize Deep Learning's network structure and parameters

Hi FBT

 

Oh I see, I was trying to put eval() into the hidden layer size in the Deep Learning operator. So it was an issue with 'strings' being read by the Set Macro operator. I have implemented the workaround that you have suggested, and it works well, although sometimes the Deep Learning shows a Warning message about the macro not being defined.

 

On a separate note, do you know what the 'standardize' parameter in DL does? It says:

 

standardize (optional)
If enabled, automatically standardize the data. If disabled, the user must provide properly scaled input data.

 

 

Once again, thank you for your detailed clarifications on this topic.

 

Regards

Maven
Maven

Re: Optimize Deep Learning's network structure and parameters

I believe that this parameter rescales your attributes to have a mean of 0 and and a standard deviation of 1. Also, it assumes that your data has a Gaussian distribution. 

Contributor II cyborghijacker
Contributor II

Re: Optimize Deep Learning's network structure and parameters

I see, so its essentially an in-built Normalize operator with the Z-scaling method. I'm assuming the user can choose between a manual normalization or simply checking this parameter?

Polls
How can RapidMiner increase participation in our new competitions?
Twitter Feed