RapidMiner

Optimize Deep Learning's network structure and parameters

Regular Contributor

Optimize Deep Learning's network structure and parameters

[ Edited ]

Hi RapidMiner,

 

I'm doing regression using "Deep Learning" operator, I have 480 input features (this is a predictive maintenance problem, each feature is a meter reading, and we want to apply regression to predict the next-time-to-fail of an asset). After training the "Deep Learning" operator, the Root Mean Square Error (RMSE) applied on the training datatset is still quite high (from 0.08 to 0.2), although the training dataset is normalized into [-1; 1]. I also tried a lot of network structures, including increasing the number of hidden layers (up to 15), and increasing the number of nodes in each hidden layer (up 1000 nodes/layer). In some cases, doing so even increases the RMSE on the training dataset, which means the model is under-fitting. I used the default values for the other deep learning parameters, including the adaptive learning rate and rectifier activation.

 

So do you have any piece of advices for this situation, or is there any way to optimize the network structure? (coz I already tried the "Optimize Parameters" operator for "Deep Learning" but could not find the operator's parameters for network structure). Or is there any way to make the deep learning operator fit the training data better?

 

Thank you very much for your help!

 

Best regards,
Phivu

18 REPLIES
Moderator

Re: Optimize Deep Learning's network structure and parameters

I'm not sure what you mean by "coz I already tried the "Optimize Parameters" operator for "Deep Learning" but could not find the operator's parameters for network structure." You tried an Optimize Parameter operator, embedded a Cross Validation operator and inside a Deep Learning operator....and nothing showed up?

Regular Contributor

Re: Optimize Deep Learning's network structure and parameters

Hi Thomas,

 

Yes. I attached the process and a screenshot here. In the screenshot, you can see the "hidden_layer_sizes" parameter is disabled when I choose the "Deep Learning" operator for optimization. So how can i enable it?

 

Optimize-deep-learning-parameters.png

Attachments

Moderator

Re: Optimize Deep Learning's network structure and parameters

Huh, that's interesting.  Try using Set Macros to inside the hidden layers and vary those values. 

<?xml version="1.0" encoding="UTF-8"?><process version="7.3.001">
  <context>
    <input/>
    <output/>
    <macros/>
  </context>
  <operator activated="true" class="process" compatibility="7.3.001" expanded="true" name="Process">
    <process expanded="true">
      <operator activated="true" class="optimize_parameters_grid" compatibility="7.3.001" expanded="true" height="166" name="Optimize Parameters (Grid)" width="90" x="447" y="34">
        <list key="parameters">
          <parameter key="Set Macro.value" value="[0;100;10;linear]"/>
          <parameter key="Set Macro (2).value" value="[0;100;10;linear]"/>
        </list>
        <process expanded="true">
          <operator activated="true" class="set_macro" compatibility="7.3.001" expanded="true" height="82" name="Set Macro" width="90" x="112" y="34">
            <parameter key="macro" value="layer1"/>
            <parameter key="value" value="0.0"/>
          </operator>
          <operator activated="true" class="set_macro" compatibility="7.3.001" expanded="true" height="82" name="Set Macro (2)" width="90" x="246" y="34">
            <parameter key="macro" value="layer2"/>
            <parameter key="value" value="0.0"/>
          </operator>
          <operator activated="true" class="concurrency:cross_validation" compatibility="7.3.001" expanded="true" height="145" name="Cross Validation" width="90" x="447" y="34">
            <process expanded="true">
              <operator activated="true" class="h2o:deep_learning" compatibility="7.3.000" expanded="true" height="82" name="Deep Learning" width="90" x="179" y="34">
                <enumeration key="hidden_layer_sizes">
                  <parameter key="hidden_layer_sizes" value="%{layer1}"/>
                  <parameter key="hidden_layer_sizes" value="%{layer2}"/>
                </enumeration>
                <enumeration key="hidden_dropout_ratios"/>
                <list key="expert_parameters"/>
                <list key="expert_parameters_"/>
              </operator>
              <connect from_port="training set" to_op="Deep Learning" to_port="training set"/>
              <connect from_op="Deep Learning" from_port="model" to_port="model"/>
              <portSpacing port="source_training set" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="7.3.001" expanded="true" height="82" name="Apply Model (2)" width="90" x="112" y="34">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_regression" compatibility="7.3.001" expanded="true" height="82" name="Performance (2)" width="90" x="246" y="34"/>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (2)" to_port="labelled data"/>
              <connect from_op="Performance (2)" from_port="performance" to_port="performance 1"/>
              <connect from_op="Performance (2)" from_port="example set" to_port="test set results"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_test set results" spacing="0"/>
              <portSpacing port="sink_performance 1" spacing="0"/>
              <portSpacing port="sink_performance 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="log" compatibility="7.3.001" expanded="true" height="82" name="Log" width="90" x="648" y="136">
            <parameter key="filename" value="D:\Predictive-Maintenance-Project\RapidMiner-Processes\SVM-Parameters\Fault1C-log-values-gamma-0.001-0.05-XV.txt.log"/>
            <list key="log">
              <parameter key="Count" value="operator.SVM.value.applycount"/>
              <parameter key="SVM C" value="operator.SVM.parameter.C"/>
              <parameter key="SVM gamma" value="operator.SVM.parameter.gamma"/>
              <parameter key="Testing Accuracy" value="operator.Validation.value.performance"/>
            </list>
          </operator>
          <connect from_port="input 1" to_op="Set Macro" to_port="through 1"/>
          <connect from_op="Set Macro" from_port="through 1" to_op="Set Macro (2)" to_port="through 1"/>
          <connect from_op="Set Macro (2)" from_port="through 1" to_op="Cross Validation" to_port="example set"/>
          <connect from_op="Cross Validation" from_port="model" to_port="result 1"/>
          <connect from_op="Cross Validation" from_port="example set" to_port="result 2"/>
          <connect from_op="Cross Validation" from_port="test result set" to_port="result 3"/>
          <connect from_op="Cross Validation" from_port="performance 1" to_op="Log" to_port="through 1"/>
          <connect from_op="Log" from_port="through 1" to_port="performance"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="source_input 2" spacing="0"/>
          <portSpacing port="sink_performance" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
          <portSpacing port="sink_result 4" spacing="0"/>
        </process>
      </operator>
      <connect from_op="Optimize Parameters (Grid)" from_port="parameter" to_port="result 1"/>
      <connect from_op="Optimize Parameters (Grid)" from_port="result 1" to_port="result 2"/>
      <portSpacing port="source_input 1" spacing="0"/>
      <portSpacing port="sink_result 1" spacing="0"/>
      <portSpacing port="sink_result 2" spacing="0"/>
      <portSpacing port="sink_result 3" spacing="0"/>
    </process>
  </operator>
</process>
Regular Contributor

Re: Optimize Deep Learning's network structure and parameters

Hi Thomas

 

I have encountered the same issue, where the network hyperparameters are greyed out in the grid search UI. May I know how do I implement the method you have mentioned? Where do I place the Set Macros?

 

Thank you

 

Regards

Corse

Moderator

Re: Optimize Deep Learning's network structure and parameters

Ah I think you have to toggle on 'List.' 

Regular Contributor

Re: Optimize Deep Learning's network structure and parameters

How would I do that? The options are greyed out actually (i.e. Grid/List)

rapidminer_qn.png

FBT
Super Contributor

Re: Optimize Deep Learning's network structure and parameters

You need to select the parameters you would like to optimize first (the middle column on top). Once you have done that the selection between "Grid" and "List" becomes available. 

Regular Contributor

Re: Optimize Deep Learning's network structure and parameters

Yes it works with the other parameters, but in this case the Parameters of hidden_layer_sizes and hidden_dropout_ratios are greyed out, clicking on them yields no further selection options.

 

Has anyone else encountered this issue? Apart from this, doing a manual search is possible, but it will be a hair thinning exercise. I shuddered at that thought.

Highlighted
FBT
Super Contributor

Re: Optimize Deep Learning's network structure and parameters

Ok. Apologies, I did not read the full thread up until now. It is indeed a bit unusual that this is greyed out. Have you taken a look at the process that @Thomas_Ott posted above? In it, he set macros for the layer sizes within the "Optimize Parameters" operator, which are then used in the hidden layers panel of the "Deep Learning" operator, instead of using actual numerical values --> see screenshot.

DeepLearningScreenshot.png

 

If you do that, you should be able to select those said "Set Macros" operators within the optimization panel and enter the values that you would like to iterate through.