Accuracy remains constant

chris_mlchris_ml Member Posts: 17 Maven
edited August 2019 in Help
I want to optimize the parameters of the decision tree learner using EvolutionaryParameterOptimization.
That's my model:

<?xml version="1.0" encoding="US-ASCII"?>
<process version="4.3">
  <operator name="Root" class="Process" expanded="yes">
      <operator name="CSVExampleSource" class="CSVExampleSource">
          <parameter key="filename"    value="example.csv"/>
          <parameter key="label_name"  value="label"/>
      </operator>
      <operator name="EvolutionaryParameterOptimization" class="EvolutionaryParameterOptimization" expanded="yes">
          <list key="parameters">
            <parameter key="DecisionTree.maximal_depth" value="[-1.0;10000.0]"/>
            <parameter key="DecisionTree.minimal_leaf_size"    value="[1.0;10000.0]"/>
            <parameter key="DecisionTree.confidence"    value="[1.0E-7;0.5]"/>
            <parameter key="DecisionTree.minimal_size_for_split"        value="[1.0;10000.0]"/>
            <parameter key="DecisionTree.minimal_gain"  value="[0.0;Infinity]"/>
          </list>
          <parameter key="population_size"      value="10"/>
          <operator name="IteratingPerformanceAverage" class="IteratingPerformanceAverage" expanded="yes">
              <parameter key="iterations"      value="3"/>
              <operator name="XValidation" class="XValidation" expanded="yes">
                  <parameter key="sampling_type"        value="shuffled sampling"/>
                  <operator name="DecisionTree" class="DecisionTree">
                      <parameter key="confidence"      value="0.33223030703745715"/>
                      <parameter key="maximal_depth"    value="6341"/>
                      <parameter key="minimal_gain"    value="Infinity"/>
                      <parameter key="minimal_leaf_size"        value="1424"/>
                      <parameter key="minimal_size_for_split"  value="2961"/>
                  </operator>
                  <operator name="OperatorChain" class="OperatorChain" expanded="yes">
                      <operator name="ModelApplier" class="ModelApplier">
                          <list key="application_parameters">
                          </list>
                      </operator>
                      <operator name="ClassificationPerformance" class="ClassificationPerformance">
                          <parameter key="absolute_error"      value="true"/>
                          <parameter key="accuracy"    value="true"/>
                          <list key="class_weights">
                          </list>
                          <parameter key="classification_error" value="true"/>
                      </operator>
                  </operator>
              </operator>
          </operator>
          <operator name="ProcessLog" class="ProcessLog">
              <parameter key="filename" value="process3.log"/>
              <list key="log">
                <parameter key="iteration"      value="operator.XValidation.value.iteration"/>
                <parameter key="time"  value="operator.XValidation.value.time"/>
                <parameter key="deviation"      value="operator.XValidation.value.deviation"/>
                <parameter key="accuracy"      value="operator.XValidation.value.performance1"/>
                <parameter key="max_depth"      value="operator.DecisionTree.parameter.maximal_depth"/>
                <parameter key="max_leaf_size"  value="operator.DecisionTree.parameter.minimal_leaf_size"/>
                <parameter key="confidence"    value="operator.DecisionTree.parameter.confidence"/>
              </list>
              <parameter key="persistent"      value="true"/>
          </operator>
      </operator>
      <operator name="PerformanceWriter" class="PerformanceWriter">
          <parameter key="performance_file"    value="final_performance.per"/>
      </operator>
      <operator name="ParameterSetWriter" class="ParameterSetWriter">
          <parameter key="parameter_file"      value="parameters.par"/>
      </operator>
  </operator>
</process>
The strange thing is that the accuracy remains always (almost) the same. Here are the first
lines of the generated log file:

# Generated by ProcessLog[com.rapidminer.operator.visualization.ProcessLogOperator]
# iteration    time    deviation      accuracy        max_depth      max_leaf_size  confidence
10.0    23.0    0.05129744389973656    0.7932203389830508      9839.0  1251.0  0.34154054826704505
10.0    23.0    0.05507822647211116    0.7932203389830507      6341.0  7767.0  0.32651644400635255
10.0    23.0    0.0392401250942013      0.793220338983051      3707.0  48.0    0.397380892497813
10.0    22.0    0.0726246958834861      0.7932203389830509      9008.0  5164.0  0.40415961507086146
10.0    25.0    0.03774755500223652    0.7932203389830509      7652.0  3992.0  0.27277836592326715
10.0    28.0    0.055078226472110144    0.7932203389830507      293.0  1424.0  0.03868159703154637
10.0    27.0    0.06287198979997247    0.7932203389830507      615.0  6926.0  0.12025785562389255
10.0    46.0    0.026037782196170964    0.7932203389830507      4846.0  1825.0  0.4842464140080626
Even when I replace the decision tree learned by a neural network, the accuracy values are always
the same. But I can't imagine that the number of correctly classified examples is always the same. Any
ideas what I'm dong wrong?

Chris
Tagged:

Answers

  • haddockhaddock Member Posts: 849 Maven
    Hi Chris,

    Are you really sure that you should have 'infinity' as your minimal split parameter?  ;) I'm trying to get my head round an infinite gain....

    HAPPY NEW YEAR TO ALL!
  • chris_mlchris_ml Member Posts: 17 Maven
    Hi,

    OK, maybe infinity as minimal gain is little bit too large. :-)

    However, this seems not to be the problem. After reducing this
    parameter to max. 100, I still get same results where the accuracy
    is remaining (almost) constant. Any other ideas what might be wrong?

    By the way, what exactly is the definition of an "minimal size of
    a leaf" for the DecisionTree operator? This is used in the parameter
    "minimal_size_for_split". Is it the value computed by the criterion?

    Also happy new year to all.

    Chris
  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi Chris,
    as far as I can see from your posted log, the parameters have never been within a region, where changes might occur.
    One possibility for constant accuracy is the collapsing of the models into a default model. Perhabs you smaller class is never discovered by the learner?

    Greetings,
      Sebastian
  • chris_mlchris_ml Member Posts: 17 Maven
    Hi Sebastian,

    as far as I can see from your posted log, the parameters have never been within a region, where changes might occur.
    Could you explain that in more detail? I didn't fully catch what you mean by "regions, where changes might occur".

    One possibility for constant accuracy is the collapsing of the models into a default model. Perhabs you smaller class is never discovered by the learner?
    When can it happen that RapidMiner collapses a model? Can this be somehow recognized?

    I've now simplified my model:

    <?xml version="1.0" encoding="US-ASCII"?>
    <process version="4.3">

      <operator name="Root" class="Process" expanded="yes">
          <operator name="CSVExampleSource" class="CSVExampleSource">
              <parameter key="filename" value="examples.csv"/>
              <parameter key="label_name" value="label"/>
          </operator>
          <operator name="EvolutionaryParameterOptimization" class="EvolutionaryParameterOptimization" expanded="yes">
              <list key="parameters">
                <parameter key="DecisionTree.maximal_depth" value="[-1.0;100.0]"/>
                <parameter key="DecisionTree.minimal_leaf_size" value="[1.0;100.0]"/>
                <parameter key="DecisionTree.confidence" value="[1.0E-7;0.5]"/>
                <parameter key="DecisionTree.minimal_size_for_split" value="[1.0;100.0]"/>
                <parameter key="DecisionTree.minimal_gain" value="[0.0;100.0]"/>
              </list>
              <parameter key="population_size" value="10"/>
              <operator name="IteratingPerformanceAverage" class="IteratingPerformanceAverage" expanded="yes">
                  <parameter key="iterations" value="3"/>
                  <operator name="XValidation" class="XValidation" expanded="yes">
                      <parameter key="sampling_type" value="shuffled sampling"/>
                      <operator name="DecisionTree" class="DecisionTree">
                          <parameter key="confidence" value="0.397380892497813"/>
                          <parameter key="maximal_depth" value="36"/>
                          <parameter key="minimal_gain" value="29.447796060205512"/>
                          <parameter key="minimal_leaf_size" value="15"/>
                          <parameter key="minimal_size_for_split" value="27"/>
                      </operator>
                      <operator name="OperatorChain" class="OperatorChain" expanded="yes">
                          <operator name="ModelApplier" class="ModelApplier">
                              <list key="application_parameters">
                              </list>
                          </operator>
                          <operator name="ClassificationPerformance" class="ClassificationPerformance">
                              <parameter key="absolute_error" value="true"/>
                              <parameter key="accuracy" value="true"/>
                              <list key="class_weights">
                              </list>
                              <parameter key="classification_error" value="true"/>
                          </operator>
                      </operator>
                  </operator>
              </operator>
              <operator name="ProcessLog" class="ProcessLog">
                  <parameter key="filename" value="process.log"/>
                  <list key="log">
                    <parameter key="iteration" value="operator.XValidation.value.iteration"/>
                    <parameter key="time" value="operator.XValidation.value.time"/>
                    <parameter key="deviation" value="operator.XValidation.value.deviation"/>
                    <parameter key="accuracy" value="operator.XValidation.value.performance1"/>
                    <parameter key="max_depth" value="operator.DecisionTree.parameter.maximal_depth"/>
                    <parameter key="max_leaf_size" value="operator.DecisionTree.parameter.minimal_leaf_size"/>
                    <parameter key="confidence" value="operator.DecisionTree.parameter.confidence"/>
                  </list>
                  <parameter key="persistent" value="true"/>
              </operator>
          </operator>
      </operator>

    </process>
    and here is the complete log file:

    # Generated by ProcessLog[com.rapidminer.operator.visualization.ProcessLogOperator]
    # iteration time deviation accuracy max_depth max_leaf_size confidence
    10.0 322.0 0.05129744389973656 0.7932203389830508 98.0 13.0 0.34154054826704505
    10.0 316.0 0.05507822647211116 0.7932203389830507 63.0 78.0 0.32651644400635255
    10.0 315.0 0.0392401250942013 0.793220338983051 36.0 1.0 0.397380892497813
    10.0 344.0 0.0726246958834861 0.7932203389830509 90.0 52.0 0.40415961507086146
    10.0 319.0 0.03774755500223652 0.7932203389830509 76.0 41.0 0.27277836592326715
    10.0 321.0 0.055078226472110144 0.7932203389830507 2.0 15.0 0.03868159703154637
    10.0 323.0 0.06287198979997247 0.7932203389830507 5.0 70.0 0.12025785562389255
    10.0 323.0 0.026037782196170964 0.7932203389830507 48.0 19.0 0.4842464140080626
    10.0 322.0 0.033728387698533355 0.7932203389830508 62.0 86.0 0.28231036291700745
    10.0 326.0 0.049588945214671463 0.7932203389830508 76.0 41.0 0.2588474520496027
    10.0 354.0 0.04900621116881689 0.7932203389830509 90.0 52.0 0.394739905705027
    10.0 329.0 0.05240550791098398 0.7932203389830509 2.0 15.0 0.024606579573607863
    10.0 331.0 0.04660246469446598 0.7932203389830509 76.0 41.0 0.2749646967881053
    10.0 326.0 0.06054430881183808 0.7932203389830508 63.0 78.0 0.03277810211166991
    10.0 327.0 0.03774755500223652 0.7932203389830509 2.0 15.0 0.33223030703745715
    10.0 332.0 0.04598189819068104 0.7932203389830508 2.0 15.0 0.05108435729123231
    10.0 333.0 0.05611167917710849 0.7932203389830507 36.0 1.0 0.4118684184539058
    10.0 340.0 0.03924012509420413 0.7932203389830509 76.0 41.0 0.27811571436906546
    10.0 334.0 0.046602464694467174 0.7932203389830508 2.0 15.0 0.03330102480728759
    10.0 330.0 0.055078226472110144 0.7932203389830508 2.0 15.0 0.04044172165402028
    10.0 337.0 0.039965512279838404 0.7932203389830507 90.0 52.0 0.40559465185597704
    10.0 337.0 0.04137815462960765 0.7932203389830508 63.0 78.0 0.32377306309346343
    10.0 338.0 0.039965512279837016 0.7932203389830508 2.0 15.0 0.06783963315527239
    10.0 339.0 0.042744136314977955 0.7932203389830509 2.0 15.0 0.055966407234910774
    10.0 333.0 0.03012947260107371 0.7932203389830507 76.0 41.0 0.27478141583203897
    10.0 333.0 0.03697868513435921 0.7932203389830508 76.0 41.0 0.2709776940166765
    10.0 351.0 0.03286562615197772 0.7932203389830507 2.0 15.0 0.04512058551300815
    10.0 343.0 0.06195141316100628 0.793220338983051 76.0 41.0 0.28224938856896803
    10.0 354.0 0.05712643914017982 0.7932203389830508 36.0 1.0 0.3985242094663377
    10.0 344.0 0.04900621116881689 0.7932203389830508 76.0 41.0 0.27277836592326715
    10.0 337.0 0.038501073530848126 0.7932203389830509 90.0 52.0 0.40415961507086146
    10.0 346.0 0.0651165176688085 0.7932203389830509 2.0 15.0 0.03868159703154637
    10.0 347.0 0.05073433744778388 0.7932203389830508 76.0 41.0 0.27277836592326715
    10.0 348.0 0.04471493545177314 0.7932203389830508 63.0 78.0 0.03868159703154637
    10.0 376.0 0.03539086952172904 0.7932203389830509 2.0 15.0 0.32651644400635255
    10.0 359.0 0.036193485600110376 0.7932203389830507 2.0 15.0 0.03868159703154637
    10.0 376.0 0.04067796610169709 0.7932203389830508 36.0 1.0 0.397380892497813
    10.0 370.0 0.061016949152542 0.7932203389830509 76.0 41.0 0.27277836592326715
    10.0 368.0 0.07496048944913757 0.7932203389830509 2.0 15.0 0.03868159703154637
    10.0 367.0 0.034569623820971486 0.7932203389830507 36.0 1.0 0.2850651570329497
    10.0 371.0 0.05295084526038379 0.7932203389830509 76.0 41.0 0.40325282749551083
    10.0 362.0 0.04341101177920808 0.7932203389830508 76.0 41.0 0.2953841219126293
    10.0 415.0 0.04206669032539301 0.7932203389830508 76.0 41.0 0.27532383744770006
    10.0 450.0 0.04471493545177314 0.7932203389830509 36.0 1.0 0.3866757225955784
    10.0 358.0 0.04274413631498055 0.7932203389830507 2.0 15.0 0.03845641301147732
    10.0 372.0 0.03456962382096667 0.7932203389830509 2.0 1.0 0.40548652887913794
    10.0 346.0 0.04721487551588266 0.7932203389830507 36.0 15.0 0.03225434093850507
    10.0 361.0 0.018254795956391408 0.7932203389830508 2.0 15.0 0.04556657107596725
    10.0 360.0 0.05762711864406968 0.7932203389830507 2.0 15.0 0.07011796273391388
    10.0 377.0 0.061951413161007184 0.7932203389830509 36.0 1.0 0.40087800008364416
    10.0 362.0 0.04471493545177314 0.7932203389830509 76.0 41.0 0.27915934844746276
    10.0 378.0 0.06511651766880934 0.7932203389830508 76.0 41.0 0.28168690337376767
    10.0 349.0 0.03774755500223946 0.7932203389830508 2.0 15.0 0.07293158627660976
    10.0 381.0 0.05712643914017982 0.7932203389830508 36.0 1.0 0.3947367937506493
    10.0 366.0 0.035390869521732184 0.7932203389830508 2.0 15.0 0.026593489652135923
    10.0 380.0 0.0591037144886865 0.7932203389830509 2.0 15.0 0.032553155159174216
    10.0 368.0 0.0534906231798622 0.7932203389830509 76.0 41.0 0.28000872950589595
    10.0 366.0 0.05958778247880523 0.7932203389830508 2.0 15.0 0.03494975107057588
    10.0 353.0 0.06467384416386161 0.7932203389830508 36.0 1.0 0.4276158956902939
    10.0 386.0 0.03106830979631423 0.7932203389830508 36.0 1.0 0.27277836592326715
    10.0 371.0 0.0524055079109861 0.7932203389830507 76.0 41.0 0.397380892497813
    10.0 388.0 0.04406779661017154 0.7932203389830508 76.0 41.0 0.27277836592326715
    10.0 372.0 0.041378154629604966 0.7932203389830509 76.0 41.0 0.27277836592326715
    10.0 372.0 0.04598189819068104 0.7932203389830508 36.0 1.0 0.397380892497813
    10.0 381.0 0.04721487551588031 0.7932203389830507 2.0 15.0 0.03868159703154637
    10.0 393.0 0.07061920900338971 0.7932203389830508 2.0 1.0 0.397380892497813
    10.0 376.0 0.055597354124937784 0.7932203389830509 36.0 15.0 0.03868159703154637
    10.0 395.0 0.049006211168820285 0.7932203389830507 2.0 15.0 0.03868159703154637
    10.0 357.0 0.0495889452146737 0.7932203389830507 2.0 15.0 0.06783963315527239
    10.0 376.0 0.036193485600105775 0.7932203389830509 76.0 15.0 0.037015432195162346
    10.0 378.0 0.024910065180848328 0.7932203389830508 2.0 41.0 0.2807067926588812
    10.0 398.0 0.05762711864406775 0.7932203389830508 76.0 41.0 0.2894590092601272
    10.0 382.0 0.060067949649726594 0.7932203389830509 76.0 41.0 0.2879003972663676
    10.0 396.0 0.042066690325395645 0.7932203389830508 76.0 41.0 0.02664175488889414
    10.0 361.0 0.0281580469929397 0.7932203389830509 2.0 15.0 0.3007605640107514
    10.0 382.0 0.05559735412493978 0.7932203389830507 36.0 1.0 0.39735325254783843
    10.0 382.0 0.033728387698535 0.7932203389830507 36.0 1.0 0.4257667803880479
    10.0 384.0 0.0561116791771075 0.7932203389830509 36.0 1.0 0.049004769840074784
    10.0 384.0 0.061016949152542 0.7932203389830509 36.0 15.0 0.3948290429670914
    10.0 364.0 0.04274413631498185 0.7932203389830507 2.0 15.0 0.03672310007521456
    10.0 364.0 0.037747555002240925 0.7932203389830508 36.0 1.0 0.39710883071270425
    10.0 386.0 0.03850107353085101 0.7932203389830508 36.0 15.0 0.05950635065611809
    10.0 387.0 0.03774755500223799 0.7932203389830508 76.0 41.0 0.29086202403328454
    10.0 388.0 0.055078226472110144 0.7932203389830508 76.0 41.0 0.25978959222770964
    10.0 425.0 0.0440677966101665 0.793220338983051 36.0 1.0 0.4073614132304591
    10.0 372.0 0.01976593862659389 0.7932203389830509 36.0 1.0 0.4360241019919045
    10.0 420.0 0.04341101177920552 0.7932203389830509 76.0 41.0 0.2650590211401542
    10.0 399.0 0.04958894521467034 0.7932203389830509 2.0 15.0 0.03541818035342996
    10.0 400.0 0.042744136314977955 0.7932203389830509 76.0 41.0 0.2712751604859657
    10.0 400.0 0.03539086952173375 0.7932203389830507 76.0 15.0 0.03868159703154637
    10.0 381.0 0.06980088231177763 0.7932203389830507 2.0 41.0 0.27915934844746276
    10.0 387.0 0.02491006518084387 0.7932203389830509 76.0 41.0 0.27915934844746276
    10.0 395.0 0.04900621116881802 0.7932203389830508 76.0 41.0 0.27915934844746276
    10.0 396.0 0.039965512279838404 0.7932203389830507 76.0 41.0 0.03868159703154637
    10.0 396.0 0.061485956431246415 0.7932203389830507 2.0 15.0 0.27915934844746276
    10.0 419.0 0.036193485600105775 0.7932203389830509 36.0 1.0 0.397380892497813
    10.0 371.0 0.05611167917710948 0.7932203389830508 36.0 1.0 0.4276158956902939
    10.0 400.0 0.03924012509420555 0.7932203389830508 36.0 1.0 0.03868159703154637
    10.0 399.0 0.05507822647210813 0.7932203389830509 36.0 15.0 0.397380892497813
    So, you can see that the accuracy is still constant. All parameters to be optimized seem also to be correctly
    evaluated. Is this behavior  really OK or do you have any other
    suggestions what might went wrong?

    Regards,
    Chris
  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi Chris,
    first at all:
    What you did post here is not your modell, its your process. Models are the results of learning algorithms like trees, rulesets and so on.

    The most basic model is called "default model". It means, that it always predicts the same class, which is the most frequent class in the trainingset, independent from the values of the current example.
    All learning algorithms might produce models equivalent to this default models. For example this is the case if a tree only consists of one node. Another example would be an SVM hyperplane laid far beyond the data itself.

    The learning process and hence the building process of the models are often controlled by parameters. Thats why you are optimizing this parameters, because you want to optimize the learning performance.
    BUT not every distinct value will lead to a different model. Many values create the same result. So if the model building will change if the parameter value a exceeds 6, 7, 8, 1000 or greater have not to change it again.
    For example the maximal depth of a tree is not important if the generation never reaches this limit. So 100, 100000, or 782379821412 will not result in a different tree than 15 if the maximal depth ever generate during leraning is 15.

    Hope this will get you some ideas.

    Greetings,
      Sebastian
  • chris_mlchris_ml Member Posts: 17 Maven
    Hi Sebastian,

    thank you for your detailed answer.This was very helpful. I didn't know that the "maximal" parameters
    are just upper bounds for the parameter values. My assumption was that RapidMiner is always trying to
    learn a model that tries to use the given values, i.e. RM learns a model with the maximal (or minimal)
    parameter values specified.

    So, parameter optimization requires some deeper knowledge on the model to be optimized, right? I mean
    when "arbitrary" parameter values are evaluated, it might happen, as in my case, that many combinations
    don't make sense since they create the same model, thus their evaluation is wasted time. Can you give me
    any advices how parameter optimization should be effectively performed? Does it make sense to use just
    small values for "maximal" parameters since they will more likely lead to different results? Or was my approach
    OK and I can infer from the results that the learned DecisionTrees is not sensitive to my learning data, i.e. it
    will mostly end up in the same accuracy?

    Regards,
    Chris
Sign In or Register to comment.