"Creating SVM learning sets"

Answers

• land RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn

  June 2008

  Hi Gil,
  whats about sample your data? If I got you right, you don't want to use all your examples for learning. Perhabs you could a sampling algorithm for discarding that portion of data?
  
  Greetings,
    Sebastian

  0
• Legacy User Member Posts: 0 Newbie

  June 2008

  Hi Land,
  
  Thanks for answering so quicklly.
  
  You are right - I want to use only a small part of my set for learning, a much smaller part than what is offered by cross-validation. However, I don't know how to apply a sampling algorithm for a TextInput operator. Will it be possible for you (or anyone else, for that matter) to post an example how do do this?
  
  In an attemp to overcome this problem from a different direction, I wrote a java code that can go over all the documents of my dataset and randomly create subsets, which I intended to use as learning sets. I then wrote two simple experiments - one for creating a model based on the subsets I created, and another one that loads that model and applies it one the entore dataset.
  
  In order to make sure these two experiments function properly, I used half the dataset as the learning set (I thought this way I could compare my results to those pruduced by a 2-fold cross validation). Sadly, the results I got were much poorer than those produced by the cross-validation experiment - and I can't understand why that is the case. The XML of the two experiments is posted below - if I made a mistake, please help me understand what it is.
  
  If someone could help me solve even one of these two problems, I think it will be all I need.
  
  Thanks in advance,
  Gil
  
  The Two experiments:
  1) The learning phase - creating the SVM model:
  
  <?xml version="1.0" encoding="windows-1252"?>
  <process version="4.1">
  
    <operator name="Root" class="Process" expanded="yes">
            <operator name="TextInput" class="TextInput" expanded="no">
            <parameter key="create_text_visualizer" value="true"/>
            <list key="namespaces">
            </list>
            <parameter key="prune_below" value="3"/>
            <list key="texts">
              <parameter key="type1" value="D:\exp\type1_learnign_set"/>
              <parameter key="type2" value="D:\exp\type2_learnign_set"/>
            </list>
            <operator name="StringTokenizer" class="StringTokenizer">
            </operator>
            <operator name="EnglishStopwordFilter" class="EnglishStopwordFilter">
            </operator>
            <operator name="TokenLengthFilter" class="TokenLengthFilter">
                <parameter key="min_chars" value="3"/>
            </operator>
            <operator name="PorterStemmer" class="PorterStemmer">
            </operator>
            <operator name="TermNGramGenerator" class="TermNGramGenerator">
            </operator>
        </operator>
        <operator name="LibSVMLearner" class="LibSVMLearner">
            <list key="class_weights">
            </list>
            <parameter key="kernel_type" value="linear"/>
        </operator>
        <operator name="ModelWriter" class="ModelWriter">
            <parameter key="model_file" value="C:\Documents and Settings\Admin\Desktop\SVM_Model.mod"/>
        </operator>
    </operator>
  
  </process>
  
  2) The test phase - applying the model
  
  <?xml version="1.0" encoding="windows-1252"?>
  <process version="4.1">
  
    <operator name="Root" class="Process" expanded="yes">
        <operator name="TextInput" class="TextInput" expanded="no">
            <parameter key="create_text_visualizer" value="true"/>
            <list key="namespaces">
            </list>
            <parameter key="prune_below" value="3"/>
            <list key="texts">
              <parameter key="type1" value="D:\exp\type1_full_set"/>
              <parameter key="type2" value="D:\exp\type2_full_set"/>
            </list>
            <operator name="StringTokenizer" class="StringTokenizer">
            </operator>
            <operator name="EnglishStopwordFilter" class="EnglishStopwordFilter">
            </operator>
            <operator name="TokenLengthFilter" class="TokenLengthFilter">
                <parameter key="min_chars" value="3"/>
            </operator>
            <operator name="PorterStemmer" class="PorterStemmer">
            </operator>
            <operator name="TermNGramGenerator" class="TermNGramGenerator">
            </operator>
        </operator>
        <operator name="ModelLoader" class="ModelLoader">
            <parameter key="model_file" value="C:\Documents and Settings\Admin\Desktop\SVM_Model.mod"/>
        </operator>
        <operator name="ModelApplier" class="ModelApplier">
            <list key="application_parameters">
            </list>
        </operator>
        <operator name="BinominalClassificationPerformance" class="BinominalClassificationPerformance">
            <parameter key="AUC" value="true"/>
            <parameter key="f_measure" value="true"/>
        </operator>
    </operator>
  
  </process>
  
  
  

  0
• TobiasMalbrecht Moderator, Employee, Member Posts: 294 RM Product Management

  June 2008

  Hi Gil,
  
  well there is no direct and easy way to execute a cross validation but to use say only 10% of the examples for training and the other 90% for testing purposes. The easy-to-accomplish option you have is to simply use a sampling operator (e.g. [tt]StratifiedSampling[/tt]) before a cross validation. Therewith you may simply discard perhaps about 50% of your data and do a "normal" cross validation on the remaining 50%.
  
  Otherwise you can nearly simulate a kind of multiple validation by the following process:

  
  <operator name="Root" class="Process" expanded="yes">
      <operator name="NominalExampleSetGenerator" class="NominalExampleSetGenerator">
      </operator>
      <operator name="ParameterIteration" class="ParameterIteration" expanded="yes">
          <parameter key="keep_output"	value="true"/>
          <list key="parameters">
            <parameter key="SimpleValidation.local_random_seed"	value="1,2,3,4,5,6,7,8,9,10"/>
          </list>
          <operator name="SimpleValidation" class="SimpleValidation" expanded="yes">
              <parameter key="local_random_seed"	value="10"/>
              <operator name="NaiveBayes" class="NaiveBayes">
              </operator>
              <operator name="OperatorChain" class="OperatorChain" expanded="yes">
                  <operator name="ModelApplier" class="ModelApplier">
                      <list key="application_parameters">
                      </list>
                  </operator>
                  <operator name="Performance" class="Performance">
                  </operator>
              </operator>
          </operator>
      </operator>
      <operator name="AverageBuilder" class="AverageBuilder">
      </operator>
  </operator>
  

  Note, however, that the examples are not partitioned in the iterations.
  
  Regards,
  Tobias

  0