🎉 🎉 RAPIDMINER 9.10 IS OUT!!! 🎉🎉

Download the latest version helping analytics teams accelerate time-to-value for streaming and IIOT use cases.

CLICK HERE TO DOWNLOAD

"How are L pos and L neg set in an SVM?"

eraskineraskin Member Posts: 1 Contributor I
edited June 2019 in Help
Noob here.  I am working on a basic classifcation problem (responders vs non-responders), where the cost of missing a responder is much higher than the cost of including a non-responder.  Specifically, the average sale is $115 and the average cost to mail somebody is $0.68.  We want to bias in favor of maximizing sales over minimizing cost to mail.

I can't find any documentation anywhere that shows how the "L pos" and "L neg" settings in an SVM are actually set.  Are they numbers between 0 and 1?  Are they relative weights for mis-classifying a positive "L pos" and a negative "L neg"?  In my case, I would interpret that as setting "L pos" much higher than "L neg", since I want to ensure that we avoid as many false negatives as possible.

Can somebody help me understand how this works?

Thanks in advance...

  Eric
Tagged:

Answers

  • lel5287lel5287 Member Posts: 4 Contributor I
    I know this is a few months old, but I'm working on a very similar problem, so if anyone has any insight on this, that would be greatly appreciated.

    Additionally, I'm still exploring classification methods for what I'm working on.  Is it possible to set similar loss criteria in other classification methods?  (False positives aren't too big of a deal, but false negatives are very costly--also, the test set includes very few responders compared to the number of non-responders, so a lot of methods that just look to improve accuracy without weighing false positives against false negatives classify almost everyone as a non-responder.)
  • mschmitzmschmitz Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,055  RM Data Scientist
    Hi,

    i don't know what it is but i got eager to find it out.

    So apprently it is used in /src/main/java/com/rapidminer/operator/learner/functions/kernel/jmysvm/svm/SVM.java

    I figured out there that the balance cost option sets the same values.
    				double generalCpos = paramOperator.getParameterAsDouble(JMySVMLearner.PARAMETER_L_POS);
    double generalCneg = paramOperator.getParameterAsDouble(JMySVMLearner.PARAMETER_L_NEG);
    if (paramOperator.getParameterAsBoolean(JMySVMLearner.PARAMETER_BALANCE_COST)) {
    generalCpos *= the_examples.count_examples()
    / (2.0d * (the_examples.count_examples() - the_examples.count_pos_examples()));
    generalCneg *= ((the_examples.count_examples()) / (2.0d * the_examples.count_pos_examples()));
    }
    for (int i = 0; i < cPos.length; i++) {
    cPos = generalCpos;
    cNeg = generalCneg;
    So i did a quick test (process XML below). I classified golf with and without balance cost. Further i added weights to ensure class balance this way.

    Weights w/o balance cost and w/o class balancing

    Temperature -0.32830007919309123
    Outlook = sunny -0.2990901852444362
    Humidity -0.2759359389650189
    Wind = true -0.23094010767584963
    Outlook = rain 0.0
    Wind = false 0.17320508075688723
    Outlook = overcast 0.35248116362113585
    w balance cost but w/o class balancing

    Outlook = sunny -0.33391948463996746
    Humidity -0.2776748983378322
    Wind = true -0.25278262899833226
    Outlook = rain -0.12521980673998773
    Temperature -0.060810818269701084
    Wind = false 0.18958697174874925
    Outlook = overcast 0.5411008440732543
    and finallyw/ scaling but w/o balancing

    Outlook = sunny -0.33391948463996746
    Humidity -0.2776748983378322
    Wind = true -0.25278262899833226
    Outlook = rain -0.12521980673998773
    Temperature -0.060810818269701084
    Wind = false 0.18958697174874925
    Outlook = overcast 0.5411008440732543
    So apprently the balancing here does the same thing than balancing via weights. Thus I expect LPos and LNeg to be class_weights.

    Cheers,
    Martin

    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="6.5.002">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="6.5.002" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="retrieve" compatibility="6.5.002" expanded="true" height="60" name="Retrieve Golf" width="90" x="45" y="120">
            <parameter key="repository_entry" value="//Samples/data/Golf"/>
          </operator>
          <operator activated="true" class="nominal_to_numerical" compatibility="6.5.002" expanded="true" height="94" name="Nominal to Numerical" width="90" x="179" y="120">
            <list key="comparison_groups"/>
          </operator>
          <operator activated="true" class="generate_weight_stratification" compatibility="6.5.002" expanded="true" height="76" name="Generate Weight (Stratification)" width="90" x="313" y="120">
            <parameter key="total_weight" value="14.0"/>
          </operator>
          <operator activated="true" class="support_vector_machine" compatibility="6.5.002" expanded="true" height="112" name="SVM" width="90" x="514" y="120"/>
          <connect from_op="Retrieve Golf" from_port="output" to_op="Nominal to Numerical" to_port="example set input"/>
          <connect from_op="Nominal to Numerical" from_port="example set output" to_op="Generate Weight (Stratification)" to_port="example set input"/>
          <connect from_op="Generate Weight (Stratification)" from_port="example set output" to_op="SVM" to_port="training set"/>
          <connect from_op="SVM" from_port="weights" to_port="result 1"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="0"/>
          <portSpacing port="sink_result 2" spacing="0"/>
        </process>
      </operator>
    </process>
    - Head of Data Science Services at RapidMiner -
    Dortmund, Germany
  • JEdwardJEdward RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 574   Unicorn
    For other classifiers (& SVM too) another way to tackle this with different costs is to use cost based modelling. 
    Here's an example on how it works.  Please note, I chose to use Naive Bayes because it demonstrated the extemes well chosing to send to the entire dataset to 'maximise profit' (Have the least cost)

    You can chose to either use the Meta Cost operator or use the Performance (Costs) operators separately.
    Meta cost builds several models to favour the cost.
    Performance (Costs) tells you the expected cost of the model.  (I tend to combine this one with an Optimise operator and then build my model parameters with the ones that decrease the cost the most). 
    <?xml version="1.0" encoding="UTF-8" standalone="no"?>
    <process version="7.0.000">
      <context>
        <input/>
        <output/>
        <macros/>
      </context>
      <operator activated="true" class="process" compatibility="6.0.002" expanded="true" name="Process">
        <process expanded="true">
          <operator activated="true" class="generate_direct_mailing_data" compatibility="7.0.000" expanded="true" height="68" name="Generate Direct Mailing Data" width="90" x="45" y="34">
            <parameter key="number_examples" value="100000"/>
          </operator>
          <operator activated="true" class="multiply" compatibility="7.0.000" expanded="true" height="124" name="Multiply" width="90" x="112" y="136"/>
          <operator activated="true" class="split_validation" compatibility="7.0.000" expanded="true" height="124" name="Validation" width="90" x="246" y="34">
            <process expanded="true">
              <operator activated="true" class="metacost" compatibility="7.0.000" expanded="true" height="82" name="MetaCost" width="90" x="45" y="34">
                <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
                <process expanded="true">
                  <operator activated="true" class="naive_bayes" compatibility="7.0.000" expanded="true" height="82" name="Naive Bayes" width="90" x="179" y="136"/>
                  <connect from_port="training set" to_op="Naive Bayes" to_port="training set"/>
                  <connect from_op="Naive Bayes" from_port="model" to_port="model"/>
                  <portSpacing port="source_training set" spacing="0"/>
                  <portSpacing port="sink_model" spacing="0"/>
                </process>
              </operator>
              <connect from_port="training" to_op="MetaCost" to_port="training set"/>
              <connect from_op="MetaCost" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="7.0.000" expanded="true" height="82" name="Apply Model" width="90" x="45" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_costs" compatibility="7.0.000" expanded="true" height="82" name="Performance (Cost Meta)" width="90" x="112" y="187">
                <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
                <enumeration key="class_order_definition"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="7.0.000" expanded="true" height="82" name="Performance" width="90" x="179" y="30">
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="Apply Model" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model" to_port="unlabelled data"/>
              <connect from_op="Apply Model" from_port="labelled data" to_op="Performance (Cost Meta)" to_port="example set"/>
              <connect from_op="Performance (Cost Meta)" from_port="example set" to_op="Performance" to_port="labelled data"/>
              <connect from_op="Performance (Cost Meta)" from_port="performance" to_op="Performance" to_port="performance"/>
              <connect from_op="Performance" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="create_lift_chart" compatibility="7.0.000" expanded="true" height="103" name="Create Lift Chart" width="90" x="447" y="289">
            <parameter key="target_class" value="response"/>
          </operator>
          <operator activated="true" class="split_validation" compatibility="7.0.000" expanded="true" height="124" name="Validation (2)" width="90" x="246" y="187">
            <process expanded="true">
              <operator activated="true" class="naive_bayes" compatibility="7.0.000" expanded="true" height="82" name="Naive Bayes (2)" width="90" x="45" y="34"/>
              <connect from_port="training" to_op="Naive Bayes (2)" to_port="training set"/>
              <connect from_op="Naive Bayes (2)" from_port="model" to_port="model"/>
              <portSpacing port="source_training" spacing="0"/>
              <portSpacing port="sink_model" spacing="0"/>
              <portSpacing port="sink_through 1" spacing="0"/>
            </process>
            <process expanded="true">
              <operator activated="true" class="apply_model" compatibility="7.0.000" expanded="true" height="82" name="Apply Model (2)" width="90" x="45" y="30">
                <list key="application_parameters"/>
              </operator>
              <operator activated="true" class="performance_costs" compatibility="7.0.000" expanded="true" height="82" name="Performance (Cost Without Meta)" width="90" x="45" y="238">
                <parameter key="cost_matrix" value="[0.0 115.0;0.68 -115.0]"/>
                <enumeration key="class_order_definition"/>
              </operator>
              <operator activated="true" class="performance_classification" compatibility="7.0.000" expanded="true" height="82" name="Performance (2)" width="90" x="179" y="30">
                <list key="class_weights"/>
              </operator>
              <connect from_port="model" to_op="Apply Model (2)" to_port="model"/>
              <connect from_port="test set" to_op="Apply Model (2)" to_port="unlabelled data"/>
              <connect from_op="Apply Model (2)" from_port="labelled data" to_op="Performance (Cost Without Meta)" to_port="example set"/>
              <connect from_op="Performance (Cost Without Meta)" from_port="example set" to_op="Performance (2)" to_port="labelled data"/>
              <connect from_op="Performance (Cost Without Meta)" from_port="performance" to_op="Performance (2)" to_port="performance"/>
              <connect from_op="Performance (2)" from_port="performance" to_port="averagable 1"/>
              <portSpacing port="source_model" spacing="0"/>
              <portSpacing port="source_test set" spacing="0"/>
              <portSpacing port="source_through 1" spacing="0"/>
              <portSpacing port="sink_averagable 1" spacing="0"/>
              <portSpacing port="sink_averagable 2" spacing="0"/>
            </process>
          </operator>
          <operator activated="true" class="subprocess" compatibility="7.0.000" expanded="true" height="124" name="Just playing around" width="90" x="380" y="85">
            <process expanded="true">
              <operator activated="true" class="performance_to_data" compatibility="7.0.000" expanded="true" height="82" name="Performance to Data" width="90" x="45" y="34"/>
              <operator activated="true" class="performance_to_data" compatibility="7.0.000" expanded="true" height="82" name="Performance to Data (2)" width="90" x="45" y="238"/>
              <operator activated="true" class="extract_macro" compatibility="7.0.000" expanded="true" height="68" name="Extract Macro" width="90" x="179" y="136">
                <parameter key="macro" value="CostWith"/>
                <parameter key="macro_type" value="data_value"/>
                <parameter key="attribute_name" value="Value"/>
                <parameter key="example_index" value="1"/>
                <list key="additional_macros"/>
              </operator>
              <operator activated="true" class="extract_macro" compatibility="7.0.000" expanded="true" height="68" name="Extract Macro (2)" width="90" x="179" y="238">
                <parameter key="macro" value="CostWithout"/>
                <parameter key="macro_type" value="data_value"/>
                <parameter key="attribute_name" value="Value"/>
                <parameter key="example_index" value="1"/>
                <list key="additional_macros"/>
              </operator>
              <operator activated="true" class="generate_data_user_specification" compatibility="7.0.000" expanded="true" height="68" name="Gen Data" width="90" x="313" y="289">
                <list key="attribute_values">
                  <parameter key="CostDifference" value="parse(%{CostWith})-parse(%{CostWithout})"/>
                </list>
                <list key="set_additional_roles"/>
              </operator>
              <operator activated="true" class="format_numbers" compatibility="7.0.000" expanded="true" height="82" name="Saving by using meta model" width="90" x="447" y="238">
                <parameter key="format_type" value="currency"/>
                <parameter key="locale" value="English (United Kingdom)"/>
                <parameter key="use_grouping" value="true"/>
              </operator>
              <connect from_port="in 1" to_op="Performance to Data" to_port="performance vector"/>
              <connect from_port="in 2" to_op="Performance to Data (2)" to_port="performance vector"/>
              <connect from_op="Performance to Data" from_port="example set" to_op="Extract Macro" to_port="example set"/>
              <connect from_op="Performance to Data" from_port="performance vector" to_port="out 2"/>
              <connect from_op="Performance to Data (2)" from_port="example set" to_op="Extract Macro (2)" to_port="example set"/>
              <connect from_op="Performance to Data (2)" from_port="performance vector" to_port="out 3"/>
              <connect from_op="Gen Data" from_port="output" to_op="Saving by using meta model" to_port="example set input"/>
              <connect from_op="Saving by using meta model" from_port="example set output" to_port="out 1"/>
              <portSpacing port="source_in 1" spacing="0"/>
              <portSpacing port="source_in 2" spacing="0"/>
              <portSpacing port="source_in 3" spacing="0"/>
              <portSpacing port="sink_out 1" spacing="0"/>
              <portSpacing port="sink_out 2" spacing="0"/>
              <portSpacing port="sink_out 3" spacing="0"/>
              <portSpacing port="sink_out 4" spacing="0"/>
            </process>
          </operator>
          <connect from_op="Generate Direct Mailing Data" from_port="output" to_op="Multiply" to_port="input"/>
          <connect from_op="Multiply" from_port="output 1" to_op="Validation" to_port="training"/>
          <connect from_op="Multiply" from_port="output 2" to_op="Validation (2)" to_port="training"/>
          <connect from_op="Multiply" from_port="output 3" to_op="Create Lift Chart" to_port="example set"/>
          <connect from_op="Validation" from_port="model" to_op="Create Lift Chart" to_port="model"/>
          <connect from_op="Validation" from_port="averagable 1" to_op="Just playing around" to_port="in 1"/>
          <connect from_op="Create Lift Chart" from_port="model" to_port="result 5"/>
          <connect from_op="Create Lift Chart" from_port="lift pareto chart" to_port="result 6"/>
          <connect from_op="Validation (2)" from_port="model" to_port="result 1"/>
          <connect from_op="Validation (2)" from_port="averagable 1" to_op="Just playing around" to_port="in 2"/>
          <connect from_op="Just playing around" from_port="out 1" to_port="result 2"/>
          <connect from_op="Just playing around" from_port="out 2" to_port="result 3"/>
          <connect from_op="Just playing around" from_port="out 3" to_port="result 4"/>
          <portSpacing port="source_input 1" spacing="0"/>
          <portSpacing port="sink_result 1" spacing="84"/>
          <portSpacing port="sink_result 2" spacing="0"/>
          <portSpacing port="sink_result 3" spacing="0"/>
          <portSpacing port="sink_result 4" spacing="0"/>
          <portSpacing port="sink_result 5" spacing="0"/>
          <portSpacing port="sink_result 6" spacing="0"/>
          <portSpacing port="sink_result 7" spacing="0"/>
        </process>
      </operator>
    </process>
  • mschmitzmschmitz Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,055  RM Data Scientist
    While i totally support you i would like to add, that in a best case scenario you can give each example (e.g. customer) a cost (or revenue). Than you can aggregate on this and optimize on the costs. This is very similar but even more detailed. I think this is a better way - if the data is available.

    ~Martin
    - Head of Data Science Services at RapidMiner -
    Dortmund, Germany
Sign In or Register to comment.