confidences and probability

babalumababaluma Member Posts: 3 Contributor I
edited November 2018 in Help
I am trying to feed RapidMiner output (predictions and probabilities)  into an external program. I understand how to output confidence scores using ExampleSetWriter, but not probabilities. I found a thread in the old SourceForge forum that suggests using PlattScaling to convert confidence scores to probabilities. I looked around a little bit, but could not get this operator to get what I want. How should I go about doing this? How do I find and write the probability that a particular instance belongs to a class?

Thanks,
B

Answers

  • steffensteffen Member Posts: 347 Maven
    Hello babaluma

    For others: Here is the link to the mentioned sf-thread: click.

    Regarding your problem:
    I understand Platt Scaling this way:
    input: the ExampleSet the original model was created with and the created model
    output: the "corrected" model.

    Applying the corrected model to your test set results in real probabilities in the confidence column, i.e. confidence(c)=p(y=c|x=current Example).

    hope this was helpful

    Steffen

    PS: This topic is really interesting. Recently I stumbled upon a paper converting scores of multiple-class-classifiers, but unfortunately, I did not have the time to study it yet. Here is the link (PDF):Transforming Classifier Scores into Accurate Multiclass Probability Estimates
  • babalumababaluma Member Posts: 3 Contributor I
    thanks for the reply.

    while that seems to work, I am having trouble in getting probabilities from the training set with cross-validation. Where exactly should PlattScaling operator be placed in such a scenario?
    My cross validation step looks like the following:
    -XValidation
    --NaiveBayes
    --OperatorChain
    ---ModelApplier
    ---BinomialClassificationPerformance
    ---ExampleWriter
    ---ResultWriter
    -ModelWriter
    -PerformanceWriter

    It seems that PlattScaling should be performed after the learning step and before the ExampleWriter operator. However, by placing it before the ExampleWriter operator, ExampleSet is lost (PlattScaling takes ExampleSet and Model and returns Model). Placing it after the learning step, I somehow get an error in the PerformanceVector operator. The error is that "an object of PerformanceVector type is needed". Without PlattScaling, it works fine. I suppose if PlattScaling had the parameter "keep_example_set", this would work. I would appreciate any feedback.

    Thanks
    B


  • steffensteffen Member Posts: 347 Maven
    Hello again

    If you want to create a "platt-scaled" model for each training step, I suggest something like this: (simple copy and paste it to the xml-tab of rm-interface)
    <operator name="Root" class="Process" expanded="yes">
        <operator name="XValidation" class="XValidation" expanded="yes">
            <operator name="create_scaled_model" class="OperatorChain" expanded="yes">
                <operator name="NaiveBayes" class="NaiveBayes">
                    <parameter key="keep_example_set" value="true"/>
                </operator>
                <operator name="PlattScaling" class="PlattScaling">
                </operator>
            </operator>
            <operator name="OperatorChain" class="OperatorChain" expanded="yes">
                <operator name="ModelApplier" class="ModelApplier">
                    <list key="application_parameters">
                    </list>
                </operator>
                <operator name="BinominalClassificationPerformance" class="BinominalClassificationPerformance">
                </operator>
            </operator>
        </operator>
    </operator>
    Hope this is what you wanted to do :)

    greetings

    Steffen
  • babalumababaluma Member Posts: 3 Contributor I
    That seems to work. Thanks a lot.

    B
Sign In or Register to comment.