"LibSVM: how to calculate MSE from confidence?"

doc_plasmondoc_plasmon Member Posts: 2 Contributor I
edited June 2019 in Help
I'm using the LibSVMlearner process for classification (C-SVM), and I'd like to make use of the likelihoods determined via Platt scaling.  I assume these are output via the "calculate-confidences" option.  I'd like to calculate the mean squared error between these likelihoods and the target.  I'd then like to use these as a performance metric, perhaps via the Data2Performance process?  Can someone give advice on 1) how to calculate the MSE when the confidences are calculated, and 2) how to use the result in Data2Performance?

Many thanks.


  • cantabcantab Member Posts: 6 Contributor II
    The operator ClassificationPerformance will tell you the MSE etc. of confidences, i.e. of real-valued predictions. 

    This is different from the operator BinominalClassificationPerformance, which will only evaluate 0/1 predictions.
  • Legacy UserLegacy User Member Posts: 0 Newbie
    Hmm I don't see a confidence option in the ClassficationPerformance operator. I'm trying to perform Platt Scaling
    on the output of the FastLargeMargin SVM, then measure the MSE between the true label and the probability of that label. Maybe I need to make sure that previous operators are providing real valued predictions?
  • cantabcantab Member Posts: 6 Contributor II
    My impression is that using confidences is automatic.  The documentation is silent about this, but it does say  "The cross-entropy of a classifier, defined as the sum over the logarithms of the true label’s confidences divided by the number of examples."

    For MSE my impression is that real-valued predictions are used also.  If they aren't, then MSE doesn't make much sense.

    Note that if you use the Logistic Regression option of FastLargeMargin, then Platt scaling is not necessary.
  • doc_plasmondoc_plasmon Member Posts: 2 Contributor I
    Using the PlattScaling process after the SVM learner seems to do the trick.  Then the "squared error" setting of ClassificationPerformance outputs the MSE.  Thanks!
Sign In or Register to comment.