"LibSVM: how to calculate MSE from confidence?"

doc_plasmon

I'm using the LibSVMlearner process for classification (C-SVM), and I'd like to make use of the likelihoods determined via Platt scaling.  I assume these are output via the "calculate-confidences" option.  I'd like to calculate the mean squared error between these likelihoods and the target.  I'd then like to use these as a performance metric, perhaps via the Data2Performance process?  Can someone give advice on 1) how to calculate the MSE when the confidences are calculated, and 2) how to use the result in Data2Performance?

Many thanks.

cantab

The operator ClassificationPerformance will tell you the MSE etc. of confidences, i.e. of real-valued predictions. 

This is different from the operator BinominalClassificationPerformance, which will only evaluate 0/1 predictions.

Legacy User

Hmm I don't see a confidence option in the ClassficationPerformance operator. I'm trying to perform Platt Scaling
on the output of the FastLargeMargin SVM, then measure the MSE between the true label and the probability of that label. Maybe I need to make sure that previous operators are providing real valued predictions?

cantab

My impression is that using confidences is automatic.  The documentation is silent about this, but it does say  "The cross-entropy of a classifier, defined as the sum over the logarithms of the true label’s confidences divided by the number of examples."

For MSE my impression is that real-valued predictions are used also.  If they aren't, then MSE doesn't make much sense.

Note that if you use the Logistic Regression option of FastLargeMargin, then Platt scaling is not necessary.

doc_plasmon

Using the PlattScaling process after the SVM learner seems to do the trick.  Then the "squared error" setting of ClassificationPerformance outputs the MSE.  Thanks!