Optimize model at certain confidence levels

Frequent User

Optimize model at certain confidence levels

Hallo community,


I am building a classification model with two classes. The goal is to make predictions on a class of an example, but only when the confidence level of the model passes a threshold level. If the confidence threshold is not reached, no class should be predicted --> "Drop Uncertain Predictions Operator".


My question now is: is there a way to automatically optimize the model for predictions on examples above the threshold? I.e. I would like to make sure that the accuracy of the model for predictions above the threshold is as high as possible, but don't care much about the classification error below the threshold. Using the optimize operators seems to only optimize for accuracy of the entire example set. 


Also and related to the above: how would such an approach work with ensembles and algorithms like Gradient Boosted Trees for which the confidence levels are the product (?) of confidence levels of the individual models/trees?


Thank you very much for any hints.

RM Staff
Accepted by topic author FBT
‎06-15-2017 11:14 PM

Re: Optimize model at certain confidence levels



try Create Threhsold and apply threshold or Find Threshold/Find Recalll operators. You can chain these to gether with an optimize. The question is the correct performance measure.



Head of Data Science Services at RapidMiner
Frequent User

Re: Optimize model at certain confidence levels

Thank you very much, that was helpful!