Optimize model at certain confidence levels

FBTFBT Member Posts: 106 Unicorn
edited November 2018 in Help

Hallo community,

 

I am building a classification model with two classes. The goal is to make predictions on a class of an example, but only when the confidence level of the model passes a threshold level. If the confidence threshold is not reached, no class should be predicted --> "Drop Uncertain Predictions Operator".

 

My question now is: is there a way to automatically optimize the model for predictions on examples above the threshold? I.e. I would like to make sure that the accuracy of the model for predictions above the threshold is as high as possible, but don't care much about the classification error below the threshold. Using the optimize operators seems to only optimize for accuracy of the entire example set. 

 

Also and related to the above: how would such an approach work with ensembles and algorithms like Gradient Boosted Trees for which the confidence levels are the product (?) of confidence levels of the individual models/trees?

 

Thank you very much for any hints.

Best Answer

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,507 RM Data Scientist
    Solution Accepted

    Hi,

     

    try Create Threhsold and apply threshold or Find Threshold/Find Recalll operators. You can chain these to gether with an optimize. The question is the correct performance measure.

     

    ~Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany

Answers

  • FBTFBT Member Posts: 106 Unicorn

    Thank you very much, that was helpful!

Sign In or Register to comment.