Optimize model at certain confidence levels
I am building a classification model with two classes. The goal is to make predictions on a class of an example, but only when the confidence level of the model passes a threshold level. If the confidence threshold is not reached, no class should be predicted --> "Drop Uncertain Predictions Operator".
My question now is: is there a way to automatically optimize the model for predictions on examples above the threshold? I.e. I would like to make sure that the accuracy of the model for predictions above the threshold is as high as possible, but don't care much about the classification error below the threshold. Using the optimize operators seems to only optimize for accuracy of the entire example set.
Also and related to the above: how would such an approach work with ensembles and algorithms like Gradient Boosted Trees for which the confidence levels are the product (?) of confidence levels of the individual models/trees?
Thank you very much for any hints.