Confidence or Prediction Intervals

npapan69npapan69 Member Posts: 12 Contributor II
edited July 9 in Help
Dear All,
When reporting a performance metric (ie AUC) of a model that was trained on a single data set and tested on a hold-out set, what is the proper way to assess its variance? Calculating the confidence intervals of the AUC or the prediction intervals?
Many thanks
Nikos
Tagged:

Answers

  • varunm1varunm1 Member Posts: 724   Unicorn
    Hello @npapan69

    This is a bit tricky as they both are centered around the same value, but a prediction interval is wider than the confidence interval. So in case of error reduction or a better accurate prediction you can go with prediction intervals.
  • npapan69npapan69 Member Posts: 12 Contributor II
    Thanks Varunm1 for your answer. Is the prediction interval provided by any of the rapidminer operators?
  • varunm1varunm1 Member Posts: 724   Unicorn
    Hello @npapan69

    I am checking that, and I see no option in RM for that. I think it is a bit complicated to calculate.

    @mschmitz or @IngoRM any comments on this.

    Thanks
    mbs
Sign In or Register to comment.