🦉 🎤   RapidMiner Wisdom 2020 - CALL FOR SPEAKERS   🦉 🎤

We are inviting all community members to submit proposals to speak at Wisdom 2020 in Boston.


Whether it's a cool RapidMiner trick or a use case implementation, we want to see what you have.
Form link is below and deadline for submissions is November 15. See you in Boston!

CLICK HERE TO GO TO ENTRY FORM

"[SOLVED] Using user set confidence for AUC performance, not working.."

mafern76mafern76 Member Posts: 45 Contributor II
edited June 11 in Help
Hi!

What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.

What I did was:

1. Rename confidence(class) to confidence(1).
2. Generate attribute confidence(0) = 0.
3. Set roles, confidence(1) to confidence_1 and 0 to 0.
4. Rename prediction(multiclass) to prediction(singleclass).
5. Map classofinterest to 1 and default to 0.
6. Transform the polynominal prediction to binominal.

This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.

Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.

On the other hand, the prediction label works because I can get accuracy.

I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?

Thanks a lot!

Answers

  • mafern76mafern76 Member Posts: 45 Contributor II
    Oh well an apple fell upon my head and I realized the problem was I was assuming AUC would use confidence_1, when it could perfectly use confidence_0.

    Solution: instead of (2) generate attribute confidence(0) = 0, generate it equal to 1- confidence_1...

    :)
Sign In or Register to comment.