Options

# "[SOLVED] Using user set confidence for AUC performance, not working.."

Hi!

What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.

What I did was:

1. Rename confidence(class) to confidence(1).

2. Generate attribute confidence(0) = 0.

3. Set roles, confidence(1) to confidence_1 and 0 to 0.

4. Rename prediction(multiclass) to prediction(singleclass).

5. Map classofinterest to 1 and default to 0.

6. Transform the polynominal prediction to binominal.

This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.

Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.

On the other hand, the prediction label works because I can get accuracy.

I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?

Thanks a lot!

What I'm trying to do is use a polynominal model's confidence for a single class to plot AUC with a binominal label corresponding to that same class.

What I did was:

1. Rename confidence(class) to confidence(1).

2. Generate attribute confidence(0) = 0.

3. Set roles, confidence(1) to confidence_1 and 0 to 0.

4. Rename prediction(multiclass) to prediction(singleclass).

5. Map classofinterest to 1 and default to 0.

6. Transform the polynominal prediction to binominal.

This gives me the 3 prediction roles (prediction, confidence_0 and 1) that look exactly the same in metadata as the model specifically done for the binominal label.

Yet the AUC isn't using confidence, because AUC gives 0.5, optimistic 1 and pessimistic 0.

On the other hand, the prediction label works because I can get accuracy.

I guess there's some other meta-metadata that I can't see. Any way to achieve my goal?

Thanks a lot!

Tagged:

0

## Answers

45Contributor IISolution: instead of (2) generate attribute confidence(0) = 0, generate it equal to 1- confidence_1...