Different cost of misclassification for every single example – how to implement?

mcunimcuni Member Posts: 4 Contributor I
Hello all!
Please advise on the following problem.
[Dataset description]
Dataset contains about 10 thn examples with about 15-20 numeric attributes (not sure that all of them are useful, many are correlated, so this amount can be diminished to 7-10 attributes). Target (label) for these is binominal, let’s say it’s ‘One’ and ‘Two’.
By nature the data is quite noisy which means that it’s normal to talk about probability that case X belongs to class ‘One’ or ‘Two’. The normal expected probability is not expected to be higher than 0.75 (lower than 0.25) for 95% of cases.
The above is pretty typical. Now what I believe is special: for every case in dataset there is a confidence threshold.
Let’s say we have case X1, it’s actual class is ‘One’, threshold is T1. If we learn a model that predicts case X1 to be class ‘One’ with confidence P1, and P1 > T1 then Cost = 0. But if P1 < T1 then Cost = 1/T1.
For case X2 actual class is ‘Two’, threshold is T2. If a model predicts case X2 to be class ‘One’ with confidence P2, and P2 > T2 then Cost = 1/(1-T2). But if P2 < T2 then Cost = 0.
[Problem]
Normally I use some standard classifier like kNN which provides me with confidence output. Then I compare those confidences computed with my thresholds for every case (I do it with Excel). And this gives me final estimation for training set and for testing set. Then I compare performances and decide whether model is good or bad.
What I want is to make a model minimize the Cost function, which is sum of Costs for every case in set.
Does anyone know how It can be done with Rapidminer? I will appreciate any feedback.

Answers

  • aborgaborg Member Posts: 66 Contributor II
    Hi,

    Maybe the custom cost function for the labels would be a good option for you? The Performance (Costs) operator does that (you can specify the cost for each misclassification in a matrix), and there is also a meta-modelling operator called MetaCost.
    Hope this helps, gabor
  • mcunimcuni Member Posts: 4 Contributor I
    Gabor, thanks for your answer. It was useful as I'm new to rapidminer and I didn't know anything about meta-operators, now I do. Still the problem is that what MetaCost and couple of WEKA operators do is that they use the cost matrix *by classes*. And I need different cost per every single case, not class.
    Performance doesn't really help as I want to learn using these costs, not just to estimate the result.
    Or maybe I have to think of how to reformulate my task to make it solvable by existing algorythmes.
  • aborgaborg Member Posts: 66 Contributor II
    You might try something with the weight role. Some of the operators use that information. From your description it seems a column can be generated, and its role can be set to weight.
Sign In or Register to comment.