Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
"[SOLVED] Performance Operator Evaluation - Mathematics / Derivation"
jaysunice3401
Member Posts: 6 Contributor II
Would someone be able to enlighten me on the mathematics behind some of the performance evaluation metrics and/or point me to a nice resource/website? Specifically, if I am using a Performance (Classification) Operator, I would like to know how the following are derived:
Jason
- Accuracy: specifcially, the +/- %
- The difference between the mikro percentages and the given percentages
- Classification Error vs. Relative Error vs. Root Mean Squared Error
- How/Why the +/- % for Accuracy, Weighted Mean Recall, and Classification Error are different and why
Jason
Tagged:
0
Answers
- the accuracy is defined as the probability that a new example is classified correctly. It is calculated as (#ofCorrectPrediction/#numberOfExamples)
- the classification error is 1-accuracy
- the absolute error is calculated via the following formula: sum(1-confidence(trueClass)) / #numberOfExamples
- the relative error is absolute_error * 100%
- the root mean squared error is calculated as: sqrt( sum( (1-confidence(trueClass))^2 ) / #numberOfExamples )
The +- and the makro/mikro values are only calculated if the performance is estimated by a Cross Validation. In that case, the accuracy is calculated for each fold (iteration) of the validation. The makro performance is the average of the performance value of all folds, the +- states the standard deviation of that value.
For the mikro average remember that each fold of the X-Validation uses 10% of the data set as test set and creates predictions on that set. After all 10 folds, there exist predictions for the complete dataset, and you can calculate the accuracy based on these predictions. The result is the mikro average. Since it is calculated from only on single dataset, there is no standard deviation.
Hope this helps!
Best regards,
Marius