🎉 🎉 RAPIDMINER 9.10 IS OUT!!! 🎉🎉

Download the latest version helping analytics teams accelerate time-to-value for streaming and IIOT use cases.


How to rate / score created models

qwertzqwertz Member Posts: 130  Maven
edited November 2018 in Help
Dear community,

I've got the following puzzle to solve but my problem is that I currently dont't know how to get started.

I have 10 models that provide sales prediction for different countries. These models have each been built on several training and testing iterations so that I have quality figures for each model like "prediction trend accuracy" or "standard deviation" (based on ratio prediction to label).

Now I feed all the models with the latest market data and get a prediction value from every applied model. But which predicted value has the best potential?

Model 1 has high quality figures, e.g. "prediction trend accuracy" is 85% and "standard deviation" is 1,5. The predicted value however is rather low, e.g. 15.
Model 2 has low quality figures, e.g. "prediction trend accuracy" is 60% and "standard deviation" is 8.  The predicted value in this case is rather high, e.g. 30.
How to find out which scenario is the better one - the one that predicts a higher market value but with less accuracy or the other one?

I thought of maybe applying a decision tree and train it with data from the past but not sure whether this is a valid way at all.
Another idea is simply to multiply "prediction trend accuracy" with the predicted value increase / decrease.

It would be glad if you could help me to find the right approach.

Kind regards
Sign In or Register to comment.