Training and testing correlation

PapadPapad Member Posts: 68 Guru
edited March 2020 in Help
When I test a model on known data, as a result I have an accuracy. Is this accuracy always greater than the accuracy I have when I use cross validation where I want to see how my model reacts to unknown data? 
So,  doing the steps in tutorial "Testing a model" , if I have bad accuracy, should I try to find another model or prepare different my data and not go on a cross validation or something else?
Thanks in advance.

Best Answer

  • Options
    yyhuangyyhuang Administrator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 364 RM Data Scientist
    edited May 2019 Solution Accepted
    Hi @Papad,

    Most of the time training accuracy is better than the testing accuracy since the model is trained on it and worst case scenario, like k-nn (very small k), the model will try to memorize all the data points in training. 

    We usually ignore the training performances when we optimize the models, cos it would not be helpful for us to understand the predictive capabilities on unseen data. Some exceptions here like detecting outliers, verifying randomness of errors when we compare the predictions and labels for the training set.

    If you had bad accuracy on the testing set, rethink about the validation process and any shifted concept between testing and training data. 

    An insightful post from Ingo can also help you understand why we need cross validation and how to interpret it  (strongly recommend also this)



Sign In or Register to comment.