Student Dataset is giving different classification accuracy using cross validation on RapidMiner 9.6

VikasRattanVikasRattan Member Posts: 7 Learner I
Student Dataset is giving different classification accuracy using cross validation on RapidMiner 9.6(educational version)

Best Answer

  • varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
    Solution Accepted
    Try to run the attached process without changing multiple times as see the results. I enable random seed for SMOTE, Cross-Validation & random tree. You can import this process by going to File --> Import process. You need to set a random seed for all operators that have that option. A random seed will help generate the same data all the time and even in the random tree, it will do the same randomization. These are critical to producing reproducible results. 

    Let me know if you still have issues.
    Regards,
    Varun
    https://www.varunmandalapu.com/

    Be Safe. Follow precautions and Maintain Social Distancing

Answers

  • varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
    edited October 2020
    Hello @VikasRattan

    Did you set "Random Seed" option in cross-validation? If not, your folds might be divided differently during different runs. Also which algorithm are you using inside cross-validation?


    Regards,
    Varun
    https://www.varunmandalapu.com/

    Be Safe. Follow precautions and Maintain Social Distancing

  • VikasRattanVikasRattan Member Posts: 7 Learner I
    Varun Ji, I observed it for Random forest, Random tree, Knn, Naive Bayes. I have set Random seed, which is 1922, and without setting random seed. In both cases, got different accuracy on different runs. Even though, i used startified sampling, shuffled sampling and linear sampling, i got different accuracy when executing at different point of time.
  • varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
    Can you share your .rmp file? You can go to File --> Export Process and then attach that process here.
    Regards,
    Varun
    https://www.varunmandalapu.com/

    Be Safe. Follow precautions and Maintain Social Distancing

  • VikasRattanVikasRattan Member Posts: 7 Learner I
    Sure Sir

    File is attached.
Sign In or Register to comment.