Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

Different performance between grid search and real test

YADAOYADAO Member Posts: 1 Learner I
edited October 2020 in Help
Hi guys,
I used optimize parameters to find the best parameters' values with 10 fold cross-validation, and it showed out that the best kappa was 0.397. But when I applied these parameter values to 10 fold cross-validation myself, the kappa was only 0.337.
What's wrong?
Thank you.
Tagged:

Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,527 RM Data Scientist
    Hi @YADAO ,
    can it just be sampling differences? What happens if you change the seed of your x-val?

    Best,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • swordfswordf Member Posts: 3 Learner II

    Is there still someone on this issue now?
    I have the same problem, different results are presented using the suggested parameter by grid optimizer,

    I don't understand why, even if it's not the same it should be close at least
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,527 RM Data Scientist
    Hi @swordf ,
    close in number of standard_deviations, yes.
    Best,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
Sign In or Register to comment.