Can I see the individual results of each X-Validation fold (before they are averaged)?

jochen_hartmannjochen_hartmann Member Posts: 5 Contributor I
edited November 2018 in Help

Dear community,

 

Is there a way to view the individual performance results of each fold of X-Validation or only the overall average?

 

I am applying a 5-fold X-Validation and would be interested in understanding the how different the results of each fold are, e.g., in terms of accuracy (or even better F1 score)?

 

If this is not possible with the X-Validation operator, what alternatives would I have? (As a work-around I created three 80/20 splits with the Split Validation operator, using the three different "sampling types", but I am not really satisfied with this appraoch.)

 

 

Thanks in advance!

Best Answers

  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist
    Solution Accepted

    Dear Jochen,

     

    it is possible, but just with a little trick. You can use performance to data on your performance vector and put this on the inner tes port. If you do this, you get a table with performances back at the outer tes port.

     

    The reason that you need to do this trick is, that it often is not wanted to look at the individual performances. Keep in mind: The average of the perfomance of a cross validation is the best unbiased estimator for the true performance of the method. (with an emphesis on method). 

     

    I saw time and time again that scientist report the best fold of their x-val in papers. This is simply wrong. Only the average is to be reported.

     

    Best,

    Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,503 RM Data Scientist
    Solution Accepted

    Hi,

     

    i personally go for a simple t-test on the two performances. It's very tough to do more tests with a sample size of 5-10. In your case both will be fine and you usually take the "easier explainable" one. For SVM and k-NN you might take SVM because it is WAY faster in application (if you need this speed).

     

    Best,

    Martin

    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany

Answers

  • jochen_hartmannjochen_hartmann Member Posts: 5 Contributor I

    Thanks so much for your swift and helpful reply, Martin!

     

    Understood. Then, my follow-up question is rather a statistical one:

     

    How can I tell if the X-Validation results of two different classifiers, e.g., kNN and SVM, are in fact statistically distinguishable if they are very close to each other (accuracies of 44.5 and 45.2, respectively, for example). I thought it may be a good idea to look at the underlying distributions of accuracies from each fold, but I may also be completely off.

     

    In other words, is there a way to determine a threshold to be able to tell if two results are "different enough"?

     

    Thanks!

  • jochen_hartmannjochen_hartmann Member Posts: 5 Contributor I

    Thank you, Martin!

Sign In or Register to comment.