"Regarding training Performance metrics in cross Validation"

varunm1varunm1 Member Posts: 563   Unicorn
edited May 23 in Help
Hi,

I am looking to get performance metrics (AUC, Accuracy & RMSE) during training in a cross-validation operator. Are there any suggestions for this?

@lionelderkrikor @Telcontar120

Thanks
Varun

Best Answer

Answers

  • hughesfleming68hughesfleming68 Member Posts: 220   Unicorn
    edited December 2018
    Hi Varun, have you tried logging the output using the log operator? Also see if connecting the performance to data operator inside the cross validation operator does what you want.
    varunm1
  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,172   Unicorn
    As my colleagues have mentioned, it is possible to get this information from RapidMiner using the Log operator.  However, I would be quite careful, typically the training error is NOT useful for understanding your model performance.  That's the whole reason you are doing cross-validation, to understand the error on the test set instead.

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
    varunm1lionelderkrikor
  • varunm1varunm1 Member Posts: 563   Unicorn
    Hi @Telcontar120. Thanks for your response. I am looking into CV test performance, this training performance is to compare with some other work going on.

    @hughesfleming68 @lionelderkrikor Thank you.
    lionelderkrikor
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,619  RM Founder
    Just wanted to chime in since this is a topic I care a lot about (as many probably know by now :wink: )  Even this type of comparison can be pretty much useless.  Reminder: the most simple machine learning model in the world (K-Nearest Neighbors with k=1) has always training error of 0% :smiley:
    Anyway, here is my "magnum opus" on validations and why training errors should be always completely ignored IMHO:
    Hope I do not sound like a cranky school teacher here though...
    Cheers,
    Ingo
    varunm1lionelderkrikorTelcontar120
Sign In or Register to comment.