"Regarding training Performance metrics in cross Validation"

varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
edited May 2019 in Help
Hi,

I am looking to get performance metrics (AUC, Accuracy & RMSE) during training in a cross-validation operator. Are there any suggestions for this?

@lionelderkrikor @Telcontar120

Thanks
Varun
Regards,
Varun
https://www.varunmandalapu.com/

Be Safe. Follow precautions and Maintain Social Distancing

Best Answer

Answers

  • hughesfleming68hughesfleming68 Member Posts: 323 Unicorn
    edited December 2018
    Hi Varun, have you tried logging the output using the log operator? Also see if connecting the performance to data operator inside the cross validation operator does what you want.
  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    As my colleagues have mentioned, it is possible to get this information from RapidMiner using the Log operator.  However, I would be quite careful, typically the training error is NOT useful for understanding your model performance.  That's the whole reason you are doing cross-validation, to understand the error on the test set instead.

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
    Hi @Telcontar120. Thanks for your response. I am looking into CV test performance, this training performance is to compare with some other work going on.

    @hughesfleming68 @lionelderkrikor Thank you.
    Regards,
    Varun
    https://www.varunmandalapu.com/

    Be Safe. Follow precautions and Maintain Social Distancing

  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Just wanted to chime in since this is a topic I care a lot about (as many probably know by now :wink: )  Even this type of comparison can be pretty much useless.  Reminder: the most simple machine learning model in the world (K-Nearest Neighbors with k=1) has always training error of 0% :smiley:
    Anyway, here is my "magnum opus" on validations and why training errors should be always completely ignored IMHO:
    Hope I do not sound like a cranky school teacher here though...
    Cheers,
    Ingo
Sign In or Register to comment.