The Altair Community is migrating to a new platform to provide a better experience for you. The RapidMiner Community will merge with the Altair Community at the same time. In preparation for the migration, both communities are on read-only mode from July 15th - July 24th, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here.
Options

Compare Models Side by Side

btibertbtibert Member, University Professor Posts: 146 Guru
Is there any way to compare the performance of two or more models side-by-side?  The Compare models in the Model Management Extension appears to auto-select the winner, and Group Models -> Apply appears to process in order, effectively using the output from the last model.

The use-case is showing my students how to compare and think about the output to make a smart decision on what is best for their business problem.  While I could have them export the results individually, which requires tabbing across the results, I was hoping for something that aggregated and displayed the performance metrics side by side for each.

Best Answer

Answers

  • Options
    lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn
    Hi @btibert,

    Why not use the AutoModel tool ?. In the final screen (results panels) , the performances of all selected models are compared...

    Regards,

    Lionel
  • Options
    varunm1varunm1 Moderator, Member Posts: 1,207 Unicorn
    edited September 2019
    Hi @lionelderkrikor

    Any idea, if it is possible to get a table just like the one auto model creates in our own process that we create? I looked for it but I didn't find any luck in getting, except the reporting tools that create documents. I think in auto model they are fetching values from the operators and displaying in the UI, not sure we have some operators to do the same.
    Regards,
    Varun
    https://www.varunmandalapu.com/

    Be Safe. Follow precautions and Maintain Social Distancing

  • Options
    lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn
    Hi @varunm1,

    Yes, in practice, you're right, Varun : To my knowledge there is no operator that generates a table like the one presented by AutoModel...
    But as a partial solution, you have the Compare ROCs operator. This operator can be useful to compare the performances of classifiers (binary classification).

    Regards,

    Lionel
  • Options
    lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn
    @varunm1, @btibert,

    It is not exactly what you are looking for, but I found a process  that I used one time. It can be a good starting point.
    This process is called "Automatic Model Selection and Optimization"
    You can find it in the Training Ressources repository.

    Concretely, in this first release of the process, you can compare the performance (here the accuracy) of 3 models in the results panels.
    I think that this process, can be easily improved to compare N models and/or to compare other performance metrics than accuracy.

    Here a screenshot of the table of performances comparaison : 



    Regards,

    Lionel



  • Options
    btibertbtibert Member, University Professor Posts: 146 Guru
    Thanks @lionelderkrikor, I will give your process a look.  I am aware of Automodel, but because I am teaching a class where the application of these techniques is new for them, I want the group to wrestle with the mental model of what is going on; the fitting of a model, the application of it, and the assessment.  I want the class to understand the mechanics of what is happening first.
  • Options
    btibertbtibert Member, University Professor Posts: 146 Guru
    And @lionelderkrikor , silly question, but in your process, you are able to use a note/comment and include a hyperlink.  How can I do that for my own students, as it would be great to link to external resources.  I must be missing something obvious.
  • Options
    btibertbtibert Member, University Professor Posts: 146 Guru
    Thanks @Telcontar120 , I wasn't aware of the performance to data, and that did the trick.  Obviously it's a manual approach and complexity grows linearly with the number of models we want to compare, but this does what I need.  And just for sake of argument, screenshot below on what it looks like on my side, as there was a need to add a column to identify the model type for sorting in the results.


  • Options
    lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 1,195 Unicorn
    Hi @btibert,

    OK, I understand your point of view concerning the use of AutoModel.
    Concerning the link to external ressources in the note, .............it's a good question !!!
    Me too, I don't see any easy method via the UI to have such links in a note...

    I just see one method : manually modify the XML of the process to add the link (See the last lines of the XML of the process, tag "description" to understand).

    Regards,

    Lionel

  • Options
    btibertbtibert Member, University Professor Posts: 146 Guru
    edited September 2019
    @lionelderkrikor
    Ahhh, I didn't think about modifying the XML, good to know!
Sign In or Register to comment.