🦉 🎤   RapidMiner Wisdom 2020 - CALL FOR SPEAKERS   🦉 🎤

We are inviting all community members to submit proposals to speak at Wisdom 2020 in Boston.


Whether it's a cool RapidMiner trick or a use case implementation, we want to see what you have.
Form link is below and deadline for submissions is November 15. See you in Boston!

CLICK HERE TO GO TO ENTRY FORM

[SOLVED] Performance RAM Rapid Miner

marcopomarcopo Member Posts: 14 Contributor II
edited June 27 in Help
Hi,

at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:

criterion: accurancy

minimal size for split: 4

minimal leaf size: 2

minimal gain : 0,1

maximal depth 10

confidence 0,25

Cross Validation 5

sampling type : stratifield sampling

Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?

Best Regards and thank you

Marco
Tagged:

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869   Unicorn
    Hi Marco,

    the decision tree is one of the worst choices for data with many attributes. (It is perfectly fit to process data with a huge amount of examples though, with only few attributes).

    For data with many attributes, but only a few examples as in your case, try a Linear SVM. Please note that the parameter C of the SVM must be optimized. You can use the Optimize Parameters (Grid) for that. For a choice of the search range for C, please have a look at this thread. Please let me know if you have any problems.

    Concerning the usage of RAM, we are planning to make the maximum amount of memory available to RapidMiner configurable with the next release, which is planned to be available to the public on next monday.

    Best regards,
    Marius
  • marcopomarcopo Member Posts: 14 Contributor II
    Thanks a lot for the replay.

    It's funny I am doing at the moment the SVM with optimization ;-)
    I am eager to use the new release.

    Best Regards
    Marco

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869   Unicorn
    For SVM optimization and understanding you may be interested in this thread: http://rapid-i.com/rapidforum/index.php/topic,6194.0.html
    I'll mark this topic as solved now.

    Best regards,
    Marius
Sign In or Register to comment.