Options
[SOLVED] Performance RAM Rapid Miner
Contributor II
Hi,
at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:
criterion: accurancy
minimal size for split: 4
minimal leaf size: 2
minimal gain : 0,1
maximal depth 10
confidence 0,25
Cross Validation 5
sampling type : stratifield sampling
Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?
Best Regards and thank you
Marco
at first so much thanks for this nice software. I am working on a text mining case. I have got 4000 example rows and 35000 (words TF-IDF) attributes. I try to classify the articles into three classes and I am using the decision tree. My settings are:
criterion: accurancy
minimal size for split: 4
minimal leaf size: 2
minimal gain : 0,1
maximal depth 10
confidence 0,25
Cross Validation 5
sampling type : stratifield sampling
Unfortunately RapidMiner uses just 6 GB of 16 GB RAM and the calculation goes now over 4 hours. What can I do to improve the performance of the calculation?
Best Regards and thank you
Marco
Tagged:
0
Answers
the decision tree is one of the worst choices for data with many attributes. (It is perfectly fit to process data with a huge amount of examples though, with only few attributes).
For data with many attributes, but only a few examples as in your case, try a Linear SVM. Please note that the parameter C of the SVM must be optimized. You can use the Optimize Parameters (Grid) for that. For a choice of the search range for C, please have a look at this thread. Please let me know if you have any problems.
Concerning the usage of RAM, we are planning to make the maximum amount of memory available to RapidMiner configurable with the next release, which is planned to be available to the public on next monday.
Best regards,
Marius
It's funny I am doing at the moment the SVM with optimization ;-)
I am eager to use the new release.
Best Regards
Marco
I'll mark this topic as solved now.
Best regards,
Marius