It looks like you're new here. If you want to get involved, click one of these buttons!
Prune method on the "Process Documents" is filtering the tokens by the number of document occurrences. How can I filter them by "Total Occurrences" instead?
You should be able to review the output wordlist and identify the tokens you want to eliminate by sorting by total occurrences, and create a small text file with those words. Then you can use Filter Stopwords (dictionary) to suppress those tokens from your document processing.
Thanks for the reply. I see what you are saying however,the problem is that I cannot filter or use the total occurrences field. Sorting seems to be an unnecessary step since I need to filter the ones under certain threshold.
Sorting is also not working properly. There is nothing in populated in attribute list for sort operator but if I put total_occurrences, it works with a warning.
RapidMiner AI Hub
Automated Data Science
Training Classes & Certification
ML Algorithm Reference
Educational License Program