Text Mining- Select Token based on Dictionary File

mbjassermbjasser Member Posts: 4 Contributor I
edited November 2018 in Help
Hi every one,

I'm trying to work on a text mining workflow to filter specific language contents based on  a specific language dictionary (TXT file most probably).

I was able to filter stopwords using the operator "Filter Stopwords (Dictionary)" to filter the content depending on a dictionary, but I'm still trying to select tokens based also on a dictionary, but it seems that the only operator offered is Filter Tokens (by contents) (which enables selecting tokens based on a regular expression, there is no option for selecting tokens based on a dictionary file).

I need your support if you have an idea if there exists any operator to do that, or if I'm missing something.

Thank you in advance

Answers

  • JEdwardJEdward RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 578 Unicorn
    Seconded. 

    See my post here: http://rapid-i.com/rapidforum/index.php/topic,8008.msg27328.html#msg27328 suggesting that a group of us band together to pay for RM to improve language support in the text extension.  Particularly for more difficult languages such as Arabic, Chinese, Indonesian or even TxtSpk. 
  • frasfras Member Posts: 93 Contributor II
    There is a operator "Filter Stopwords" where you can place a
    txt file with one stopword per line.
  • mbjassermbjasser Member Posts: 4 Contributor I
    Thanks fras, I'm already using the operator of Filter Stop words based on dictionary, but this operator is filtering words, I want to select tokens based on dictionary not filtering (removing). I'm not able to find any up to now. The only option offered is filter tokens by content using the regular option field.

  • mbjassermbjasser Member Posts: 4 Contributor I
    it's god idea JEdward, hopefully it'll work
  • JEdwardJEdward RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 578 Unicorn
    Yes, definitely something I'm going to follow up on in the next couple of weeks. 
Sign In or Register to comment.