Options

"Text dictionary matching"

sbsb Member Posts: 9 Contributor II
edited May 2019 in Help
The Filter (Dictionary) filters OUT words - is there a way to keep words matching those in a dictionary.  I can use FiilterTokens(byContent), but this needs a verrrrrry long list  of words as a regular expression.  Am looking for something akin to an 'Invert' choice in the Filter(Dictionary) operator.
Thanks.
Tagged:

Answers

  • Options
    colocolo Member Posts: 236 Maven
    Hi,
    sb wrote:

    The Filter (Dictionary) filters OUT words
    that's why it is called "Filter Stopwords" ;)

    Since the "Filter Documents/Tokens" operators do not provide the ability to use a dictionary file, you could perhaps modiy these operators (or invert the bevavior of the stopword filter) if you are familiar with Java programming. If you don't want to look at the source code you might possible load a dictionary file and automatically build a regular expression from it (just concatenate the dictionary words separated by a vertical bar). But I don't know if there are some length limitations for macros that would prevent you from using this long expression as parameter for "Filter Documents (by Content)".
    Just a few thoughts...

    Regards
    Matthias
  • Options
    landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi,
    the good news are: No length limitation. The Bad news: Currently no possibility to invert it. But might be easier implementing a script that will build the "opposite" of tokens in a document given two documents than implementing the hole dictionary method again.

    Greetings,
      Sebastian
Sign In or Register to comment.