RapidMiner 9.8 Beta is now available

Be one of the first to get your hands on the new features. More details and downloads here:

GET RAPIDMINER 9.8 BETA

Parts of Speech (POS) Filtering

HyramHyram Member Posts: 39 Contributor II
edited June 9 in Help
Hi. 

I am have tokenised some text and am now trying to remove POS, using the Filter by POS Operator. I have used the following expression: N.*|VB.*|RB.*|JJ.*|MD.*|PP.* in an attempt to keep nouns, adjectives, verbs and adverbs. The problem is that as an example, nouns and verbs were filtered out e.g. the word "need" is no longer present in my text. 
What am I doing wrong and do I have the right expression for the POS tokens I want to keep (nouns, adjectives, verbs and adverbs)?

Thanks,
Hyram 
Tagged:

Best Answers

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,530   Unicorn
    Solution Accepted
    As you seem to know, you need to use the PENN POS tags, which are available here:
    https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
    Your expression doesn't look like is has anything obviously wrong with it to me.
    @kayman can you take a look and see if you can find anything wrong?
    Also you might try doing one selection at a time to see if there is a problem with the compound expression?
    Or try filtering out specific tags rather than keeping only certain tags?
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
    Hyram
  • kaymankayman Member Posts: 510   Unicorn
    Solution Accepted
    Hi @Hyram , @Telcontar120

    Seems ok at first glance indeed.
    As recommended by Brian try to do the same without filters (or filter one by one) so you get an understanding which tag is given to 'need' by RM. POS tags are sensitive to word location etc, so depending on the string the same word can get different tags.

    Need can be both verb or noun for instance, but since you are capturing both it shouldn't be a problem to start with.
    You didn't accidentally select the invert option?
    Hyram

Answers

  • HyramHyram Member Posts: 39 Contributor II
    Thank you very much @Telcontar120 and @kayman. I will try what you've suggested. I didn't mistakenly select invert. I've also subsequently found that some of the words reside in the dictionary I'm using for stop words and hence some of them have been filtered out accordingly.
Sign In or Register to comment.