Parts of Speech (POS) Filtering

HyramHyram Member Posts: 39 Contributor II
edited June 2020 in Help
Hi. 

I am have tokenised some text and am now trying to remove POS, using the Filter by POS Operator. I have used the following expression: N.*|VB.*|RB.*|JJ.*|MD.*|PP.* in an attempt to keep nouns, adjectives, verbs and adverbs. The problem is that as an example, nouns and verbs were filtered out e.g. the word "need" is no longer present in my text. 
What am I doing wrong and do I have the right expression for the POS tokens I want to keep (nouns, adjectives, verbs and adverbs)?

Thanks,
Hyram 
Tagged:

Best Answers

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Solution Accepted
    As you seem to know, you need to use the PENN POS tags, which are available here:
    https://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos.html
    Your expression doesn't look like is has anything obviously wrong with it to me.
    @kayman can you take a look and see if you can find anything wrong?
    Also you might try doing one selection at a time to see if there is a problem with the compound expression?
    Or try filtering out specific tags rather than keeping only certain tags?
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • kaymankayman Member Posts: 662 Unicorn
    Solution Accepted
    Hi @Hyram , @Telcontar120

    Seems ok at first glance indeed.
    As recommended by Brian try to do the same without filters (or filter one by one) so you get an understanding which tag is given to 'need' by RM. POS tags are sensitive to word location etc, so depending on the string the same word can get different tags.

    Need can be both verb or noun for instance, but since you are capturing both it shouldn't be a problem to start with.
    You didn't accidentally select the invert option?

Answers

  • HyramHyram Member Posts: 39 Contributor II
    Thank you very much @Telcontar120 and @kayman. I will try what you've suggested. I didn't mistakenly select invert. I've also subsequently found that some of the words reside in the dictionary I'm using for stop words and hence some of them have been filtered out accordingly.
  • PenelopemaxPenelopemax Member Posts: 3 Learner I
    Having tokenized text, I'm attempting to remove parts of speech (POS) using the Filter by POS Operator. The expression is used to retain nouns, adjectives, verbs, and adverbs. However, I encountered an issue where, for example, the word "need" was filtered out. Seeking a solution to preserve essential words like "need" in the filtered text.

Sign In or Register to comment.