RapidMiner

Include Negations in Dictionary based Sentiment Approach

KD Nuggets logo The annual KDnuggets poll on data science tools is live. RapidMiner needs your vote!VOTE NOW
Highlighted
Learner I katrin_kandlbin
Learner I

Include Negations in Dictionary based Sentiment Approach

Hi all,

 

I want to use the dictionary based sentiment approach for texts by using a self-created dictionary.

The dictionary sentiment approach does not account for negations as far as I know.

For example: Today I was not really productive.

The word "productive" is positive but because of the "not" the whole sentence has a negative meaning instead of positive.

 

How does the dictionary based sentiment approach of Rapidminer account for this issue?

 

All the best,

Katrin

4 REPLIES
RM Staff RM Staff
RM Staff

Re: Include Negations in Dictionary based Sentiment Approach

Hi Katrin,

 

welcome to the community! I've written the operator myself and I am very happy that you use it. The current version of the operator does not handle negations. Negations are in general a tricky thing also in more sophisticated approaches like SVMs. I am not sure how this dictionary based approach could handle this. If you have a solution in mind I am happy to have a look if we can implement this.

 

Cheers,

Martin

--------------------------------------------------------------------------
Head of Data Science Services at RapidMiner

Re: Include Negations in Dictionary based Sentiment Approach

Hi,

Negation is in fact only a small part of the problem: imagine the sentence "probably a possible tumor" in the notes of a radiologist. 

Smiley Happy

Sven

Learner I katrin_kandlbin
Learner I

Re: Include Negations in Dictionary based Sentiment Approach

Hi Martin,

first of all: happy new year. Thank you very much for your reply.

One possible solution to account for negations might be to include another "dictionary" of self-created word list which consists of negation words.

This word list is applied to the same text corpus as the sentiment dictionary in the following way:

  1. all words in the text corpus should be numbered consecutively
  2. the sentiment dictionary operator recognizes that (for example) word Nr. 12 is a sentiment word and assigns the corresponding value (1 if positive and -1 if negative)
  3. the new negation operator then concentrates only on the sentiment words of the text corpus (here word Nr. 12) and examines the surrounding words - let's say 5 words before the sentiment word and 5 words after.
  4. If there is a negation word included 5 words before or after the sentiment word, the value of the sentiment word will be multiplied by (-1) in order to reverse the sentiment value.

Do you think this might work?

 

Best,

Katrin

RM Staff RM Staff
RM Staff

Re: Include Negations in Dictionary based Sentiment Approach

Hi @katrin_kandlbin,

i've implemented this (without counting but same idea). I will share a preview version of this via e-mail since i need to write some docs and it needs to run through our internal code-quality process before it appears on market place.

 

Best,

Martin

--------------------------------------------------------------------------
Head of Data Science Services at RapidMiner
ezCater's RapidMiner Journey