RapidMiner

normalize informal text

Wisdom logo Registration now open for RapidMiner Wisdom Americas | New Orleans | October 10-12, 2018   Learn More
Highlighted
Contributor II jabra
Contributor II

normalize informal text

Hi everyone
I want to normalize my text but I do not know how?
Is the normalized operator doing this?
I want to identify the words that the user has written briefly and write in full. For example
img -> image
Or replace the abbreviations with the original. Like
DM -> dimention reduction
  Or I can correct words that are spelling mistakes. Like
gud -> good
Please guide
I'm waiting for the answer.
Thankful

12 REPLIES
Unicorn
Unicorn

Re: normalize informal text

Hi @jabra

 

You can use 'Replace Tokens' operator from text processing extension for performing exactly this kind of replacement.

However note that this operator requires that you manually create a dictionary for such replacement. Obviously it is not able to autoimatically detect that 'img' stands for 'image', 'gud' for 'good' and so on.   

Unicorn
Unicorn

Re: normalize informal text

To clarify further, the regular Normalize operator has to do with transformations of numerical attributes into a consistent scale (range) for comparison, it is not related to text data at all.  Your needs with your text data are not "normalization" from a statistical perspective.  They are text substitutions, which can be accomplished in RapidMiner as @kypexin said with the Replace Tokens operator.  You can also use the Stopwords (Dictionary) operator if you want to remove certain words/tokens entirely.  Or you can use either Map or Replace operators if you have text inside an ordinary polynominal attribute in an example set that you want to modify. But all of these require you to generate your own list of the specific substitutions or eliminations you want to make.

 

Brian T., Lindon Ventures - www.lindonventures.com
Analytics Consulting and Training by Certified RapidMiner Experts
Contributor II jabra
Contributor II

Re: normalize informal text

Hello .
Thankful
what is your suggestion?
How to create a dictionary. And use it?
Please help.
Thanks

 

Unicorn
Unicorn

Re: normalize informal text

Hi @jabra

 

Dictionary has to be created manually in case you would use 'Replace Tokens' operator. 

First you go to operator's parameters tab: 

 

Screenshot 2018-04-17 09.31.37.png

 

After pressing 'Edit List' button you will be able to add dictionary entries one by one. replace what token on the left, replace by what token on the right: 

 

Screenshot 2018-04-17 09.31.45.png

Contributor II jabra
Contributor II

Re: normalize informal text

Hello
Can you send me a small sample dictionary?
I want to know what format and what kind of dictionary should I do?
Many thanks

Unicorn
Unicorn

Re: normalize informal text

Hi @jabra

 

Dictionary in this case stays within operator parameter settings and is not represented by a standalone file or table. 

What I meant is that in 'Replace Tokens' parameters you have to manually list each pair of entries, with your examples from the first post it would look like this (so each time it sees 'img' in the input document it replaces with 'image' and so on; but you should also note that there's no certain 'magic' here, all the pairs have to be 'hardcoded' in some sense): 

 

Screenshot 2018-04-18 10.23.53.png

Contributor II jabra
Contributor II

Re: normalize informal text

Thank you very much
How should I correct the spelling of words? For example, I'll get to know. That user, spelling a word image-->Imege
 
Or how to figure out the word gud. What word has been
good? Or GUD
Thanks for helping this too
Can I use wordnet? How? My data is twitter data.

Unicorn
Unicorn

Re: normalize informal text

Replace Token operator can be used to correct misspellings or otherwise substitute one token for another.

Brian T., Lindon Ventures - www.lindonventures.com
Analytics Consulting and Training by Certified RapidMiner Experts
Unicorn
Unicorn

Re: normalize informal text

I have never used Wordnet myself so can't help really much with that.

'Replace tokens' operator does not offer any way to "guess" which word can be represented by a certain type of misspelling; you still have to handle all the token pairs manually.