RapidMiner

RapidMiner now offering a 30 day free trial of RapidMiner Studio Large! Learn more

normalize informal text

Contributor I jabra
Contributor I

normalize informal text

Hi everyone
I want to normalize my text but I do not know how?
Is the normalized operator doing this?
I want to identify the words that the user has written briefly and write in full. For example
img -> image
Or replace the abbreviations with the original. Like
DM -> dimention reduction
  Or I can correct words that are spelling mistakes. Like
gud -> good
Please guide
I'm waiting for the answer.
Thankful

12 REPLIES
RM Certified Analyst
RM Certified Analyst

Re: normalize informal text

Hi @jabra

 

You can use 'Replace Tokens' operator from text processing extension for performing exactly this kind of replacement.

However note that this operator requires that you manually create a dictionary for such replacement. Obviously it is not able to autoimatically detect that 'img' stands for 'image', 'gud' for 'good' and so on.   

Highlighted
RM Certified Expert
RM Certified Expert

Re: normalize informal text

To clarify further, the regular Normalize operator has to do with transformations of numerical attributes into a consistent scale (range) for comparison, it is not related to text data at all.  Your needs with your text data are not "normalization" from a statistical perspective.  They are text substitutions, which can be accomplished in RapidMiner as @kypexin said with the Replace Tokens operator.  You can also use the Stopwords (Dictionary) operator if you want to remove certain words/tokens entirely.  Or you can use either Map or Replace operators if you have text inside an ordinary polynominal attribute in an example set that you want to modify. But all of these require you to generate your own list of the specific substitutions or eliminations you want to make.

 

Brian T., Lindon Ventures - www.lindonventures.com
Analytics Consulting and Training by Certified RapidMiner Analysts
Contributor I jabra
Contributor I

Re: normalize informal text

Hello .
Thankful
what is your suggestion?
How to create a dictionary. And use it?
Please help.
Thanks

 

RM Certified Analyst
RM Certified Analyst

Re: normalize informal text

Hi @jabra

 

Dictionary has to be created manually in case you would use 'Replace Tokens' operator. 

First you go to operator's parameters tab: 

 

Screenshot 2018-04-17 09.31.37.png

 

After pressing 'Edit List' button you will be able to add dictionary entries one by one. replace what token on the left, replace by what token on the right: 

 

Screenshot 2018-04-17 09.31.45.png

Contributor I jabra
Contributor I

Re: normalize informal text

Hello
Can you send me a small sample dictionary?
I want to know what format and what kind of dictionary should I do?
Many thanks

RM Certified Analyst
RM Certified Analyst

Re: normalize informal text

Hi @jabra

 

Dictionary in this case stays within operator parameter settings and is not represented by a standalone file or table. 

What I meant is that in 'Replace Tokens' parameters you have to manually list each pair of entries, with your examples from the first post it would look like this (so each time it sees 'img' in the input document it replaces with 'image' and so on; but you should also note that there's no certain 'magic' here, all the pairs have to be 'hardcoded' in some sense): 

 

Screenshot 2018-04-18 10.23.53.png

Contributor I jabra
Contributor I

Re: normalize informal text

Thank you very much
How should I correct the spelling of words? For example, I'll get to know. That user, spelling a word image-->Imege
 
Or how to figure out the word gud. What word has been
good? Or GUD
Thanks for helping this too
Can I use wordnet? How? My data is twitter data.

RM Certified Expert
RM Certified Expert

Re: normalize informal text

Replace Token operator can be used to correct misspellings or otherwise substitute one token for another.

Brian T., Lindon Ventures - www.lindonventures.com
Analytics Consulting and Training by Certified RapidMiner Analysts
RM Certified Analyst
RM Certified Analyst

Re: normalize informal text

I have never used Wordnet myself so can't help really much with that.

'Replace tokens' operator does not offer any way to "guess" which word can be represented by a certain type of misspelling; you still have to handle all the token pairs manually. 

ezCater's RapidMiner Journey