I want to normalize my text but I do not know how?
Is the normalized operator doing this?
I want to identify the words that the user has written briefly and write in full. For example
img -> image
Or replace the abbreviations with the original. Like
DM -> dimention reduction
Or I can correct words that are spelling mistakes. Like
gud -> good
I'm waiting for the answer.
You can use 'Replace Tokens' operator from text processing extension for performing exactly this kind of replacement.
However note that this operator requires that you manually create a dictionary for such replacement. Obviously it is not able to autoimatically detect that 'img' stands for 'image', 'gud' for 'good' and so on.
To clarify further, the regular Normalize operator has to do with transformations of numerical attributes into a consistent scale (range) for comparison, it is not related to text data at all. Your needs with your text data are not "normalization" from a statistical perspective. They are text substitutions, which can be accomplished in RapidMiner as @kypexin said with the Replace Tokens operator. You can also use the Stopwords (Dictionary) operator if you want to remove certain words/tokens entirely. Or you can use either Map or Replace operators if you have text inside an ordinary polynominal attribute in an example set that you want to modify. But all of these require you to generate your own list of the specific substitutions or eliminations you want to make.
Dictionary has to be created manually in case you would use 'Replace Tokens' operator.
First you go to operator's parameters tab:
After pressing 'Edit List' button you will be able to add dictionary entries one by one. replace what token on the left, replace by what token on the right:
Dictionary in this case stays within operator parameter settings and is not represented by a standalone file or table.
What I meant is that in 'Replace Tokens' parameters you have to manually list each pair of entries, with your examples from the first post it would look like this (so each time it sees 'img' in the input document it replaces with 'image' and so on; but you should also note that there's no certain 'magic' here, all the pairs have to be 'hardcoded' in some sense):
Thank you very much
How should I correct the spelling of words? For example, I'll get to know. That user, spelling a word image-->Imege
Or how to figure out the word gud. What word has been
good? Or GUD
Thanks for helping this too
Can I use wordnet? How? My data is twitter data.
Replace Token operator can be used to correct misspellings or otherwise substitute one token for another.
I have never used Wordnet myself so can't help really much with that.
'Replace tokens' operator does not offer any way to "guess" which word can be represented by a certain type of misspelling; you still have to handle all the token pairs manually.