Text Pre-processing

HyramHyram Member Posts: 39 Contributor II
edited June 2020 in Help
Hi there
I am trying to do some preprocessing on text and looking for the relevant operators in RapidMiner, if they are indeed available. 
I am extracting features from a sentence, using Information Gain operator. This seems to be possible. From there, I need to construct a feature vector using Bag of Words (BOW) and Term Frequency (TF). I should end up with a vector of unigrams. I want this vector of unigrams to be based on Part of Speech (POS) for each term in the sentence. 

The operators I am looking for are:
1. BOW;
2. TF;
3. PoS tagging.

Are these available in RapidMiner or am I looking in the wrong operator directories?

Thanks

Best Answers

  • Pavithra_RaoPavithra_Rao Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 123 RM Data Scientist
    Solution Accepted
    Hi @Hyram,

    Text processing extension could help you out with text mining feature extraction and processing.


    Also here's some good resource to help you get started with RM Text processing

    https://academy.rapidminer.com/learn/course/text-and-web-mining-with-rapidminer/text-and-web-mining/lets-get-started

    https://rapidminer.com/resource/text-mining-document-classification/

    https://rapidminer.com/resource/text-mining-document-classification/

    Cheers,
    Pavithra

  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Solution Accepted
    You need to download and install the free text mining extension from the marketplace.
    The operator "Process Documents" will generate a word vector using term frequency if you set that as the option in the parameters (TF-IDF is the default), and it will also automatically generate the bag of words for you if you use the Tokenize operator inside and then output the wordlist and the exampleset (depending on the format you want it in). 
    There is also an operator for "Filter Tokens (by POS Tags)" but I am not sure if you can get it to actually output the POS tag, or whether you can only filter by the tags (in which case I guess you could add them manually based on the filtered results? but that seems inefficient).
    @mschmitz is there any way to output the POS tag directly?
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Solution Accepted
    Yes the values in the word vector correspond to the TF-IDF values calculated across the exampleset.  
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts

Answers

  • HyramHyram Member Posts: 39 Contributor II
    Thanks for your assistance Telcontar120 and Pavithra_Rao!
  • HyramHyram Member Posts: 39 Contributor II
    Hi @mschmitz

    Brian has answered my question perfectly. The only issue outstanding is how I use PoS tagging. Can I reflect the tags or only filter by them?

    Thanks
  • HyramHyram Member Posts: 39 Contributor II
    Thank you. 
    I have one more question @mschmitz. How would I remove hashtags and URLs from my text? What operator would I use? Replace? I've looked at the previous posts around this and the tutorial which a community member suggested I look at, no longer exists.

    Thanks
  • Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    Yes, you should be able to use the Replace operator to get rid of hashtags and URLs with some creative regular expressions. I am not a regex expert so there are other community members who could probably help with the specifics of that more than I can. In both cases you are probably going to want to look for some pattern (such as the # sign or the htttps://) followed by some arbitrary number of characters, then a space---and you want to get rid of everything up to the space.

    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • HyramHyram Member Posts: 39 Contributor II
    @Telcontar120 Thank you so much for your help. I have managed to produce the word vector now and the example set, using TF.IDF. I assume the values in the example set (fractions), represent the TF.IDF number?

    Just need to sort out URLs now. Filter Tokens using 'non-letter', seems to sort out # but not the test immediately after, as you've suggested. At least now I know that I need to look for regular expressions.

    Thanks again!
  • HyramHyram Member Posts: 39 Contributor II
    @Telcontar120 awesome! Thank you
Sign In or Register to comment.