Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

How do I tokenize an example set?

darwin_h_poritzdarwin_h_poritz Member Posts: 1 Learner II
edited December 2018 in Help
I have an example set of 451 comments (rows).  I want to generate mult-word tokens (n-Grams) from this example set.  What is the sequence of operators?
Tagged:

Answers

  • Telcontar120Telcontar120 RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    You'll first need to set your attribute to text data type (it is probably nominal by default), and then use the Process Documents from Data operator.  Within that you will put the tokenize operator followed by the N-Grams operator.  You may also wish to conduct other types of text preprocessing as well. 
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • greg_lorincz79greg_lorincz79 Member Posts: 18 Maven
    edited March 2021
     <3 
Sign In or Register to comment.