The Altair Community is migrating to a new platform to provide a better experience for you. The RapidMiner Community will merge with the Altair Community at the same time. In preparation for the migration, both communities are on read-only mode from July 15th - July 24th, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here.

How do I tokenize an example set?

darwin_h_poritzdarwin_h_poritz Member Posts: 1 Newbie
edited December 2018 in Help
I have an example set of 451 comments (rows).  I want to generate mult-word tokens (n-Grams) from this example set.  What is the sequence of operators?


  • Options
    Telcontar120Telcontar120 Moderator, RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,635 Unicorn
    You'll first need to set your attribute to text data type (it is probably nominal by default), and then use the Process Documents from Data operator.  Within that you will put the tokenize operator followed by the N-Grams operator.  You may also wish to conduct other types of text preprocessing as well. 
    Brian T.
    Lindon Ventures 
    Data Science Consulting from Certified RapidMiner Experts
  • Options
    greg_lorincz79greg_lorincz79 Member Posts: 18 Maven
    edited March 2021
Sign In or Register to comment.