Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
"Loop over Tokens in RapidMiner"
eisioriginal
Member Posts: 4 
Contributor I
Hello,
is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But then the rest of my intended operations are very complicated.
I'm doing this to simulate the creation of a language model like Katz Backoff and so on. Maybe there is a better way to produce language models but in the moment i can only think of iterating the tokens and calculate the values and store them into an example set.
Thank you
Andreas
is there any possibility to iterate over the tokens of a text document within RapidMiner? My first try was to window the document after tokenisation. But then the rest of my intended operations are very complicated.
I'm doing this to simulate the creation of a language model like Katz Backoff and so on. Maybe there is a better way to produce language models but in the moment i can only think of iterating the tokens and calculate the values and store them into an example set.
Thank you
Andreas
Tagged:
0
Answers
maybe you can use the "Generate n-Grams (Terms)" operator.
Roland
to loop over the tokens you can use the Loop Attributes operator, since each token is represented by a single attribute.
Best regards,
Marius