🎉 🎉 RAPIDMINER 9.10 IS OUT!!! 🎉🎉

Download the latest version helping analytics teams accelerate time-to-value for streaming and IIOT use cases.

CLICK HERE TO DOWNLOAD

"Text Mining and the Word List"

mschmitzmschmitz Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,153  RM Data Scientist
edited June 2019 in Knowledge Base

Symptoms

Using Process Documents (from Data) you are able to generate a tokenized example set from a given set of documents. If you use one Process documents for your training and another for the testing you might get the error incompatible number of attributes if you apply the model.

Diagnosis

The problem probably that you did not transfer the word list from one Process Documents to the other. The wordlist contains mainly two information:

  •  Which attribute to generate
  • The normalization

If you do not transfer the wordlist over, words which do not occur in your document won't create a attribute. In case of pruning different words will be deleted from your bag of words. Another effect is of course that even if you create the same attributes, your normalization (of TF/IDF) might be different.

 

Solution

 Wordlist.png

Transfer over the wordlist created in your training stream over to the application stream. Thus you create

- Head of Data Science Services at RapidMiner -
Dortmund, Germany
JEdwardbhupendra_patilmariannita

Comments

  • cicicici Member Posts: 6 Newbie
    Sorry I don’t quite understand , how can I generate and transfer wordlist???Thank you for your reply
  • mschmitzmschmitz Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,153  RM Data Scientist
    by connecting the wor port of the upper process documents with the wor port of the lower one as shown in the screenshot.
    - Head of Data Science Services at RapidMiner -
    Dortmund, Germany
  • cicicici Member Posts: 6 Newbie

    Hello, I have referred to the icon connection operator you gave, but it still shows the actual attributes. Don’t know if I’m doing this right?
  • cicicici Member Posts: 6 Newbie
    Thank u for your reply! I have followed just the connection you gave, but it still shows that the attributes do not match. I want to ask you another question. Since I already have a training set and a test set, why should I use validation?
Sign In or Register to comment.