Options

[SOLVED]RapidMiner Sentiment Analysis Problem

ChikoChiko Member Posts: 26 Maven
edited November 2018 in Help
Hi,

I have a RapidMiner studio process that trains a liner SVM using positive and negative product reviews. The training part works ok upto performance calculation. However, when I Apply my model on unseen unlabeled data, I am getting the error:

Problem occured. The input ExampleSet does not match the training ExampleSet. Missing attribute: 'aaaahh'. The operator expects the input ExampleSet to have a set of attributes which is equal or a superset of the ExampleSet used for training of the input model. Please make sure that the attributes of the two ExampleSets satisfy this condition. This beats me, because what is happening here is that during training, I am using the Process Documents from Data operator to tokenize my text, similarly I do the same to the unlabelled data just before passing it through to the model. Considering that the training and testing ExampleSet will contain different words and phrases,  and that these words are turned into attributes by the Process Documents operator, I cannot understand why the Apply model operator thinks that the attributes in training example set should match the attributes in the testing set should match, hence its expection to find the word 'aaaahh' also in the training set. Could anyone point me in the right direction please. (technically I can see why this is happening but it seems that it is illogical, so I must have done something wrong with my process design)

Unfortunately I cannot embed the code as my message would exceed the 20k character limit.

Thanks

Answers

  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,517 RM Data Scientist
    Hi,

    have you added the word list from training to the process documents operator from applying?

    ~Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    bsboddenbsbodden Member Posts: 1 Contributor I

    Why is this marked as [SOLVED]? Is the one reply the correct answer? I'm having the same problem.

  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,517 RM Data Scientist
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,761 Unicorn

    To build on what @mschmitz's knowledge base post, once you do your text transformations (i.e. tokenize, filter stop words, etc) in the Process Documents operator, many words will be stripped out of the corpus (i.e. the, a, lol). The TDIDF values of the remaining words get passed downstream via the EXA port to your machine learning algorithm. It will have "X" columns. 

     

    The problem comes in when your testing set gets processes and there are "X + n" columns to apply your model too, then the process breaks.  Hence the passing the Wordlist from WOR port to the testing set. This way only the columns you trained your model one will be selected for the testing set.

Sign In or Register to comment.