Overfitting - Sentiment Analysis
Newbie
Hi together,
I am not very experienced. I did use the sentiment template and created a model with about 83 % accuracy. But the model does not predict the sentiments of my unseen data well. The confidence average is about 50 to 60% only. What can I do to get a model which generalizes better? And is there an opportunity to compare my labeled data with the unlabeled data to see if the bad confidence is really so so bad.
my training data is balanced about 1000 positive / 1000 negative. And I applied the model to about 100 unlabeled data.
Thank you very much for your help Silke
I am not very experienced. I did use the sentiment template and created a model with about 83 % accuracy. But the model does not predict the sentiments of my unseen data well. The confidence average is about 50 to 60% only. What can I do to get a model which generalizes better? And is there an opportunity to compare my labeled data with the unlabeled data to see if the bad confidence is really so so bad.
my training data is balanced about 1000 positive / 1000 negative. And I applied the model to about 100 unlabeled data.
Thank you very much for your help Silke
Tagged:
0
Answers
For example, How did you construct your model? Did you use cross-validation? What ML algorithm are you using?
Are the new 100 cases that you are validating similar to the original 2000 that you built the model on?
Did you ensure they were all having their text data processed in the same way that you did on your original model development sample? You should be using a pre-built wordlist when you use this approach, and this is something that many users forget to do. There are some good tutorials and lessons on text mining in the RapidMiner Academy that you might want to check out.
Lindon Ventures
Data Science Consulting from Certified RapidMiner Experts
Maybe SVM is not the most relevant model for your data.
Have you tried to submit your dataset to AutoModel to determine how some other models perform ?
Regards,
Lionel