Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.

IDF Calculation for Test Set

smjsmj1smjsmj1 Member Posts: 3 Contributor I
edited November 2018 in Help
Can anyone explain the calculation of IDF value for Test sets?
Is it based on the IDF of Training sets?
I see that test set take only the word list used by the training set and IDF is Calculated solely based on the test set. So, if Test set contain only 1 document, then there is a chance that IDF becomes 0, correct?

Answers

  • frasfras Member Posts: 93 Contributor II
    If you are using TF-IDF you must store model _and_ wordlist after training.
    To test or score unseen data you have to preprocess with exactly the same
    "Process Documents"-Operator that you used for training including the wordlist.
  • smjsmj1smjsmj1 Member Posts: 3 Contributor I
    Thank you for the reply
Sign In or Register to comment.