Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
Answers
term frequency is simply the normalized number of occurrences of a term divided by the number of terms of the document (from the souce code):
int numTerms = wordList.size();
double totalTermNumber = 0;
for (float value: frequencies)
totalTermNumber += value;
// Create the result structure
double[] wv = new double[numTerms];
// If document contains at least one term
if (totalTermNumber > 0) {
// Create the vector
double length = 0.0;
for (int i = 0; i < wv.length; i++) {
wv = frequencies / totalTermNumber;
length += wv * wv;
}
length = Math.sqrt(length);
// Normalize the vector
if (length > 0.0)
for (int i = 0; i < wv.length; i++)
wv = wv / length;
}
In case of TDIDF, the term frequency of each term is normalized with the inverse document frequency (from the source code):
// Obtain the total number of documents and the document frequencies
int numDocuments = wordList.getNumberOfDocuments();
int[] docFrequencies = wordList.getDocumentFrequencies();
double totalTermNumber = 0;
for (float value: frequencies)
totalTermNumber += value;
// Create the result structure
double[] wv = new double[docFrequencies.length];
// Create the vector
// If the document contains at least one term
if (totalTermNumber > 0) {
double length = 0.0;
for (int i = 0; i < wv.length; i++) {
// Note: docFrequencies is always > 0 as otherwise the word
// would not be in the word list, it is also always smaller as
// the total number of documents
double idf = Math.log(((double) numDocuments) / ((double) docFrequencies));
wv = (frequencies / totalTermNumber) * idf;
length += wv * wv;
}
length = Math.sqrt(length);
// Normalize the vector
if (length > 0.0)
for (int i = 0; i < wv.length; i++)
wv = wv / length;
}
You can find a discussion about the normalization somewhere in the forum if this is interesting for you.
Cheers,
Ingo
On Text classification with one are more suitable? Thanks in advance!!
as always in data mining: just test it. If one of both preprocessings works better for your data then just go for it. In general, however, TFIDF has outperformed term frequency alone on all practical settings I have experienced so far.
Cheers,
Ingo