The Altair Community is migrating to a new platform to provide a better experience for you. The RapidMiner Community will merge with the Altair Community at the same time. In preparation for the migration, both communities are on read-only mode from July 15th - July 24th, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here.

Options

## Answers

1,751RM Founderterm frequency is simply the normalized number of occurrences of a term divided by the number of terms of the document (from the souce code):

int numTerms = wordList.size();

double totalTermNumber = 0;

for (float value: frequencies)

totalTermNumber += value;

// Create the result structure

double[] wv = new double[numTerms];

// If document contains at least one term

if (totalTermNumber > 0) {

// Create the vector

double length = 0.0;

for (int i = 0; i < wv.length; i++) {

wv

= frequencies/ totalTermNumber;length += wv

* wv;

}

length = Math.sqrt(length);

// Normalize the vector

if (length > 0.0)

for (int i = 0; i < wv.length; i++)

wv

= wv/ length;

}

In case of TDIDF, the term frequency of each term is normalized with the inverse document frequency (from the source code):

// Obtain the total number of documents and the document frequencies

int numDocuments = wordList.getNumberOfDocuments();

int[] docFrequencies = wordList.getDocumentFrequencies();

double totalTermNumber = 0;

for (float value: frequencies)

totalTermNumber += value;

// Create the result structure

double[] wv = new double[docFrequencies.length];

// Create the vector

// If the document contains at least one term

if (totalTermNumber > 0) {

double length = 0.0;

for (int i = 0; i < wv.length; i++) {

// Note: docFrequencies

is always > 0 as otherwise the word

// would not be in the word list, it is also always smaller as

// the total number of documents

double idf = Math.log(((double) numDocuments) / ((double) docFrequencies

));wv

= (frequencies/ totalTermNumber) * idf;length += wv

* wv;

}

length = Math.sqrt(length);

// Normalize the vector

if (length > 0.0)

for (int i = 0; i < wv.length; i++)

wv

= wv/ length;

}

You can find a discussion about the normalization somewhere in the forum if this is interesting for you.

Cheers,

Ingo

9Contributor IIOn Text classification with one are more suitable? Thanks in advance!!

1,751RM Founderas always in data mining: just test it. If one of both preprocessings works better for your data then just go for it. In general, however, TFIDF has outperformed term frequency alone on all practical settings I have experienced so far.

Cheers,

Ingo