The Altair Community is migrating to a new platform to provide a better experience for you. The RapidMiner Community will merge with the Altair Community at the same time. In preparation for the migration, both communities are on read-only mode from July 15th - July 24th, 2024. Technical support via cases will continue to work as is. For any urgent requests from Students/Faculty members, please submit the form linked here.
Options

# Language modeling with RapidMiner

Member Posts: 4 Contributor I
edited November 2018 in Help
Hello,

I am trying to process a collection of text documents in the following manner:
1. For each document d compute a term frequency vector fv;
2. transform term frequencies in fv into probabilities and obtain a term probability vector pv
3. Use a precomputed language model lm to compute Kullback-Leibler divergence between fv and lm.

I was successful with completing step 1  Step 2 is less clear - how do I get the count of all terms in fv? Step 3 is totally unclear. My language model has a vertical layout, i.e. entries are roughly in form [term][probability of occurence]. The probability vector in turn has a horizontal layout - (p_of_term_1, ... p_of_term_n).

Any suggestions?

• Options
RapidMiner Certified Expert, Member Posts: 458 Unicorn
• Options
Member Posts: 4 Contributor I
Hi Andrew,

Thanks! Your suggestion actually helped complete step 2. There was one difficulty though - I had my label attribute mixed with the probabilities after 'Transpose' and therefore attribute statistics couldn't be computed properly and all sorts of other strange things happened. Hope this helps someone else.

Regarding step 3, i guess 'Transpose' operator can be used to join the document term probability vector and language model using the n-gram text as a key. Neat! Hopefully, there will not be any transposition problems due to lots of n-grams (my largest language model has around 92 000 n-grams).

Thanks again and have a nice evening!
• Options
Member Posts: 15 Contributor II
Hi,

>> I had my label attribute mixed with the probabilities after 'Transpose' and therefore attribute statistics couldn't >> be computed properly and all sorts of other strange things happened. Hope this helps someone else.

Dos the label attibute set as the lable role? If not, use the Set Role operator, and set the label attribute with the correct role.