"Dictionaries and stemming"

LorenzoLorenzo Member Posts: 7 Contributor II
edited May 2019 in Help
Hi everyone!
I'm currently using the text plug-in and I want to clarify a bit some of its peculiarities. I'm not using the block DictionaryStemmer and I'm simply working with the English stopword filter, the tokenizer and the Porter stemmer.

What I guessed is that:
1) my text is filtered against a set of English stop words and some words are pruned (Ex. and, or..).
    - I have to work with texts on biology and so I'm wondering what happens with strange words such as IL-6. Are these words filtered or maintained?
2) The stemmer keeps only the "basic chunks" of my words. I think that this is based on a dictionary.
    - Could you tell me which dictionary is that? I need to know that precisely in order to answer to the question "does it contain some medical terms such as glicolase..?" that is crucial for me now
    - What does it happen to my strange word (Ex. IL-6)? Are they pruned, chunked in some way or kept as they are?

Thanks for your kind attention. Hope that someone can help!
Lorenzo

Tagged:

Answers

  • TobiasMalbrechtTobiasMalbrecht Moderator, Employee, Member Posts: 294 RM Product Management
    Hi Lorenzo,

    unfortunately I am not that familiar with the text plugin and thus I can not answer your questions immidiately. But I will try to get that information from the developer of the text plugin. This however might take a while. I post again as soon as I obtained the information.

    Regards,
    Tobias
  • LorenzoLorenzo Member Posts: 7 Contributor II
    Thank you very much for your kind attention and for your availability.
  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,751 RM Founder
    Hello,

    1) my text is filtered against a set of English stop words and some words are pruned (Ex. and, or..).
        - I have to work with texts on biology and so I'm wondering what happens with strange words such as IL-6. Are these words filtered or maintained?
    As far as I know they should be maintained.

    2) The stemmer keeps only the "basic chunks" of my words. I think that this is based on a dictionary.
        - Could you tell me which dictionary is that? I need to know that precisely in order to answer to the question "does it contain some medical terms such as glicolase..?" that is crucial for me now
        - What does it happen to my strange word (Ex. IL-6)? Are they pruned, chunked in some way or kept as they are?
    The stemming is not performed based on a dictionary but on a stemming algorithm. Here you can find a description of the algorithm:

    http://tartarus.org/~martin/PorterStemmer/def.txt

    Cheers,
    Ingo
  • Legacy UserLegacy User Member Posts: 0 Newbie
    Hi,

    there are several steps to word vector creation. The first of them is tokenizing. The simple tokenizer in RM discards everything that is not recognized as character and uses it as split point. Thus, I am afraid that your fancy words will not survive this step. You can either:

    1. Use the Feature Extractor to extract your fancy words explicitly
    2. Implement a custom tokenizer (the API is really easy to implement against)

    Instead of (or in combination with) the Porter Stemmer, you can provide your own dictionary containing regex..

    Best regards,
    Michael
Sign In or Register to comment.