[SOLVED] Deleting text noise from large corpus

Legacy UserLegacy User Member Posts: 0 Newbie
edited November 2018 in Help

I have a pdf file which contains several thousand pages of emails.  The problem is that each email contains a unique set of noise (unique because it does not repeat).  For example:
x-Mail: hbcFNvIWLDtFlpP.yxyP9bkreUY5ZzdUGPpkOhYIoR
This noise sometimes fills entire pages.  

Can anyone point me in the right direction on how to minimize this noise, or somehow go around it?



  • Options
    MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Hi Sal,

    if you use the TF-IDF measure, the noise will be ignored (gets value 0), because it appears in only one document and thus does not bring in any advantage for text classification.
    Furthermore, the Process Documents operator has parameters to filter out words that appear too seldom (or too often).

Sign In or Register to comment.