image

🎉 🎉 RAPIDMINER 9.10 IS OUT!!! 🎉🎉

Download the latest version helping analytics teams accelerate time-to-value for streaming and IIOT use cases.

CLICK HERE TO DOWNLOAD

[SOLVED] Deleting text noise from large corpus

Legacy UserLegacy User Member Posts: 0 Newbie
edited November 2018 in Help
Hi

I have a pdf file which contains several thousand pages of emails.  The problem is that each email contains a unique set of noise (unique because it does not repeat).  For example:
x-Mail: hbcFNvIWLDtFlpP.yxyP9bkreUY5ZzdUGPpkOhYIoR
This noise sometimes fills entire pages.  

Can anyone point me in the right direction on how to minimize this noise, or somehow go around it?

Thanks.

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869   Unicorn
    Hi Sal,

    if you use the TF-IDF measure, the noise will be ignored (gets value 0), because it appears in only one document and thus does not bring in any advantage for text classification.
    Furthermore, the Process Documents operator has parameters to filter out words that appear too seldom (or too often).

    Best,
      Marius
Sign In or Register to comment.