Using HIndi Language in Tokenizer

alabiitalabiit Member Posts: 2 Contributor I
edited November 2018 in Help
Hi,

My documents to be analyzed are in Hindi. The encoding format is UTF-8. For creating the word Vector I have used WVplugin. The problem is that  I am not getting all the tokens (I used all the tokenizers in rapidminer 4.6), in fact i am getting too low - 4 to be precise  ???

I changed the  content language and encoding to Hindi and UTF, but without any sucess - is there any additional setup to be done to tokenize the  text properly?

~alabiit

Answers

  • landland RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 2,531 Unicorn
    Hi,
    from RapidMiner 5.0 on, you can configure the tokenizer more detailed. You can enter arbitrary split characters so that it should work with any language that splits its words with a character at all.

    Greetings,
      Sebastian
  • alabiitalabiit Member Posts: 2 Contributor I
    Hi sebastin,

    Thanks. But upgrading is always tough job  :'(.

    will check out.
    ~alabiit
Sign In or Register to comment.