Web Mining for Other Languages

krsnewwavekrsnewwave Member Posts: 1 Contributor I
edited June 2019 in Help
Hi!

I have some Japanese pages in my analysis, and I noticed that the operator "Extract Content" isn't friendly with texts other than UTF-8. Is there any way to change how it handles its encoding?

(EDIT: While this question is hanging, I think I'm going to try the complement - removing all html tags and the regions with <script>. It seems to work okay thus far.)

Answers

  • restuarrestuar Member Posts: 8 Contributor II
    I agree. My colleagues and I plan to conduct comparative studies on English and Chinese online newspapers. Would you have anything in the pipeline for Cantonese and Mandarin character recognition? Thanks!
  • haddockhaddock Member Posts: 849 Maven
    Hi there,

    I know people who do this sort of thing, they say that a problem with Chinese is tokenising the sentences, as there are no spaces to separate the words, check this out http://www.foreverastudent.com/2012/03/chinese-word-frequency-list-news.html .

    It is possible, but not easy!

    Good luck.
  • restuarrestuar Member Posts: 8 Contributor II
    Thank you. My team mates and I have agreed to use the english version of the website. My problem now is how to tell rapidminer that when it accesses the chinese website, it should use the translated one. Would you know how to solve this problem?
Sign In or Register to comment.