Due to recent updates, all users are required to create an Altair One account to login to the RapidMiner community. Click the Register button to create your account using the same email that you have previously used to login to the RapidMiner community. This will ensure that any previously created content will be synced to your Altair One account. Once you login, you will be asked to provide a username that identifies you to other Community users. Email us at Community with questions.
"a bug while getting the token in arabic"
Hello,
Im trying to compare a word I get as a token and a word that is hardcoded in a dictionary but the result is always false. even though when printed to the log it shows the same word Im looking for I dont know what Im missing and I dont know how to debug this. I just tried to get the bytes for the token and the bytes for the preset word and they are different so why is this happening with me considering that it runs perfectly in english.
is there a flag I should set to support UTF8?
Im trying to compare a word I get as a token and a word that is hardcoded in a dictionary but the result is always false. even though when printed to the log it shows the same word Im looking for I dont know what Im missing and I dont know how to debug this. I just tried to get the bytes for the token and the bytes for the preset word and they are different so why is this happening with me considering that it runs perfectly in english.
is there a flag I should set to support UTF8?
Tagged:
0
Answers
the description is a bit vague. Did you set your process encoding to UTF-8 (click the empty white space of your process in the RapidMiner GUI and select UTF-8 as encoding parameter.
Regards,
Marco
a sample usecase
if (token.contains("arabic_word") then remove it
when I print the token text it show the same value as the "arabic_word" but they are not equal when they are compared in the operator.
this problem is specific to rapidminer because I wrote my code first in java and it worked just find but when I changed the source of the word from the tokens it started to give wrong results.