"a bug while getting the token in arabic"
Im trying to compare a word I get as a token and a word that is hardcoded in a dictionary but the result is always false. even though when printed to the log it shows the same word Im looking for I dont know what Im missing and I dont know how to debug this. I just tried to get the bytes for the token and the bytes for the preset word and they are different so why is this happening with me considering that it runs perfectly in english.
is there a flag I should set to support UTF8?