Options

Closest encoding to utf8mb4

Robi_MeRobi_Me Member Posts: 32 Maven
in Help
I am working with social media data and all those emojis are driving me crazy, when I import them they are getting changed to system encoding and are a bunch of squiggles. What encoding is closets to utf8mb4 so that I can preserve the encoding when reading from a CSV?

Best Answer

  • Options
    Robi_MeRobi_Me Member Posts: 32 Maven
    Solution Accepted
    @jwpfau when I am importing into the DB it is failing saying the character is not UTF8 with error message: Incorrect string value: '\xE2 \x94 \x82....'

    This is basically all of the emojis that were being rejected. I was under the impression that I needed to set the encoding inside of Rapid Miner, however it was a change that was needed on the DB. Changing the free text field to TEXT and making the encoding UTF8mb4 sorted the issue out. 

Answers

  • Options
    kaymankayman Member Posts: 662 Unicorn
    If your csv is utf8 you should just be able to read it when you import it as utf8. Never had any issues with this, as long as the source was properly encoded. 
  • Options
    Robi_MeRobi_Me Member Posts: 32 Maven
    Nope, all emojis and non ascii characters are turning into their base encoding. Any idea why encoding is closest to UTF8MB4?
  • Options
    jwpfaujwpfau Employee, Member Posts: 282 RM Engineering
    Hi,

    UTF8MB4 is a workaround for the broken UTF8 type in mysql which only supports up to 3 byte character.
    In the csv export it should be just regular utf-8.

    Maybe the selected RapidMiner Studio font doesn't contain all the smileys and is displaying squares instead?

    Greetings,
    Jonas


Sign In or Register to comment.