I cannot read the data correctly. Help.

wdengwdeng Member Posts: 4 Contributor I
edited November 30 in Help

I come across a task is to read the data from a publication from the government. when I read it with rapid miner, it shows the wrong label and wrong cell content. the file is in CSV format. For example, if I read it in excel, the cell has 2 lines but rapid miner shows 2 cells instead.

https://files.ontario.ca/opendata/2536_bridge_conditions.csv

Tagged:

Answers

  • kypexinkypexin Moderator, RapidMiner Certified Analyst, Member Posts: 193   Unicorn

    Hi @wdeng

     

    It's a bit tricky to understand the issue without knowing how exactly you are expecting the data to look like.

     

    I tried to read this file with RM and logically the structure looks ok, except that the heading row seems to be consisting of multiple (bi-lingual) entries (?). Maybe this is where you should go for manual columns renaming. Otherwise, comma delimited values in the columns seem valid to me.

     

    Screenshot 2017-11-13 10.53.11.pngScreenshot 2017-11-13 10.52.56.png   

     

    UPD: opening file in Excel (Mac version) gives me that: 

     

    Screenshot 2017-11-13 11.31.03.png

    sgenzer
  • SGolbertSGolbert RapidMiner Certified Analyst, Member Posts: 184   Unicorn

    I am afraid the header of the csv is incorrect. I think that you should edit it to be only one line separated by single commas.

  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 234  RM Data Scientist

    Hi @wdeng,

     

    indeed it would be better to have a file with better (i.e. consistent) formatting.

     

    In the actual case you could use the Read CSV Operator with the Wizard and play around with the settings. Screenshots with my settings are attached. They worked for me to get at least a portion of your data. 

     

    Best,

    Edin

     

    image.png

     

    image.png 

     

    image.png

    sgenzer
  • wdengwdeng Member Posts: 4 Contributor I

    csv.PNG

    I expect the data look like this. Thank you for replying.

  • wdengwdeng Member Posts: 4 Contributor I

    Once I edit the CSV file, the file will be corrupted. 

  • wdengwdeng Member Posts: 4 Contributor I

    I did this but rapide miner only shows half of the header. It cuts off at current header.

     

    csv.PNG

  • Thomas_OttThomas_Ott RapidMiner Certified Analyst, RapidMiner Certified Expert, Member Posts: 1,762   Unicorn
    This particular CSV file is trickier than the rest because of the it's malforming when you load it into RM. There are errors relating to Row 3 and it's header file. I think this might require the writing of a REGEX instead of using just the comma as a default deliminator
  • Edin_KlapicEdin_Klapic Moderator, Employee, RMResearcher, Member Posts: 234  RM Data Scientist

    Hi @wdeng,

     

    Unfortunately, I think fancy regular expressions also won't help you in this case.

    You need to have all relevant attributes in one line. Your desired output reflects that you want a combination of the lines 4,5 and 6 as Attribute names. So you need to manually combine them using a file editor. While doing this I recommend to delete the lines 1-3.

    Then you should be all set to read in your file with Separator "Comma" and "Use Quotes" checked.

     

    Best,

    Edin

     

    sgenzer
Sign In or Register to comment.