[SOLVED] Example Set Transformation - Performance Problem

South2woodSouth2wood Member Posts: 5 Contributor II
edited November 2018 in Help

I have a large data set (10 millionn lines)  where each line/example of the table is identified by two id-attributes, for example like this:

domain (id), week (id), value (attribute)
facebook.com; 1; 0,5
facebook.com; 2; 0,6
google.com; 1; 0,9
google.com; 2; 0,4

Now I want to transform the table into a time series, like this:
domain (id), value_week1 (attribute), value_week2 (attribute)
facebook.com; 0,5; 0,6
google.com; 0,9; 0,4

I tried to solve it by importing my data into a mysql database and do the transformation process with a php script, but it lasts for hours and didn't terminate yet. Is there a transformation operator in rapidminer which can do the job?

Another question. If I define two attributes as ID, does rapidminer understand that each line is defined by two IDs? I tried some set operations (Set Minus, Join) and it doesn't seem to work?

Happy New Year by the way ;-)

Best regards,


  • Options
    earmijoearmijo Member Posts: 270 Unicorn
    For your first question: Use the operator Pivot (using domain as the group attribute and week as the index attribute).


  • Options
    South2woodSouth2wood Member Posts: 5 Contributor II
    Thanks, I dont know why I didn't see it. Exactly what I needed!
  • Options
    misanthropic789misanthropic789 Member Posts: 8 Contributor II
    I was unable to get it to see more than one ID field.  Not sure if I was doing something wrong or if its a limitation of the software.
  • Options
    MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    If you need one unique ID you could use the Generate Attribute operator to create a new attribute from the two ID attributes.

    Best, Marius
Sign In or Register to comment.