The RapidMiner community is on read-only mode until further notice. Technical support via cases will continue to work as is. For any urgent licensing related requests from Students/Faculty members, please use the Altair academic forum here.
How to remove duplicate data
![jmphillips](https://us.v-cdn.net/6030995/uploads/defaultavatar/nCCNNSPK1YM69.jpg)
![](https://s3.amazonaws.com/rapidminer.community/vanilla-rank-images/contributor-16x16.png )
Hello community: I have a problem withe duplicate data, this is an example:
Col 1 Col 2
so I want to remove one of those rows.
Help please, you are my only hope.....
Col 1 Col 2
39-2021 | 49-2021 |
49-2021 | 39-2021 |
so I want to remove one of those rows.
Help please, you are my only hope.....
Tagged:
0
Answers
this sounds like a good use case for De-Pivot, removing the duplicates, and Pivot again.
De-Pivot will put all values into subsequent rows and keep the information about the column name (Col 1, Col 2) in another column. You would then remove the duplicates according to your rules and use Pivot to build the original data structure again.
Be careful to remove entire "rows" (original rows, which are multiple rows after the De-Pivot). If you don't do that, you will have missing data upon pivoting again.
Regards,
Balázs
If I de pivot all SECOND_ID are goit to a new collum, with the FIRST_ID data, so if I then remove duplicates Im going to loose data, the problem si that I want to eliminate for example *** 24413-2016 24260-2016 because I have 24260-2016 24413-2016 the first and the second Id in the case are the same but inverted.
another approach would be duplicating the data with Multiply and then joining with Left Outer join on FIRST_ID = SECOND_ID *and* SECOND_ID = FIRST_ID. You could then decide which matches to remove.
Regards,
Balázs
Regards.
the second row is the same values but inverted, so I want to remove the complete second row.
Regards.
FIRST_ID
SECOND_ID
SIMILARITY