column-1's value is based on column-2 value; should we remove one?

ruserruser Member Posts: 40 Maven
The Database Table which I have taken for the Clustering purpose, has some clumns which are calculated based on the values in the other columns.
col-M  col-N=(5*col-M)      col-R          col-S    col-T
x          5x                            a                c        a-c
y          5y                            b                d        b-d

In such cases, is it better to remove the redundant columns(apart from the ones which will be helpful to interpret the clustering results)?


  • Options
    steffensteffen Member Posts: 347 Maven

    Here is what I am thinking:
    Removing "redundant" columns is not that easy:

    To refer to your example
    R  S  T(=R-S)
    1  1  0
    2  2  0

    The distance using euclidean metric constrained to R and S is Squareroot(2), the distance constrained to T is 0. So be careful when you are removing columns...

    On the other side: Assuming that the created columns are necessary, keeping even the redundant columns will at worst increase the absolute distance between to items.



    PS:  I cannot hold back to remark, that the results (in comparsion by just using all original columns) may change if your additional columns have been calculated on wicked (e.g. nonlinear) transformations which your metric cannot cope with. But I guess you are aware of that.
Sign In or Register to comment.