Options

clustering - how to customize similarity function

LeMarcLeMarc Member Posts: 72 Contributor II
edited May 2020 in Help
Hi

so clustering is basically based on similarity/distance functions in which one example is compared to all the other examples of the data set.
Now I want a similiraty function in which the cell of an attribute/feature is not compared to all the other examples - instead that particular cell should be compared to only a given range of defined values. For example: a cell can have two possible values [yes,no]. So the similiarity function should compare the given cell value with just the values 'yes' and 'no'.



Is this possible with RM? If so, how?

Thank you!
Tagged:

Best Answers

Answers

  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,508 RM Data Scientist
    Hi,

    i do not completly understand what you want to do, but i think this is simply not a distance measure? As you propably know, distance measures have a few assumptions..
    Technically you can register new distances, which are then available in all operators. But this requieres java and it feels like this is not want you want to do.

    Cheers,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    LeMarcLeMarc Member Posts: 72 Contributor II
    edited May 2020

    I have an example set in which some data errors exist within the cell values (e.g. spelling mistake, empty, wrong content , etc.). The goal is to detect examples with data issues.
    My approach in using clustering method is to create a cluster with all examples which do not have data errors at all [Cluster_0]. Members of the other clusters [Cluster_1, Cluster_2] consist of examples with at least 1 data error. Below is an example set.



    The method is to change the way HOW the value of the similarity function [1, 0, 0,333] for an example is calculated. Since based on the similarity the clusters are build. (1) Therefore one need to define a similarity function for each (categorical) attribute type in which the cell value is compared to a defined range of values instead of comparing the cell value to all the other existing values of that particular attribute (see below) within the example set. For Married it would be [yes, no]. For Colour [Blue, Red] and Job [Yes, No]. M stands for married.



    So if the cell value [3/Married] contains e.g. 'yes' or 'no' than the output would be '1' (see below). However if the cell value would be e.g. '-1' than the output would be '0'.



    The idea is: every time a cell value does not conform to the defined range value, the similarity value of the whole example decreases and will be <1 and therefore exclude the example from the cluster_0 where there is no data errors (similarity = 1). Does this make sense/work?

    So the question is how to implement (1)?




    @Telcontar120 Btw what is the function if I want to include more than just one value to compare? e.g. contains ([Married], "yes" ??




  • Options
    MartinLiebigMartinLiebig Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, University Professor Posts: 3,508 RM Data Scientist
    Hi @LeMarc ,
    this sounds more like you want to use CrossDistance to a Reference table?
    Cheers,
    Martin
    - Sr. Director Data Solutions, Altair RapidMiner -
    Dortmund, Germany
  • Options
    LeMarcLeMarc Member Posts: 72 Contributor II
    Sorry for the late reply. Its been quite busy here.
    Thank you @Telcontar120 for your help. It is appreciated.

    @mschmitz Yes, very similar to CrossDistance to a Reference Table because I try to find an clustering approach which is independet of clustering algorithm, settings and clustering method.
Sign In or Register to comment.