How Rapidminer handle same distance for KNN Algorithm

ademuchlisademuchlis Member Posts: 4 Contributor I
edited August 13 in Help
Maybe I'm rather stupid but I just can't find a satisfying answer: Using the KNN-algorithm,
say k=5. Now I try to classify an unknown object by getting its 5 nearest neighbours.
What to do, if distance is a lot of the same distance..
if after determining the 4 nearest neighbors, the next 2 (or more) nearest objects have the same distance and diferent label? Which object of these 2 or more rapidminer chosen as the 5th nearest neighbor?

I confused.. I try in excel, and the result is diferent with rapidminer for some data.

in case like that, how rapidminer sorting distance ?...
something wrong with my data ?, or rapidminer sorting random if same distance ?

Thanks in advance :)



  • TghadiallyTghadially Employee, Community Manager, Member Posts: 6  Community Manager
    Hi @ademuchlis,

    This link should answer your question but feel free to reach out if it did not!
  • ademuchlisademuchlis Member Posts: 4 Contributor I
    edited August 14
    Hi Tghadially,
    many thanks for your response,
    the link you provided is very useful..

    unfortunately this is a new account, and can't attach image or link,

    based on what I read in other forums and the links that you provide,
    so for KNN there are several ways to handle the same distance..?
    looking average distance, or something like that.

    and which one is used by rapidminer ? ..

    I can't understand and find what kind of algorithm used by rapidminer in determining if the distances are the same?..

    maybe can be described like this
    the results of calculating data testing against data training is :

    data training 1st to 4th distance is 0 (count distance 0 is 4)
    data training 5th to 10th distance is 1 (count of distance 1 is 6)
    data training 11th to 15th distance is 2 (count of distance 2 is 5)
    data training 16th to 20th distance is 3 (count of distance 3 is 5)
    data training 21st to 25th distance is 4 (count of distance 4 is 5)

    if the distance is sorted ascending, the result is so many same distance like that.
    if k = 5
    so in classification, the majority of labels from the data training will be used, which has the lowest 5 distance calculation..

    in the rapidminer algorithm what do the majority of the 1st to 5th data labels use? I think not, because there are some different data when I compare it with manual calculations using MS Excel.

    or is the majority of the 1st to 25th data labels?
    the distance 0 is 1st
    distance 1 is 2nd
    distance 2 is 3rd
    distance 3 is 4th
    distance 4 is 5th

    or is there averaged?
    or is there another algorithm used by rapidminer?
    and the result is different again if weighted vote is checked.

    I have not found a suitable rapidminer calculation with my manual calculations with the distance as above.

    I hope you understand what I mean..
    thanks in advance for your help..
  • TghadiallyTghadially Employee, Community Manager, Member Posts: 6  Community Manager
    Hi @ademuchlis I have promoted you, so you can now post images and screenshots!
  • ademuchlisademuchlis Member Posts: 4 Contributor I
    edited 2:31AM
    many thanks for your support..

    as I explained before
    actually the problem is like this :

    there are 7000 test data training
    and 3000 data testing.
    and so many same distance..

    I confused.. I try in excel for sorting distance the result is different with rapidminer for some data. in excel the result label is "LU" for K = 5
    I tried to train data with ID 182. column A and B is ID.
    the calculation is only from column C until L
    with label is column M.

    results from excel like this, the majority of the labels is "LU" :

    but why the result in rapidminer is "LT" : rapidminer result

    result rapidminer weighted vote is checked is "LU" : rapidminer weighted vote

    How rapidminer handle with case like that...

    how rapidminer sorting the same distance ?...
    something wrong with my data ?,
    or rapidminer sorting random for same distance?

    thanks you in advance for your help

  • ademuchlisademuchlis Member Posts: 4 Contributor I
    edited 2:40AM
    can anyone tell me about this?
  • lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 746   Unicorn
    Hi @ademuchlis,

    In order we can reproduce what you observe, and understand what's going on, can you please share : 

     - your process (XML)
     - your dataset.

    Unfortunately, I have no exact answer to your question....But in first ,approximation, considering k = 5, with no weighted vote : 
    You have for the four first closer neighbours 2 "LT" and 2 "LU" ...
    ...but for the fifth closer neighbour there is a lot of candidates which have the same distance to your test point (distance  = 1).
    My hypothese, in RapidMiner for the final choice of this fifth closer neighbour and thus for the final choice of the label of the test point are : 
     - the fifth neightbor is chosen randomly among the candidates (which have all a distance of 1 to the test point).
     - if the probability of the 2 labels are the same (here 50% (LT) / 50%(LU)), then the first training point in the dataset, in the loop of the internal code of RapidMiner, is chosen. In other words, it is equivalent to a random choice.
     - For equivalent candidates, the candidate are in alphabetic order classified so the "LT" label is chosen instead of "LU" label.
     - and finally the more logical explanation from my point of view : there is a majority of label "LT" (and a minority of label "LU") in the candidates of the fifth closer neighbour (which have all a distance of 1 to the test point). So logically the final conclusion is label = "LT" for the test point...

    Maybe some RapidMiner's developer(s) can dispel this mystery....?
    Thanks you,


  • IngoRMIngoRM Administrator, Moderator, Employee, RapidMiner Certified Analyst, RapidMiner Certified Expert, Community Manager, RMResearcher, Member, University Professor Posts: 1,642  RM Founder
    TBH, I just had a VERY brief look into the relevant classes myself and it was not immediately obvious.  My hunch is that for the fifth neighbor the selection simply is based on the order in which the data points have been added to the queue, i.e. the first data point with the (same) minimal distance will be returned.  In your case that seems to be a LU case.  You could verify by shuffling the order of data points in your data (e.g. by sorting in ascending or descending order before loading the data).  I did not see any reference to random numbers so I would rule out those options...
    Here are the links:
    Hope this helps,
Sign In or Register to comment.