"leave_one_out_performance_problem"

bojansimoskibojansimoski Member Posts: 2 Contributor I
edited June 2019 in Help
Hello guys,
so i'm using X-validation for my analysis and i have one question about interpreting the results i have from the performance operator.. So for the accuracy of the classifier i have something like : accuracy: 65.38% +/-  36.08% ; And my question is about the second argument i have here : 36.08% ... What is this? And how is computed ? I need to mention that i use leave one out technique ..
Many Thanks!!
Tagged:

Answers

  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    With leave one out on N examples, you train N models, each on (N-1) examples and measure its performance (its accuracy in this case) on the remaining example.
    The first part of the displayed accuracy is the mean accuracy of all N models, and the second part is the standard deviation.

    Best,
      Marius
  • bojansimoskibojansimoski Member Posts: 2 Contributor I
    Thanks, and can you or someone else explain me how is the standard deviation computed in this case?
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Yes, as usual it's basically the square root of the difference of the mean of squares and the squared mean of the performance values of each iteration. There is a wikipedia article about the standard deviation: http://en.wikipedia.org/wiki/Standard_deviation

    Best, Marius
  • AMTAMT Member Posts: 7 Contributor II
    I do not get. Because in that case you have only one element in the validation set so the result will be 0 (incorrect) and 1 (correct)...
    And interpreting the results in that situation they are strange.
    I got results
    84.26 +/- 36.08  or 63.38 +/- 47.57


    and if in both cases I assume that this standart deviation is computed as sqrt(p(1-p)). Taking as p=accuracy (so p=0.8426. for instance) I got then the value 0f the standard deviation shown . In the example sqrt(0.8426(1-0.8426)).  But this I think is not ok, bacause accuracy is not a bernoulli distribution. I think the value should be further divided by sqrt(N)....  So my question is as Bojan how is this standard deviation computed?

    thank you?
    AMT
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Please remember that in a leave-one-out validation of a set with n examples, you have n iterations. If you have a look at the definition e.g. at the wikipedia article linked above, you see that you are missing some sum and sqrt operations in your formula.

    Best,
      Marius
  • AMTAMT Member Posts: 7 Contributor II
    Sorry I do not get your answer. The definitions that are in wikipedia are standard way to estimate the variance and std. But if in each iteration I got real numbers, like 0.7, 0.8, 0.9 and so on...
    But here I do not think that it is what it was used.  With one example you got correct and non-correct.
    At the the end of the n iterations, a count variable with a binomial distribution is obtained as at each iteration a bernoulli distribution.
    And what I was pointing it is that this standard deviation seems to be estimated using the formulas of the standard deviation for a bernoulli distribution ----- sqrt(p(1-p))) ------  and this I did not found in wikipedia page you point.  So how it is really estimated the standard deviation. 
    Another point it is how you interpret a result like the ones I showed where performance can have such large spread? Even being larger than 100%?
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Hi,

    you can transform: p(1-p) = p - p^2, which is equivalent to the standard formula for the standard deviation where the values are only 0 or 1.

    Best,
      Marius
  • AMTAMT Member Posts: 7 Contributor II
    You mean the variance, is not the standard deviatiion is the square root...

    But this is the point. I think that to compute the std (standard deviation) of the accuracy you need further divide by sqrt(n) ...  What do you think?

    Greetings

    A.M. Tomé
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    AMT wrote:

    You mean the variance, is not the standard deviatiion is the square root...
    Sure, consider this as a typo :)
    But this is the point. I think that to compute the std (standard deviation) of the accuracy you need further divide by sqrt(n) ...  What do you think?
    Now I got you. Seems to be reasonable. What we are currently displaying is the standard deviation of the accuracy values, not the stddev of the mean accuracy value, as far as I see it. In the default case where you have continuous accuracies this is often what you are interested in, since it indicates the robustness of your model - if you have a large standard deviation in your performances, this indicates that the model does not generalize well and might be overfitted (or simply is not suited for your data, or ...).

    With accuracy values in 0 and 1 the usefulness of this value is certainly questionable. Same applies to the +- notation, since it's not the error of the accuracy.
    We will discuss that here at Rapid-I. Thanks for your input!

    Best,
      ~Marius
  • AMTAMT Member Posts: 7 Contributor II

    Any new about this comment?

    AMT
  • MariusHelfMariusHelf RapidMiner Certified Expert, Member Posts: 1,869 Unicorn
    Not yet, I am sorry. However, I created an internal ticket such that we don't lose track of this issue again.

    Best regards,
    Marius
Sign In or Register to comment.