RapidMiner 9.7 is Now Available

Lots of amazing new improvements including true version control! Learn more about what's new here.

CLICK HERE TO DOWNLOAD

"Calculate confidence interval of RMSE"

wesselwessel Member Posts: 537  Guru
edited June 2019 in Help
Dear All,

I have two forecasting algorithms that output some forecast for the temperature 24 hours a head in time.
Algorithm A uses 1-nearest neighbours.
Algorithm B is a baseline algorithm, and simply outputs the last known temperature value as a prediction.

Lets say I calculate the Mean Squared Error, and the Variance of the Squared Error for A and B on a separate test set with N data points.
Then what is the confidence interval of MSE_A?
And what is the confidence interval of MSE_B?

Best regards,

Wessel
Tagged:

Answers

  • wesselwessel Member Posts: 537  Guru
    I have solved this problem as following, although I'm not sure it is correct:

    diffErrMean = baseErrMean - predErrMean;
    diffVarMean = baseVarMean + predVarMean;
    varOverSqrtN = diffVarMean / Math.sqrt(N);
    z = diffErrMean / varOverSqrtN;
    z = Math.abs(z);
    upper = diffErrMean + z * diffVarMean
    lower = diffErrMean - z * diffVarMean
    (Where B = baseline = baseErrMean, and A = algorithm = predErrMean)

    I can then print something like:
    N: 13 // number of test points
    Target: "temp"
    Run time: 0.105 ms
    predErrMean: 0.134  predVarMean: 0.067
    baseErrMean: 0.246  baseVarMean: 0.141
    diffErrMean: 0.113 +- 0.058 = [-0.003, 0.228] // kinda weird that this is already nearly significant with only 13 test points
    Ratio:  1.843
  • wesselwessel Member Posts: 537  Guru
    Okay this does not make any sense.

    You need to use the CDF of the T distribution to convert the z at the 2.5% point.

    But this is hard in Java since there is no easy access to the CDF of the T distribution.

    So for now I think I will assume the normal distribution and use confidence interval = MEAN +- 2 * S.D.

    But then the problem is:
    The differences are not normally distributed.
    The maximum difference possible with algorithm A 0 error, and the baseline some big error, then the big error would be equal to the S.D.
    And nothing would ever be significant.
Sign In or Register to comment.