I found that the optimized parameter given by optimize parameters actually doesn't have the highest precision or accuracy. So just wonder how the optimal value is decided? Or further can user define his own rules? Thanks.
Hi, there are different optimize parameters operators - which one did you use? Some of them do not garantee to find the best parameter combination, but only a "probably quite good" solution (like evolutionary parameter optimization). Furthermore, how did you find out that your parameters don't deliver the best performance? If you tested on a different dataset than you trained with, the results may vary slightly, even if the data are from the same distribution. And there are some other things to consider, maybe this thread can help you: http://rapid-i.com/rapidforum/index.php/topic,4018.msg14881.html
I put X-validation inside Optimize Parameters Grid, SVM inside X-validation (10 fold). For example, I set up SVM.C in [1,5] with 5 steps, precision as the main criteria for Performance. The result shows SVM.C=2.0. However when I draw the precision series curve in log, 2.0 doesn't has the highest precision. That's where my question comes from. Thanks.
Which performance does your Log operator log? If it logs the performance of the X-Validation, in your case 2.0 should indeed also be the maximum of the plot. Logging the values from the Performance operator inside the X-Validation won't work beause of the reasons stated in the other thread.
unfortunately, I put the log outside X-validation, which should work. I post my codes here just in case you had time to take a quick look. Thanks a lot
indeed it is correct to put the log outside the XVal, however, you are logging the performance of the Performance operator, which delivers the performance of its last application, i.e. in the last iteration of the XVal. To get the average result of the XVal (which is what you want), log the performance of the XVal operator itself.
Answers
there are different optimize parameters operators - which one did you use? Some of them do not garantee to find the best parameter combination, but only a "probably quite good" solution (like evolutionary parameter optimization).
Furthermore, how did you find out that your parameters don't deliver the best performance? If you tested on a different dataset than you trained with, the results may vary slightly, even if the data are from the same distribution. And there are some other things to consider, maybe this thread can help you: http://rapid-i.com/rapidforum/index.php/topic,4018.msg14881.html
Cheers, Marius
I put X-validation inside Optimize Parameters Grid, SVM inside X-validation (10 fold).
For example, I set up SVM.C in [1,5] with 5 steps, precision as the main criteria for Performance.
The result shows SVM.C=2.0. However when I draw the precision series curve in log, 2.0 doesn't has the highest precision.
That's where my question comes from.
Thanks.
Cheers,
Marius
indeed it is correct to put the log outside the XVal, however, you are logging the performance of the Performance operator, which delivers the performance of its last application, i.e. in the last iteration of the XVal. To get the average result of the XVal (which is what you want), log the performance of the XVal operator itself.