Options
Prediction accuracy?
xiaobo_sxb
Member Posts: 17 
Contributor II
Hi
I tried to predict time series data by following the vedio and many other topics here. The accuracy is calculated by the "% validation" and "% forecast performance". The result "predict trend accuracy" actually describe the trend accuracy, which is usful if I predict next day stock price. But if I want to predict sales for future months, then it's not enough to describe the accuracy by trend only. I'm thinking of using something like root mean squared error. I tried to replace the operator "% forecast performance" with operator "performance" but the result looks strange. Anybody knows how to do that correctly? What the average root mean squared error is calculated then? Will the " root mean squared error" been simply averaged over all iterations?
Best Regards
Steven
I tried to predict time series data by following the vedio and many other topics here. The accuracy is calculated by the "% validation" and "% forecast performance". The result "predict trend accuracy" actually describe the trend accuracy, which is usful if I predict next day stock price. But if I want to predict sales for future months, then it's not enough to describe the accuracy by trend only. I'm thinking of using something like root mean squared error. I tried to replace the operator "% forecast performance" with operator "performance" but the result looks strange. Anybody knows how to do that correctly? What the average root mean squared error is calculated then? Will the " root mean squared error" been simply averaged over all iterations?
Best Regards
Steven
Tagged:
0
Answers
I've traded Forex ( mainly futures ) for over a decade, and use Rapidminer to help me explore that murky world. In the early years I looked for perpetual patterns, because the rational expectation theory said they should be there. Bumpy, bumpy, very very bumpy! I then tried only using the recent past to learn transient patterns, the sliding window approach. Things got a lot smoother, even until 2009 when some ancient old duffers who still believed in rational expectation theory applied as medicine precisely that which had poisoned the system in the first place, the drivel of the normal distribution and riskless return assumptions, which had underpinned derivative pricing. Nonsense when you think about it...
My stop loss policy got me out while I was still ahead, and I paused to reflect. In brief, what I had learnt was that transient patterns were detectable by support vector machines, but making use of those patterns was quite tricky, precisely because any performance measure has its weak spots. Ridiculously extensive grinding of the data had led me to using binominal classification - " up or down?", over various time horizons. That showed 5 to 7 days out on EURUSD was ~68% correct. The risk was that some old fools would intervene with their silly magic tricks, which sadly happened and we look at their continuing efforts and startling success. Haddock may well be mad, but he is not the only one saying this...
" the many aspects of the modern economy that the rational expectations approach cannot by its nature explain " Phelps, 2006 Economics Nobel.
" Anyone who has ever studied markets for financial assets such as currencies knows that it is very difficult to explain, much less predict, short to medium term fluctuations. " Rogoff, Harvard
" The rational expectations hypothesis was a bold and ingenious attempt, but it has proved empirically very far from satisfactory, most strikingly in the field of foreign exchange markets.." Arrow, another Nobel
All from the cover of " Imperfect Knowledge Economics " by Frydman and Goldberg.
As for Haddock, he continues to plough a lonely furrow across the oceans of data, exploring the one minute level and below, using evidence rather than blind faith to winkle out those naughtily profitable and transient associations.
The bottom line seems to be that pretty well any economic forecasting that depends on regression, rather than classification, must have serious weaknesses, whatever the timeframe. Globalisation means that nothing is an independent asset, which has interesting implications for dataminers!
But each to his own, I say! What I make someone else loses.
Toodles