RapidMiner 9.8 Beta is now available
Be one of the first to get your hands on the new features. More details and downloads here:
Why Does Forecast Validation give Larger Price Difference Predictions than Apply Forecast?
When the Dow Jones can regularly move 100's of points a day, the Forecast Validation operator appears to reflect this real price movement a little better and closing price differences range by about 30 points (pls see first image), whereas when I use the Apply Forecast operator it gives near flat line future predictions with virtually little or no change in price from the previous days bar a few points, which is of little use?
One explanation I've heard is that the the best forecast is just a flat line because there is no proper pattern in the data, and the zig-zag in my input data is just noise which cannot be predicted. My open/high/low/close data does also include many technical indicators including exponential smoothing. (No Fourier Transform has been done on this time series as I had trouble hooking up that operator using a working FFT process).
If this is the case I was wondering why is the Forecast Validation operator producing what appears to slightly more valid and price movements?
Here is the ARIMA Forecast Validation results (without the Apply Forecast)
The second image shows the Dow moving 5.5% over a 14 day period but with flat future price projections from Apply Forecast:
As can be seen the Apply Forecast doesn't move much from 27,950 to 27,970 but this certainly wasn't the reality with the Dow price range during the same period -- mid August to the beginning of Sept 2020:
(I tried different ARIMA p,d and q, values that made little difference).
What can I do to make it more useable for predicting actual future prices? I have an example here using Window = 2500, Step = 30 and Horizon = 5 during the Covid crash where predicted prices were realistic in capturing the large changes between consecutive closing prices of the Dow: