LSTM Deep Learning Forecast - Validation

OprickOprick Member Posts: 19 Contributor II
edited June 15 in Help
Hello,
Yesterday I posted here a question about a walkforward validation of forecasting model that was solved.
But @SGolbert pointed that LSTMs are becomimg very important in forecasting and because I've already start studying this subject I decided it was time to give it a try. Results are quite fair though tuning is more thorny.

I'm using this extension operators:
https://marketplace.rapidminer.com/UpdateServer/faces/product_details.xhtml?productId=rmx_deeplearning

My question is basically if it is possible to validate the models built using this extension with, for example, sliding window validation operator.
When I try to connect mod outport with validation operator mod ports I get a error. I understand the reason of the error, but I'm stucked. 

How can we backtest models build under this extension.

Enclosed mock example set and process.

Thanks for your help

Best Answer

Answers

  • lionelderkrikorlionelderkrikor Moderator, RapidMiner Certified Analyst, Member Posts: 748   Unicorn
    HI @Oprick,

    I just imported and executed the process you shared with your data and ..... everything works fine...
    Can you tell me more about your error ?

    Regards,

    Lionel
  • OprickOprick Member Posts: 19 Contributor II
    Hi,
    Thanks @hughesfleming68
    I saw the red warning at mod port and didn't try to run the process :S  Dumb be.

    Thanks a lot also for the tips.

    Regards,
    Pedro
    sgenzer
  • SGolbertSGolbert RapidMiner Certified Analyst, Member Posts: 336   Unicorn
    edited April 25
    You really need an expensive GPU for that. I've trained a simple RNN model based on reading for one stock of the Nasdaq exchange from the last 5 years, and it took good 20 minutes to train with CPU (csv file was 65 kb!). In comparison, a VAR model for the same task trains almost instantenously, and it's a good model.

    I think Deep Learning makes sense only when the task is intrinsictly very complex, so that simple models just don't cut it.

    Regards,
    Sebastian
    sgenzer
  • hughesfleming68hughesfleming68 Member Posts: 239   Unicorn
    Hi @SGolbert, one advantage of CNN's for time series is that they are generally much lighter for similar or better results than LSTM. Worth checking out for the time savings. In many cases a GPU is not necessary. I am using the CPU with Tensorflow and it is not unbearable in terms of training time.

    regards,

    Alex
  • SGolbertSGolbert RapidMiner Certified Analyst, Member Posts: 336   Unicorn

    the problem is that you have several parameters and NN architectures. If you really want to optimize it, it becomes impossible (unless you plan on taking a holiday). If you want to train a given NN with given parameters, then it is doable, but it's not a realistic situation.

    Do you have some experience with pre-trained networks?

    Regards,
    Sebastian

  • hughesfleming68hughesfleming68 Member Posts: 239   Unicorn
    Hi @SGolbert, I agree with you on the time it takes to optimize. It is a part of your life that you never get back! On the other hand, there are known network architectures like AlexNet, LeNet, RezNet and in my case WaveNet which provide good starting points for many different tasks. It makes more sense to fine tune a known architecture than to build a complex one from scratch.

    I have no experience with pre-trained networks except from what I have read.

    Regards,

    Alex
    SGolbertvarunm1
  • WTWT Member Posts: 4 Contributor I
    edited September 4
    Hi Alex @hughesfleming68  following your post above I have looked at Wavenet for time series prediction. My project involves using both the underlying stock price and other time series data with the same frequency i.e. daily (alpha, sector performance and some technical indicators) to predict the stock price five days forward. My questions are a) is there an existing implementation of Wavenet in Rapidminer already? b) if not, are there any existing time dilated causal neural network implementations or libraries which could solve for this problem in Rapidminer? Thank you - any assistance welcome as I could not find those answers in any of the documentation or online tutorials.

  • hughesfleming68hughesfleming68 Member Posts: 239   Unicorn
    edited September 5
    Hi WT, there isn't an existing off the shelf implementation of wavenet or modified wavenet architectures in Rapidminer. You can however do multivariate CNN with the DL4J Deep Learning extension but unless something has changed, dilation is not part of the extension. That may or may not be a problem if you intend to use multiple attributes. I have had good (repeatable) results with the Deep learning extension. 

    What you can do and what I use is Tensorflow inside Rapidminer with the execute python operator which opens more possibilities. Tensorflow is convenient because you can find lots of code on Github which makes learning by example possible. For an example of dilated cnn for time series forecasting, I would look at seriesnet.py by Krist Papadopoulos which he has kindly made available on Github. https://github.com/kristpapadopoulos/seriesnet. You would however have to install Anaconda, Keras and Tensorflow. That can be quite a big jump for many users unless you already have that working. There are also a few extra things to learn about getting repeatable results in Tensorflow and it is slow. There are also gated convolution NN's and at least five or six variations that I can think of. You might have to explore quite a few different options until you find a technique that works for you.

    regards,

    Alex





    TghadiallyWTmschmitz
  • WTWT Member Posts: 4 Contributor I
    Thank you Alex - this is super helpful and I so appreciate this as its been harder than I thought to get help on this so thrilled to read your reply today and very grateful. I will try those possible solutions and sure I can find one that will work. Kind regards Warren
    Tghadially
Sign In or Register to comment.