H20 Deep Learning error on the first run.

hughesfleming68

I have reproducible and local random seed set and when I load a new time series, the first run is always incorrect. Subsequent runs are correct until I load a new data set and then I am back to the first run being way off and all the following runs repeatable with a more reasonable prediction. Unfortunately, I can reproduce this consistently.

The first chart is repeatable after the first run and has the same values run to run. The second chart is the first run and this only happens once. Has anyone else seen this before?

Regards

Alex

hughesfleming68

I replaced all the H20 Deep learning operators with the new Deep Learning 4J operators and now everything is working properly when I change data sets. I will stick to these.

varunm1

@hughesfleming68 any preliminary thoughts on why this is happening? Is this something to do with weights initialization?
These look like very big deviations.
Thanks

hughesfleming68

Hi Varun, I don't know why the first run is different from every subsequent run. I put together a two layer MLP in in DeepLeaning 4J and that just works. With random seed and single thread set, the H20 deep learner should give identical results from the first run. The only way to really check is to run H20 from R and see if the same thing happens. 

hughesfleming68

I am pretty sure now that my problems started with how I was storing and retrieving results to get around meta data problems. I have taken them all out and everything is working. The odd thing is that the store operator was not over writing the previous run so my new data set was being combined with the prediction from the previous one.