Options
Train/Apply Model in "Sliding" Batches
Contributor II
Hi everyone.
I am trying to do the following: Let's say I have 10000 examples in my data set. I would like to take the
first 1000 to train my learner, then apply that model on the next 1000 examples for prediction. Then "slide"
the training examples forward to 1001 to 2000, train, then apply to examples 2001 to 3000. Repeat through the
whole data set. In the end, I would have 9000 predictions. The learner is being re-trained every 1000 data points.
I've been trying the "Loop Batches" operator, but that doesn't quite do what I want. Sliding the window forward
manually is a bit tedious ( esp. if I expand this to several million examples!)
Any thoughts would be greatly appreciated.
I am trying to do the following: Let's say I have 10000 examples in my data set. I would like to take the
first 1000 to train my learner, then apply that model on the next 1000 examples for prediction. Then "slide"
the training examples forward to 1001 to 2000, train, then apply to examples 2001 to 3000. Repeat through the
whole data set. In the end, I would have 9000 predictions. The learner is being re-trained every 1000 data points.
I've been trying the "Loop Batches" operator, but that doesn't quite do what I want. Sliding the window forward
manually is a bit tedious ( esp. if I expand this to several million examples!)
Any thoughts would be greatly appreciated.
0
Answers
By the way, you can hit ctrl + I, to find any operator using text search.
I will experiment with it.