Options
PLEASE HELP: Sliding Window Validation (with Custom Training Windows)
Learner I
Hi All-
[I apologize in advance for the lengthy question! My back is to the wall regarding this issue, so I wanted to provide as much detail as possible to make it easier for more knowledgeable folks to understand what I'm doing and hopefully provide assistance. Also, the scaling on some of the embedded images is off -- please bear with me!]
CONTEXT:
I've been working with the Sliding Window Validation (SWV) operator for a project. Thus far, I've been using the "basic" functionality in order to implement "sliding window validation" (see diagram below):
PROCEDURE:
To this end, I've been I've setting the SWV operator's parameters in this manner: Training Window Size to 160 periods, Step Size to 1, and Test Window Size to 1 (see below):
[NOTE: As I understand it, in this configuration, the SWV operator trains the model on 160 observations (starting at observation #1), tests the model on the next single observation (#161) and then repeats -- starting with observation #2.]
I'll touch on it below, but I wanted to experiment with other time series validation methods and have been looking into the SWV operator's Custom Windows parameters. I based my attempts on the Custom Window tutorial process:
I figured the best way to start down this path was to construct custom window input settings that mirror the "basic" parameter set I was using (described above).
Following the tutorial process, I created an Example Set with Dates to replicate the "basic" parameters:
I connected that Example Set to the SWV operator's "cus" input port and I configured its parameters thusly:
[NOTE: Since my Test Window is a single observation wide I used the same attribute (and date) for the Test Window Start and Test Window Stop values.]
An error was thrown:
I tried a different Example Set for the Custom Windows:
and slightly different parameters for the SWV operator:
Of course, the question with this approach is whether or not the Test Window is now 2 observations wide (more on that later)...
The following happened... The results using this revised SWV operator configuration were different that the "basic" ones I detailed above.
In addition, the results were different than configuring the SWV operator to have a Test Window Size of 2 (like below):
QUESTIONS/PLEA FOR HELP:
Can someone help me please understand what's going on with my custom windows inputs/parameters:
===> In addition, I'm trying to implement a so-called Expanding Window Validation (see below). Can this be achieved with the SWV operator and can someone please help me get moving in that direction?
Thank you so much!
Best,
Noel
---
[I apologize in advance for the lengthy question! My back is to the wall regarding this issue, so I wanted to provide as much detail as possible to make it easier for more knowledgeable folks to understand what I'm doing and hopefully provide assistance. Also, the scaling on some of the embedded images is off -- please bear with me!]
CONTEXT:
I've been working with the Sliding Window Validation (SWV) operator for a project. Thus far, I've been using the "basic" functionality in order to implement "sliding window validation" (see diagram below):
PROCEDURE:
To this end, I've been I've setting the SWV operator's parameters in this manner: Training Window Size to 160 periods, Step Size to 1, and Test Window Size to 1 (see below):
[NOTE: As I understand it, in this configuration, the SWV operator trains the model on 160 observations (starting at observation #1), tests the model on the next single observation (#161) and then repeats -- starting with observation #2.]
I'll touch on it below, but I wanted to experiment with other time series validation methods and have been looking into the SWV operator's Custom Windows parameters. I based my attempts on the Custom Window tutorial process:
I figured the best way to start down this path was to construct custom window input settings that mirror the "basic" parameter set I was using (described above).
Following the tutorial process, I created an Example Set with Dates to replicate the "basic" parameters:
I connected that Example Set to the SWV operator's "cus" input port and I configured its parameters thusly:
[NOTE: Since my Test Window is a single observation wide I used the same attribute (and date) for the Test Window Start and Test Window Stop values.]
An error was thrown:I tried a different Example Set for the Custom Windows:
and slightly different parameters for the SWV operator:
Of course, the question with this approach is whether or not the Test Window is now 2 observations wide (more on that later)...
The following happened... The results using this revised SWV operator configuration were different that the "basic" ones I detailed above.
In addition, the results were different than configuring the SWV operator to have a Test Window Size of 2 (like below):
QUESTIONS/PLEA FOR HELP:
Can someone help me please understand what's going on with my custom windows inputs/parameters:
- Is it possible to configure a Test Window size of 1?
- What is going on behind the scenes with my second set of custom window inputs (and why does it not mirror a "basic" configuration with Test Window size of 2?
===> In addition, I'm trying to implement a so-called Expanding Window Validation (see below). Can this be achieved with the SWV operator and can someone please help me get moving in that direction?
Thank you so much!
Best,
Noel
---
0
