How to let the program choose the window length too (Forecast, version 11)?

Howard Pong Yuen Lam · AI Notes · November 13, 2026 · 6 min read

English 繁體中文 简体中文

This is version 11 of the forecasting series. Version 10 searched five settings: how long to train, how big a step to take, which optimiser to use, how wide the hidden layer should be, and how many layers to stack. Version 11 adds a sixth, the window length — how long a run of past values each example is built from, in other words how far back the model looks before making its guess. That one is not like the others. The first five change the model; the sixth changes the data.

Six settings and 972 combinations

The search space in version 11 is the old one with a single line added. The old five settings still offer three values each; the new one offers four.

SEARCH_SPACE = {
    "epochs":       [50, 100, 200],
    "learning_rate":[0.001, 0.01, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64],
    "num_layers":   [1, 2, 3],
    "window_size":  [10, 20, 30, 50],
}

Three values in five lists and four in the last gives 3 × 3 × 3 × 3 × 3 × 4 = 972 combinations, up from 243. The budget is unchanged at N_TRIALS = 20, so the same twenty random draws now cover four times as much ground. That is the cost of adding a sixth setting.

Why the window length is different from the other settings

From version 2 onwards the window length was a fixed line near the top of the file, WINDOW_SIZE = 30. Everyone who ran the program trained on the same examples. In version 11 that line is gone, and window_size arrives as one of the settings.

It has three consequences, and only the first is obvious. Changing the window changes how many examples exist: 1,460 days cut into windows of length w gives 1,460 − w + 1 examples, which is 1,431 for w = 30, as worked through in How to split each window into features and a target (version 4). It also changes how many numbers each input holds: w − 1 of them, so 29 at a window of 30, 9 at the shortest window of 10 and 49 at the longest window of 50. And because the input length feeds the network's first layer, it changes the shape of the model itself.

How the search works

1. Rebuild the data inside every trial

Version 10 built its windows once, before the search began. Version 11 wraps the same work in a function so it can be repeated for each window length:

def make_loaders(window_size):
    windows      = series_tensor.unfold(0, window_size, 1)
    features     = windows[:, :-1]
    targets      = windows[:, -1:]
    full_dataset = TensorDataset(features, targets)
    total        = len(full_dataset)
    tr_end       = int(total * 0.70)
    val_end      = int(total * 0.85)

unfold slides a frame of the requested length along the series, and the last number of each frame becomes the answer while the rest become the question. The split points are then recomputed from the new total, so the proportions hold at every window length. The function's first caller is the training routine itself: train_loader, val_loader, test_loader = make_loaders(cfg["window_size"]).

2. Let the first layer change size

The model factory now takes the input size as an argument instead of reading a constant. In version 10 the line inside it was input_size = WINDOW_SIZE - 1; in version 11 the number is handed in from outside:

def build_model(input_size, hidden_size, num_layers):
    layers = []
    in_dim = input_size
    for _ in range(num_layers):
        layers += [nn.Linear(in_dim, hidden_size), nn.ReLU()]
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 1))
    return nn.Sequential(*layers).to(device)

The layers are built exactly as explained in How to let the program search its own settings (version 10). The only change is that the number of input values is now passed in from outside instead of being read from a fixed constant, so the first layer takes 9, 19, 29 or 49 numbers depending on the window length.

3. Train and grade each combination

for i, combo in enumerate(sampled):
    cfg = dict(zip(keys, combo))
    model, t_losses, v_losses, test_loader = train_trial(cfg)
    r2, mae, rmse, preds, actuals = evaluate(model, test_loader)

The configuration now carries six values, for example {"epochs": 50, "learning_rate": 0.001, ..., "window_size": 10}. This dictionary is stored in cfg, short for configuration: one complete set of settings for a single trial. Training works as in version 9: a trial stops early if the validation error does not improve for 10 epochs, and the weights that gave the lowest validation error are kept. A trial that stops improving therefore ends early instead of running to the maximum number of epochs. The test loader comes back out of train_trial only because it was built for that trial's window.

4. Keep the winner and write the files

if r2 > best_r2:
    best_r2    = r2
    best_trial = {"cfg": cfg, "model": model, ...}

torch.save(best_trial["model"].state_dict(), "best_model.pt")
results_df.to_excel("search_results.xlsx", index_label="Rank")

At the end, five files appear in the folder. best_model.pt holds the winning weights, search_results.xlsx holds all twenty trials ranked by R² with a window_size column that version 10 did not have, and three charts are saved as best_model_loss.png, best_model_predictions.png and trial_r2_chart.png. The spreadsheet is still the useful one, because it keeps every trial, not only the winner.

What you should see

The complete table from a real run, unedited. The Win column is the window length for that trial.

Running 20 random trials...

Trial   Win  Epochs      LR      Opt  Hidden  Layers        R2        RMSE
------------------------------------------------------------------------------
    1    30     200   0.001     Adam      10       2    0.4358      5.3364
    2    30      50    0.01     Adam      10       2    0.4138      5.4394
    3    20      50   0.001     Adam      64       1    0.4241      5.4350
    4    50     200    0.01     Adam      10       1    0.4365      5.3207
    5    20      50     0.1      SGD      64       2  -419.3689    146.8400
    6    30      50     0.1     Adam      64       3   -86.7436     66.5462
    7    10      50     0.1     Adam      32       1    0.4351      5.4060
    8    30      50    0.01     Adam      64       3    0.4424      5.3048
    9    30     200   0.001  RMSprop      64       3    0.3956      5.5230
   10    10      50   0.001  RMSprop      64       3    0.4566      5.3021
   11    10     200   0.001      SGD      10       3  -312.0232    127.2510
   12    30     200    0.01     Adam      10       1    0.4389      5.3215
   13    20     200     0.1      SGD      32       1  -835.4003    207.1267
   14    30     100     0.1     Adam      32       2    0.4367      5.3317
   15    20      50   0.001  RMSprop      32       2    0.4163      5.4716
   16    10     100     0.1      SGD      64       2  -363.2989    137.2783
   17    10     100    0.01     Adam      10       1    0.4372      5.3959
   18    10      50   0.001     Adam      64       3    0.4573      5.2983
   19    30      50   0.001     Adam      64       2    0.4389      5.3216
   20    50      50   0.001  RMSprop      32       3    0.4393      5.3073

Best model saved → best_model.pt
Best config : {'epochs': 50, 'learning_rate': 0.001, 'optimizer': 'Adam', 'hidden_size': 64, 'num_layers': 3, 'window_size': 10}
Best R²     : 0.4573  |  MAE: 4.2199  |  RMSE: 5.2983
Results table saved → search_results.xlsx

The winner looks back only ten days. Trial 18 — 50 epochs, learning rate 0.001, Adam, 64 hidden units, 3 layers, window 10 — scored R² 0.4573 with MAE 4.2199 and RMSE 5.2983. The runner-up, trial 10, also used a window of 10 and was almost identical: 0.4566. That is our first real answer to the question the search was asked. On this data, the short look-back wins.

The gain over version 10 is small. Version 10's best was 0.4511 and version 11's is 0.4573, and part of that difference is luck of the draw rather than a better idea.

Loss curves of the winning trial, dropping sharply at the first epoch and then flattening with the training and validation lines almost on top of each other
Loss curves of the winning trial. Training ended at epoch 23 of the 50 allowed, because early stopping had seen no improvement for 15 epochs.
Actual test-day sales in pale grey and predicted values as a smooth red line following the upward trend but missing the daily swings
Actual (grey) and predicted (red) values over the 218 test days with a window of 10. The model tracks the trend and smooths away the noise.
Bar chart of R squared across twenty trials sorted from best to worst, the positive bars squeezed against the zero line and a red dashed line marking the best score
R² across all twenty trials, sorted best to worst. The five bars that plunge far below zero on the right are drawn to scale, which is why the good trials are squeezed against the zero line.

The whole program

The pieces above, put back together. This is the complete file:

import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader, Subset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import random
import itertools

# ── Reproducibility ──────────────────────────────────────────────────────────
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

# ── Search space ─────────────────────────────────────────────────────────────
SEARCH_SPACE = {
    "epochs":       [50, 100, 200],
    "learning_rate":[0.001, 0.01, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64],
    "num_layers":   [1, 2, 3],
    "window_size":  [10, 20, 30, 50],
}
N_TRIALS            = 20
EARLY_STOP_PATIENCE = 15
BATCH_SIZE          = 32

# ── Data ─────────────────────────────────────────────────────────────────────
df = pd.read_csv("time_series.csv")
series = df["series"].values
series_tensor = torch.tensor(series, dtype=torch.float32)

def make_loaders(window_size):
    windows      = series_tensor.unfold(0, window_size, 1)
    features     = windows[:, :-1]
    targets      = windows[:, -1:]
    full_dataset = TensorDataset(features, targets)
    total        = len(full_dataset)
    tr_end       = int(total * 0.70)
    val_end      = int(total * 0.85)
    train_loader = DataLoader(Subset(full_dataset, range(tr_end)),           batch_size=BATCH_SIZE, shuffle=True)
    val_loader   = DataLoader(Subset(full_dataset, range(tr_end, val_end)),  batch_size=BATCH_SIZE, shuffle=False)
    test_loader  = DataLoader(Subset(full_dataset, range(val_end, total)),   batch_size=BATCH_SIZE, shuffle=False)
    return train_loader, val_loader, test_loader

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ── Model factory ─────────────────────────────────────────────────────────────
def build_model(input_size, hidden_size, num_layers):
    layers = []
    in_dim = input_size
    for _ in range(num_layers):
        layers += [nn.Linear(in_dim, hidden_size), nn.ReLU()]
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 1))
    return nn.Sequential(*layers).to(device)

def get_optimizer(name, params, lr):
    return {"Adam": optim.Adam, "SGD": optim.SGD, "RMSprop": optim.RMSprop}[name](params, lr=lr)

# ── Train one trial ───────────────────────────────────────────────────────────
def train_trial(cfg):
    train_loader, val_loader, test_loader = make_loaders(cfg["window_size"])
    model     = build_model(cfg["window_size"] - 1, cfg["hidden_size"], cfg["num_layers"])
    criterion = nn.MSELoss()
    optimizer = get_optimizer(cfg["optimizer"], model.parameters(), cfg["learning_rate"])

    best_val_loss = float("inf")
    best_state    = {k: v.clone() for k, v in model.state_dict().items()}
    patience_ctr  = 0
    train_losses, val_losses = [], []

    for epoch in range(cfg["epochs"]):
        # train
        model.train()
        t_loss = 0
        for xb, yb in train_loader:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = criterion(model(xb), yb)
            loss.backward()
            optimizer.step()
            t_loss += loss.item()
        t_loss /= len(train_loader)

        # validate
        model.eval()
        v_loss = 0
        with torch.no_grad():
            for xb, yb in val_loader:
                v_loss += criterion(model(xb.to(device)), yb.to(device)).item()
        v_loss /= len(val_loader)

        train_losses.append(t_loss)
        val_losses.append(v_loss)

        # early stopping
        if v_loss < best_val_loss:
            best_val_loss = v_loss
            best_state    = {k: v.clone() for k, v in model.state_dict().items()}
            patience_ctr  = 0
        else:
            patience_ctr += 1
            if patience_ctr >= EARLY_STOP_PATIENCE:
                break

    model.load_state_dict(best_state)
    return model, train_losses, val_losses, test_loader

# ── Evaluate on test set ──────────────────────────────────────────────────────
def evaluate(model, test_loader):
    model.eval()
    preds, actuals = [], []
    with torch.no_grad():
        for xb, yb in test_loader:
            preds.extend(model(xb.to(device)).cpu().numpy())
            actuals.extend(yb.numpy())
    p = torch.tensor(np.array(preds))
    a = torch.tensor(np.array(actuals))
    mae  = torch.mean(torch.abs(p - a)).item()
    rmse = torch.sqrt(torch.mean((p - a) ** 2)).item()
    ss_res = torch.sum((a - p) ** 2)
    ss_tot = torch.sum((a - torch.mean(a)) ** 2)
    r2 = (1 - ss_res / ss_tot).item()
    return r2, mae, rmse, preds, actuals

# ── Random search ─────────────────────────────────────────────────────────────
keys   = list(SEARCH_SPACE.keys())
combos = list(itertools.product(*SEARCH_SPACE.values()))
sampled = random.sample(combos, min(N_TRIALS, len(combos)))

results = []
best_r2    = -float("inf")
best_trial = None

print(f"Running {N_TRIALS} random trials...\n")
print(f"{'Trial':>5}  {'Win':>4}  {'Epochs':>6}  {'LR':>6}  {'Opt':>7}  {'Hidden':>6}  {'Layers':>6}  {'R2':>8}  {'RMSE':>10}")
print("-" * 78)

for i, combo in enumerate(sampled):
    cfg = dict(zip(keys, combo))
    model, t_losses, v_losses, test_loader = train_trial(cfg)
    r2, mae, rmse, preds, actuals = evaluate(model, test_loader)

    row = {**cfg, "r2": round(r2, 4), "mae": round(mae, 4), "rmse": round(rmse, 4),
           "actual_epochs": len(t_losses)}
    results.append(row)

    print(f"{i+1:>5}  {cfg['window_size']:>4}  {cfg['epochs']:>6}  {cfg['learning_rate']:>6}  "
          f"{cfg['optimizer']:>7}  {cfg['hidden_size']:>6}  {cfg['num_layers']:>6}  "
          f"{r2:>8.4f}  {rmse:>10.4f}")

    if r2 > best_r2:
        best_r2    = r2
        best_trial = {"cfg": cfg, "model": model, "t_losses": t_losses,
                      "v_losses": v_losses, "preds": preds, "actuals": actuals,
                      "r2": r2, "mae": mae, "rmse": rmse}

# ── Save best model ───────────────────────────────────────────────────────────
torch.save(best_trial["model"].state_dict(), "best_model.pt")
print(f"\nBest model saved → best_model.pt")
print(f"Best config : {best_trial['cfg']}")
print(f"Best R²     : {best_trial['r2']:.4f}  |  MAE: {best_trial['mae']:.4f}  |  RMSE: {best_trial['rmse']:.4f}")

# ── Save results table ────────────────────────────────────────────────────────
results_df = pd.DataFrame(results).sort_values("r2", ascending=False).reset_index(drop=True)
results_df.index += 1
results_df.to_excel("search_results.xlsx", index_label="Rank")
print("Results table saved → search_results.xlsx")

# ── Plots for best model only ─────────────────────────────────────────────────
# 1. Loss curves
plt.figure(figsize=(10, 5))
plt.plot(best_trial["t_losses"], label="Train Loss")
plt.plot(best_trial["v_losses"], label="Val Loss")
plt.title(f"Best Model — Loss Curves\n{best_trial['cfg']}", fontsize=11)
plt.xlabel("Epoch"); plt.ylabel("MSE Loss")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_loss.png", dpi=150)
plt.show()

# 2. Predictions vs actual
plt.figure(figsize=(12, 5))
plt.plot(best_trial["actuals"], label="Actual",    color="lightgrey", linewidth=1.5)
plt.plot(best_trial["preds"],   label="Predicted", color="red",       linewidth=1.5)
plt.title(f"Best Model — Predictions vs Actual (Test Set)\nR²={best_trial['r2']:.4f}  RMSE={best_trial['rmse']:.4f}", fontsize=11)
plt.xlabel("Time"); plt.ylabel("Value")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_predictions.png", dpi=150)
plt.show()

# 3. R² bar chart across all trials
plt.figure(figsize=(12, 4))
colors = ["#4472C4" if r < best_r2 else "#70AD47" for r in results_df["r2"]]
plt.bar(results_df.index, results_df["r2"], color=colors)
plt.axhline(best_r2, color="red", linestyle="--", linewidth=1, label=f"Best R²={best_r2:.4f}")
plt.title("R² Across All Trials (green = best)")
plt.xlabel("Rank"); plt.ylabel("R²")
plt.legend(); plt.grid(axis="y", alpha=0.4)
plt.tight_layout()
plt.savefig("trial_r2_chart.png", dpi=150)
plt.show()

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the programs in this series so far, the time_series.csv data file, and pyproject.toml. This note uses time_11.py. Open the folder in Visual Studio Code, then type these three lines in the terminal:

uv python pin 3.12.10
uv sync
uv run time_11.py

The first line records which Python version this folder uses, and the second installs every package that pyproject.toml lists. The third line runs the twenty trials, rebuilds the data once per window length, and then saves the charts to disk before showing them; close each window to continue. At the end you have best_model.pt, search_results.xlsx and three PNG files.

Those three lines are needed only once. The next time you want to run the program again, type just uv run time_11.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

What comes next

This search was still a small one: a few dozen trials, on a short list of values. Version 12 widens every range — more epochs, more learning rates, wider hidden layers, more layers — and runs 200 trials instead of 20, so the table becomes long enough to read as a landscape rather than a sample.