How wide should the search be (Forecast, version 12)?

Howard Pong Yuen Lam · AI Notes · November 13, 2026 · 7 min read

English 繁體中文 简体中文

Version 11 searched six dials with twenty trials. Version 12 keeps the same six dials but widens every one of them, then runs 200 trials instead of 20. That change is small on paper and large in practice, because this is the point in the series where the cost of searching becomes real. Ten times the trials means roughly ten times the waiting, and the answer it buys is not ten times better.

A wider space and a bigger budget

The search space in the file is the honest source for all of this. It reads:

SEARCH_SPACE = {
    "epochs":       [100, 200, 300, 500],
    "learning_rate":[0.0001, 0.001, 0.005, 0.01, 0.05, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64, 128, 256],
    "num_layers":   [1, 2, 3, 4],
    "window_size":  [10, 20, 30, 50],
}

Count the values in each list and multiply: 4 × 6 × 3 × 5 × 4 × 4 = 5,760 possible combinations. Version 11 offered 972. The budget line underneath is N_TRIALS = 200, so the program draws 200 of those 5,760 at random and trains each one. That is a little under 3.5 per cent of the space.

What "diverged" means

R² is measured against a simple alternative: predicting the average sales value for every day. That comparison is explained in How to measure the model on data it has never seen (version 8). A score of 1.0 is a perfect forecast. A score of 0.0 is exactly as good as that average-based prediction. A negative score is worse. So a large negative number is not a near miss. It means the model's guesses were further from the truth than a flat average line would have been. In practice, the weights changed too much during training and the errors did not become smaller. The program does not treat this as an error. It records the score and moves on to the next trial.

What you should see

The real run printed 200 rows. The table below is trimmed to twelve of them: the twelve printed here include the winner, several respectable trials and four of the collapses. The Trial numbers are positions in the original run order, so they are not consecutive.

Running 200 random trials...

Trial   Win  Epochs      LR      Opt  Hidden  Layers        R2        RMSE
------------------------------------------------------------------------------
    1    30     500    0.01  RMSprop      64       2    0.4421      5.3064
    2    10     100    0.01  RMSprop      64       1    0.4529      5.3201
    4    20     200    0.01      SGD      10       4  -341.2371    132.4929
    9    50     500     0.1     Adam      32       2   -91.9151     68.3195
   14    10     100   0.001     Adam      32       2    0.4612      5.2795
   27    10     300   0.005     Adam     256       4    0.4616      5.2777
   32    20     100  0.0001     Adam     128       2    0.4646      5.2406
   70    20     300  0.0001  RMSprop     256       3    0.4790      5.1695
   89    20     200   0.005      SGD      10       2  -465.9302    154.7587
   95    20     500     0.1      SGD      64       2  -470.9389    155.5865
  168    10     200    0.05     Adam      64       4    0.4660      5.2561
  185    30     100   0.001  RMSprop     256       3    0.4733      5.1559

Best model saved → best_model_12.pt
Best config : {'epochs': 300, 'learning_rate': 0.0001, 'optimizer': 'RMSprop', 'hidden_size': 256, 'num_layers': 3, 'window_size': 20}

Test Set Metrics:
  MAE:  4.1235
  MSE:  26.7236
  RMSE: 5.1695
  R²:   0.4790

Results table saved → search_results_12.xlsx

Trial 70 is the winner: 300 epochs, learning rate 0.0001, RMSprop, 256 hidden units, 3 layers and a window of 20. Early stopping ended its training after 80 of the 300 allowed epochs, so the extra epochs in the setting were never used.

Two hundred trials bought 0.0217 of R². Version 11's best was 0.4573 and this one is 0.4790, so ten times the computing moved the score by 0.0217.

Best-of-20 and best-of-200 are not the same measurement. A larger draw has more chances of catching a lucky configuration.

The winner takes very small steps. The smallest learning rate on offer, 0.0001, won with 0.4790.

The window answer also moved: version 11 pointed at a window of 10, while this larger search points at 20.

Loss curves of the winning trial, with a training line and a validation line drawn by the program
Loss curves of the winning trial, drawn by the program's own plotting code. Training stopped after 80 of the 300 allowed epochs, when early stopping had seen no improvement for 30 epochs.
Actual values in pale grey and predicted values as a red line across the test days of the winning trial
Actual (grey) and predicted (red) values on the 217 test days of the winning trial, with a window of 20.
Bar chart of R squared for all two hundred trials sorted best to worst, with a red dashed line at the best score
R² for all 200 trials, sorted best to worst, with the best score marked. The negative bars are drawn to scale, which squeezes the good ones against the zero line.

The whole program

The file below is the complete time_12.py, unedited:

import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader, Subset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import random
import itertools

# ── Reproducibility ──────────────────────────────────────────────────────────
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

# ── Search space ─────────────────────────────────────────────────────────────
SEARCH_SPACE = {
    "epochs":       [100, 200, 300, 500],
    "learning_rate":[0.0001, 0.001, 0.005, 0.01, 0.05, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64, 128, 256],
    "num_layers":   [1, 2, 3, 4],
    "window_size":  [10, 20, 30, 50],
}
N_TRIALS            = 200
EARLY_STOP_PATIENCE = 30
BATCH_SIZE          = 32

# ── Data ─────────────────────────────────────────────────────────────────────
df = pd.read_csv("time_series.csv")
series = df["series"].values
series_tensor = torch.tensor(series, dtype=torch.float32)

def make_loaders(window_size):
    windows      = series_tensor.unfold(0, window_size, 1)
    features     = windows[:, :-1]
    targets      = windows[:, -1:]
    full_dataset = TensorDataset(features, targets)
    total        = len(full_dataset)
    tr_end       = int(total * 0.70)
    val_end      = int(total * 0.85)
    train_loader = DataLoader(Subset(full_dataset, range(tr_end)),           batch_size=BATCH_SIZE, shuffle=True)
    val_loader   = DataLoader(Subset(full_dataset, range(tr_end, val_end)),  batch_size=BATCH_SIZE, shuffle=False)
    test_loader  = DataLoader(Subset(full_dataset, range(val_end, total)),   batch_size=BATCH_SIZE, shuffle=False)
    return train_loader, val_loader, test_loader

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ── Model factory ─────────────────────────────────────────────────────────────
def build_model(input_size, hidden_size, num_layers):
    layers = []
    in_dim = input_size
    for _ in range(num_layers):
        layers += [nn.Linear(in_dim, hidden_size), nn.ReLU()]
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 1))
    return nn.Sequential(*layers).to(device)

def get_optimizer(name, params, lr):
    return {"Adam": optim.Adam, "SGD": optim.SGD, "RMSprop": optim.RMSprop}[name](params, lr=lr)

# ── Train one trial ───────────────────────────────────────────────────────────
def train_trial(cfg):
    train_loader, val_loader, test_loader = make_loaders(cfg["window_size"])
    model     = build_model(cfg["window_size"] - 1, cfg["hidden_size"], cfg["num_layers"])
    criterion = nn.MSELoss()
    optimizer = get_optimizer(cfg["optimizer"], model.parameters(), cfg["learning_rate"])

    best_val_loss = float("inf")
    best_state    = {k: v.clone() for k, v in model.state_dict().items()}
    patience_ctr  = 0
    train_losses, val_losses = [], []

    for epoch in range(cfg["epochs"]):
        # train
        model.train()
        t_loss = 0
        for xb, yb in train_loader:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = criterion(model(xb), yb)
            loss.backward()
            optimizer.step()
            t_loss += loss.item()
        t_loss /= len(train_loader)

        # validate
        model.eval()
        v_loss = 0
        with torch.no_grad():
            for xb, yb in val_loader:
                v_loss += criterion(model(xb.to(device)), yb.to(device)).item()
        v_loss /= len(val_loader)

        train_losses.append(t_loss)
        val_losses.append(v_loss)

        # early stopping
        if v_loss < best_val_loss:
            best_val_loss = v_loss
            best_state    = {k: v.clone() for k, v in model.state_dict().items()}
            patience_ctr  = 0
        else:
            patience_ctr += 1
            if patience_ctr >= EARLY_STOP_PATIENCE:
                break

    model.load_state_dict(best_state)
    return model, train_losses, val_losses, test_loader

# ── Evaluate on test set ──────────────────────────────────────────────────────
def evaluate(model, test_loader):
    model.eval()
    preds, actuals = [], []
    with torch.no_grad():
        for xb, yb in test_loader:
            preds.extend(model(xb.to(device)).cpu().numpy())
            actuals.extend(yb.numpy())
    p = torch.tensor(np.array(preds))
    a = torch.tensor(np.array(actuals))
    mae  = torch.mean(torch.abs(p - a)).item()
    mse  = torch.mean((p - a) ** 2).item()
    rmse = torch.sqrt(torch.tensor(mse)).item()
    ss_res = torch.sum((a - p) ** 2)
    ss_tot = torch.sum((a - torch.mean(a)) ** 2)
    r2 = (1 - ss_res / ss_tot).item()
    return r2, mae, mse, rmse, preds, actuals

# ── Random search ─────────────────────────────────────────────────────────────
keys   = list(SEARCH_SPACE.keys())
combos = list(itertools.product(*SEARCH_SPACE.values()))
sampled = random.sample(combos, min(N_TRIALS, len(combos)))

results = []
best_r2    = -float("inf")
best_trial = None

print(f"Running {N_TRIALS} random trials...\n")
print(f"{'Trial':>5}  {'Win':>4}  {'Epochs':>6}  {'LR':>6}  {'Opt':>7}  {'Hidden':>6}  {'Layers':>6}  {'R2':>8}  {'RMSE':>10}")
print("-" * 78)

for i, combo in enumerate(sampled):
    cfg = dict(zip(keys, combo))
    model, t_losses, v_losses, test_loader = train_trial(cfg)
    r2, mae, mse, rmse, preds, actuals = evaluate(model, test_loader)

    row = {**cfg, "r2": round(r2, 4), "mae": round(mae, 4), "rmse": round(rmse, 4),
           "actual_epochs": len(t_losses)}
    results.append(row)

    print(f"{i+1:>5}  {cfg['window_size']:>4}  {cfg['epochs']:>6}  {cfg['learning_rate']:>6}  "
          f"{cfg['optimizer']:>7}  {cfg['hidden_size']:>6}  {cfg['num_layers']:>6}  "
          f"{r2:>8.4f}  {rmse:>10.4f}")

    if r2 > best_r2:
        best_r2    = r2
        best_trial = {"cfg": cfg, "model": model, "t_losses": t_losses,
                      "v_losses": v_losses, "preds": preds, "actuals": actuals,
                      "r2": r2, "mae": mae, "mse": mse, "rmse": rmse}

# ── Save best model ───────────────────────────────────────────────────────────
torch.save(best_trial["model"].state_dict(), "best_model_12.pt")
print(f"\nBest model saved → best_model_12.pt")
print(f"Best config : {best_trial['cfg']}")
print(f"\nTest Set Metrics:")
print(f"  MAE:  {best_trial['mae']:.4f}")
print(f"  MSE:  {best_trial['mse']:.4f}")
print(f"  RMSE: {best_trial['rmse']:.4f}")
print(f"  R²:   {best_trial['r2']:.4f}")

# ── Save results table ────────────────────────────────────────────────────────
results_df = pd.DataFrame(results).sort_values("r2", ascending=False).reset_index(drop=True)
results_df.index += 1
results_df.to_excel("search_results_12.xlsx", index_label="Rank")
print("Results table saved → search_results_12.xlsx")

# ── Plots for best model only ─────────────────────────────────────────────────
# 1. Loss curves
plt.figure(figsize=(10, 5))
plt.plot(best_trial["t_losses"], label="Train Loss")
plt.plot(best_trial["v_losses"], label="Val Loss")
plt.title(f"Best Model — Loss Curves\n{best_trial['cfg']}", fontsize=11)
plt.xlabel("Epoch"); plt.ylabel("MSE Loss")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_loss.png", dpi=150)
plt.show()

# 2. Predictions vs actual
plt.figure(figsize=(12, 5))
plt.plot(best_trial["actuals"], label="Actual",    color="lightgrey", linewidth=1.5)
plt.plot(best_trial["preds"],   label="Predicted", color="red",       linewidth=1.5)
plt.title(f"Best Model — Predictions vs Actual (Test Set)\nR²={best_trial['r2']:.4f}  RMSE={best_trial['rmse']:.4f}", fontsize=11)
plt.xlabel("Time"); plt.ylabel("Value")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_predictions.png", dpi=150)
plt.show()

# 3. R² bar chart across all trials
plt.figure(figsize=(12, 4))
colors = ["#4472C4" if r < best_r2 else "#70AD47" for r in results_df["r2"]]
plt.bar(results_df.index, results_df["r2"], color=colors)
plt.axhline(best_r2, color="red", linestyle="--", linewidth=1, label=f"Best R²={best_r2:.4f}")
plt.title("R² Across All Trials (green = best)")
plt.xlabel("Rank"); plt.ylabel("R²")
plt.legend(); plt.grid(axis="y", alpha=0.4)
plt.tight_layout()
plt.savefig("trial_r2_chart.png", dpi=150)
plt.show()

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the programs in this series so far, the time_series.csv data file, and pyproject.toml. This note uses time_12.py. Open the folder in Visual Studio Code, then type these three lines in the terminal:

uv python pin 3.12.10
uv sync
uv run time_12.py

The first line records which Python version this folder uses, and the second installs every package that pyproject.toml lists. The third line runs the 200 trials and then saves the three charts before showing them; close each window to continue.

Be warned that this one is slow. Two hundred trials take many minutes, far longer than any earlier version in the series, and a slow laptop may need a coffee. When it finishes you are left with best_model_12.pt and search_results_12.xlsx in the folder.

Those three lines are needed only once. The next time you want to run the program again, type just uv run time_12.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

What comes next

Version 13 does no searching at all. It loads best_model_12.pt and grades it, without training anything. That is the payoff of saving the winner: the expensive part is already paid for, and the file can be reopened whenever a fresh score is wanted.