How to let the program search its own settings (Forecast, version 10)?

Howard Pong Yuen Lam · AI Notes · November 13, 2026 · 7 min read

English 繁體中文 简体中文

This is version 10 of the forecasting series. Version 9 compared three learning rates by hand. Version 10 lets the program do the comparison: it lists the choices for five settings, works out every combination, tries twenty of them at random, keeps the best model and saves the whole comparison as a spreadsheet. The training loop itself is unchanged; what changes is who chooses the settings.

Five settings and 243 combinations

SEARCH_SPACE = {
    "epochs":       [50, 100, 200],
    "learning_rate":[0.001, 0.01, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64],
    "num_layers":   [1, 2, 3],
}

Each key is a setting and each list is the values worth trying. Three values in five lists give 3 × 3 × 3 × 3 × 3 = 243 possible combinations.

Trying all 243 would take hours for an improvement that would be hard to notice. Instead the program draws N_TRIALS = 20 combinations at random from the 243. That is enough to show which combinations perform badly, at a much lower cost.

How the search works

1. Make every combination, then draw twenty

keys   = list(SEARCH_SPACE.keys())
combos = list(itertools.product(*SEARCH_SPACE.values()))
sampled = random.sample(combos, min(N_TRIALS, len(combos)))

itertools.product builds all 243 combinations, taking one value from each list in turn. random.sample then picks 20 of them without repeating any. The seed is fixed at the top of the file, so the twenty chosen here are the same twenty on every run.

# ── Reproducibility ──────────────────────────────────────────────────────────
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

2. Turn one combination into a model

def build_model(hidden_size, num_layers):
    layers = []
    in_dim = input_size
    for _ in range(num_layers):
        layers += [nn.Linear(in_dim, hidden_size), nn.ReLU()]
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 1))
    return nn.Sequential(*layers).to(device)

Version 7's network was spelled out layer by layer. This version builds it from two numbers instead. The loop adds one hidden layer per num_layers, each of width hidden_size, and the last line outside the loop adds the single output: layers.append(nn.Linear(in_dim, 1)). With hidden_size = 64 and num_layers = 3, the result is the three-layer network of the winning trial.

3. Train and grade each trial

for i, combo in enumerate(sampled):
    cfg = dict(zip(keys, combo))
    model, t_losses, v_losses = train_trial(cfg)
    r2, mae, rmse, preds, actuals = evaluate(model)

zip(keys, combo) pairs each setting name with its value, and dict makes that a labelled configuration such as {"epochs": 50, "learning_rate": 0.001, ...}. This dictionary is stored in cfg, short for configuration: one complete set of settings for a single trial. train_trial(cfg) then builds and trains a model using those settings. Training works as in version 9: a trial stops early if the validation error does not improve for 10 epochs, and the weights that gave the lowest validation error are kept. A trial that stops improving therefore ends early instead of running to the maximum number of epochs.

4. Remember the winner and write everything out

if r2 > best_r2:
    best_r2    = r2
    best_trial = {"cfg": cfg, "model": model, ...}

torch.save(best_trial["model"].state_dict(), "best_model.pt")
results_df.to_excel("search_results.xlsx", index_label="Rank")

best_r2 starts at negative infinity, so the first trial always becomes the benchmark and later trials must beat it to replace it. At the end two files appear: the winning weights in best_model.pt, and every trial ranked by R² in search_results.xlsx. The spreadsheet is the useful part, because it shows every trial, not only the winner.

Writing the Excel file needs one extra package, openpyxl, which is why it is listed in pyproject.toml.

What you should see

Running 20 random trials...

Trial  Epochs      LR      Opt  Hidden  Layers        R2        RMSE
----------------------------------------------------------------------
    1     200   0.001     Adam      10       2    0.4358      5.3364
    2      50    0.01     Adam      10       2    0.4138      5.4394
    3      50   0.001     Adam      64       1    0.4293      5.3670
    4     200    0.01     Adam      10       1    0.4288      5.3691
    5      50     0.1      SGD      64       2  -321.8173    127.6421
    6      50     0.1     Adam      64       3  -132.1510     81.9762
    7      50     0.1     Adam      32       1    0.4507      5.2652
    8      50    0.01     Adam      64       3    0.4467      5.2846
    9     200   0.001  RMSprop      64       3    0.4481      5.2776
   10      50   0.001  RMSprop      64       3    0.3873      5.5608
   11     200   0.001      SGD      10       3  -345.0721    132.1596
   12     200     0.1      SGD      32       1  -439.2118    149.0549
   13     100     0.1     Adam      32       2   -96.2428     70.0558
   14      50   0.001  RMSprop      32       2    0.4206      5.4075
   15     100     0.1      SGD      64       2  -389.6655    140.4165
   16     100    0.01     Adam      10       1    0.4301      5.3631
   17      50   0.001     Adam      64       3    0.4511      5.2633
   18      50   0.001     Adam      64       2    0.4489      5.2737
   19      50   0.001  RMSprop      32       3    0.4479      5.2785
   20      50     0.1     Adam      10       2    0.4333      5.3481

Best model saved → best_model.pt
Best config : {'epochs': 50, 'learning_rate': 0.001, 'optimizer': 'Adam', 'hidden_size': 64, 'num_layers': 3}
Best R²     : 0.4511  |  MAE: 4.2143  |  RMSE: 5.2633
Results table saved → search_results.xlsx

Three things in that table matter more than the winning number.

Every very poor result uses a learning rate of 0.1. Trials 5, 6, 11, 12, 13 and 15 report negative R² values in the hundreds, and RMSE in the seventies and hundreds instead of around 5. A negative R² means the model performs worse than predicting the average sales value for every day. This happens when each adjustment is too large: the weights change too much, so the errors do not become smaller. Version 9 showed a mild case of this; here it is much clearer.

The Epochs column shows the maximum allowed, not the number actually used. Early stopping may have finished long before, and the spreadsheet keeps a column, actual_epochs, with the real number.

The winner is only a little better than version 9. R² of 0.4511 against 0.4394. Running twenty trials produced a small improvement on top of early stopping. The honest lesson is that tuning helps a little, and no amount of tuning made this model a good one. That is a fact about the model, not about the search.

Loss curves of the winning trial, falling and flattening with the two lines close together
Loss curves of the winning trial
Actual test-day sales in grey and predicted values in red following the trend
Actual (grey) and predicted (red) values for the test days
Bar chart of R squared across all twenty trials, the worst trials far below zero and one green bar for the best
R² across all twenty trials, green for the winner

The whole program

The pieces above, put back together. This is the complete file:

import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader, Subset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import random
import itertools

# ── Reproducibility ──────────────────────────────────────────────────────────
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)

# ── Search space ─────────────────────────────────────────────────────────────
SEARCH_SPACE = {
    "epochs":       [50, 100, 200],
    "learning_rate":[0.001, 0.01, 0.1],
    "optimizer":    ["Adam", "SGD", "RMSprop"],
    "hidden_size":  [10, 32, 64],
    "num_layers":   [1, 2, 3],
}
N_TRIALS      = 20
EARLY_STOP_PATIENCE = 15   # stop if val loss doesn't improve for this many epochs
WINDOW_SIZE   = 30
BATCH_SIZE    = 32

# ── Data ─────────────────────────────────────────────────────────────────────
df = pd.read_csv("time_series.csv")
series = df["series"].values
series_tensor = torch.tensor(series, dtype=torch.float32)

def create_sliding_windows(data, window_size):
    windows  = data.unfold(0, window_size, 1)
    features = windows[:, :-1]
    targets  = windows[:, -1:]
    return features, targets

features, targets = create_sliding_windows(series_tensor, WINDOW_SIZE)
full_dataset = TensorDataset(features, targets)

total   = len(full_dataset)
tr_end  = int(total * 0.70)
val_end = int(total * 0.85)

train_loader = DataLoader(Subset(full_dataset, range(tr_end)),          batch_size=BATCH_SIZE, shuffle=True)
val_loader   = DataLoader(Subset(full_dataset, range(tr_end, val_end)), batch_size=BATCH_SIZE, shuffle=False)
test_loader  = DataLoader(Subset(full_dataset, range(val_end, total)),  batch_size=BATCH_SIZE, shuffle=False)

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

# ── Model factory ─────────────────────────────────────────────────────────────
def build_model(hidden_size, num_layers):
    input_size = WINDOW_SIZE - 1
    layers = []
    in_dim = input_size
    for _ in range(num_layers):
        layers += [nn.Linear(in_dim, hidden_size), nn.ReLU()]
        in_dim = hidden_size
    layers.append(nn.Linear(in_dim, 1))
    return nn.Sequential(*layers).to(device)

def get_optimizer(name, params, lr):
    return {"Adam": optim.Adam, "SGD": optim.SGD, "RMSprop": optim.RMSprop}[name](params, lr=lr)

# ── Train one trial ───────────────────────────────────────────────────────────
def train_trial(cfg):
    model     = build_model(cfg["hidden_size"], cfg["num_layers"])
    criterion = nn.MSELoss()
    optimizer = get_optimizer(cfg["optimizer"], model.parameters(), cfg["learning_rate"])

    best_val_loss = float("inf")
    best_state    = {k: v.clone() for k, v in model.state_dict().items()}
    patience_ctr  = 0
    train_losses, val_losses = [], []

    for epoch in range(cfg["epochs"]):
        # train
        model.train()
        t_loss = 0
        for xb, yb in train_loader:
            xb, yb = xb.to(device), yb.to(device)
            optimizer.zero_grad()
            loss = criterion(model(xb), yb)
            loss.backward()
            optimizer.step()
            t_loss += loss.item()
        t_loss /= len(train_loader)

        # validate
        model.eval()
        v_loss = 0
        with torch.no_grad():
            for xb, yb in val_loader:
                v_loss += criterion(model(xb.to(device)), yb.to(device)).item()
        v_loss /= len(val_loader)

        train_losses.append(t_loss)
        val_losses.append(v_loss)

        # early stopping
        if v_loss < best_val_loss:
            best_val_loss = v_loss
            best_state    = {k: v.clone() for k, v in model.state_dict().items()}
            patience_ctr  = 0
        else:
            patience_ctr += 1
            if patience_ctr >= EARLY_STOP_PATIENCE:
                break

    model.load_state_dict(best_state)
    return model, train_losses, val_losses

# ── Evaluate on test set ──────────────────────────────────────────────────────
def evaluate(model):
    model.eval()
    preds, actuals = [], []
    with torch.no_grad():
        for xb, yb in test_loader:
            preds.extend(model(xb.to(device)).cpu().numpy())
            actuals.extend(yb.numpy())
    p = torch.tensor(np.array(preds))
    a = torch.tensor(np.array(actuals))
    mae  = torch.mean(torch.abs(p - a)).item()
    rmse = torch.sqrt(torch.mean((p - a) ** 2)).item()
    ss_res = torch.sum((a - p) ** 2)
    ss_tot = torch.sum((a - torch.mean(a)) ** 2)
    r2 = (1 - ss_res / ss_tot).item()
    return r2, mae, rmse, preds, actuals

# ── Random search ─────────────────────────────────────────────────────────────
keys   = list(SEARCH_SPACE.keys())
combos = list(itertools.product(*SEARCH_SPACE.values()))
sampled = random.sample(combos, min(N_TRIALS, len(combos)))

results = []
best_r2    = -float("inf")
best_trial = None

print(f"Running {N_TRIALS} random trials...\n")
print(f"{'Trial':>5}  {'Epochs':>6}  {'LR':>6}  {'Opt':>7}  {'Hidden':>6}  {'Layers':>6}  {'R2':>8}  {'RMSE':>10}")
print("-" * 70)

for i, combo in enumerate(sampled):
    cfg = dict(zip(keys, combo))
    model, t_losses, v_losses = train_trial(cfg)
    r2, mae, rmse, preds, actuals = evaluate(model)

    row = {**cfg, "r2": round(r2, 4), "mae": round(mae, 4), "rmse": round(rmse, 4),
           "actual_epochs": len(t_losses)}
    results.append(row)

    print(f"{i+1:>5}  {cfg['epochs']:>6}  {cfg['learning_rate']:>6}  "
          f"{cfg['optimizer']:>7}  {cfg['hidden_size']:>6}  {cfg['num_layers']:>6}  "
          f"{r2:>8.4f}  {rmse:>10.4f}")

    if r2 > best_r2:
        best_r2    = r2
        best_trial = {"cfg": cfg, "model": model, "t_losses": t_losses,
                      "v_losses": v_losses, "preds": preds, "actuals": actuals,
                      "r2": r2, "mae": mae, "rmse": rmse}

# ── Save best model ───────────────────────────────────────────────────────────
torch.save(best_trial["model"].state_dict(), "best_model.pt")
print(f"\nBest model saved → best_model.pt")
print(f"Best config : {best_trial['cfg']}")
print(f"Best R²     : {best_trial['r2']:.4f}  |  MAE: {best_trial['mae']:.4f}  |  RMSE: {best_trial['rmse']:.4f}")

# ── Save results table ────────────────────────────────────────────────────────
results_df = pd.DataFrame(results).sort_values("r2", ascending=False).reset_index(drop=True)
results_df.index += 1
results_df.to_excel("search_results.xlsx", index_label="Rank")
print("Results table saved → search_results.xlsx")

# ── Plots for best model only ─────────────────────────────────────────────────
# 1. Loss curves
plt.figure(figsize=(10, 5))
plt.plot(best_trial["t_losses"], label="Train Loss")
plt.plot(best_trial["v_losses"], label="Val Loss")
plt.title(f"Best Model — Loss Curves\n{best_trial['cfg']}", fontsize=11)
plt.xlabel("Epoch"); plt.ylabel("MSE Loss")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_loss.png", dpi=150)
plt.show()

# 2. Predictions vs actual
plt.figure(figsize=(12, 5))
plt.plot(best_trial["actuals"], label="Actual",    color="lightgrey", linewidth=1.5)
plt.plot(best_trial["preds"],   label="Predicted", color="red",       linewidth=1.5)
plt.title(f"Best Model — Predictions vs Actual (Test Set)\nR²={best_trial['r2']:.4f}  RMSE={best_trial['rmse']:.4f}", fontsize=11)
plt.xlabel("Time"); plt.ylabel("Value")
plt.legend(); plt.grid(True)
plt.tight_layout()
plt.savefig("best_model_predictions.png", dpi=150)
plt.show()

# 3. R² bar chart across all trials
plt.figure(figsize=(12, 4))
colors = ["#4472C4" if r < best_r2 else "#70AD47" for r in results_df["r2"]]
plt.bar(results_df.index, results_df["r2"], color=colors)
plt.axhline(best_r2, color="red", linestyle="--", linewidth=1, label=f"Best R²={best_r2:.4f}")
plt.title("R² Across All Trials (green = best)")
plt.xlabel("Rank"); plt.ylabel("R²")
plt.legend(); plt.grid(axis="y", alpha=0.4)
plt.tight_layout()
plt.savefig("trial_r2_chart.png", dpi=150)
plt.show()

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the programs in this series so far, the time_series.csv data file, and pyproject.toml. This note uses time_10.py. Open the folder in Visual Studio Code, then type these three lines in the terminal:

uv python pin 3.12.10
uv sync
uv run time_10.py

The first line records which Python version this folder uses, and the second installs every package that pyproject.toml lists. The third line runs the twenty trials, which takes noticeably longer than earlier versions, and then opens the charts; close each window to continue. Two files are left behind: best_model.pt and search_results.xlsx.

Those three lines are needed only once. The next time you want to run the program again, type just uv run time_10.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

What comes next

The window length has stayed at 30 throughout, even though version 2 already showed how much it changes the shape of the examples. The next version adds it to the search space, so the program decides how far back to look as well.