How to try several learning rates and keep the best (Forecast, version 9)?

Howard Pong Yuen Lam · AI Notes · November 13, 2026 · 6 min read

English 繁體中文 简体中文

This is version 9 of the forecasting series. Until now the settings have been guesses: Adam's default learning rate, a hundred epochs, whatever weights happened to exist when the last epoch finished. Version 9 treats the learning rate as something to compare, runs the same training three times, stops each run when it stops improving, and keeps whichever model scored best.

What the learning rate does

In version 7 the optimizer adjusted every weight slightly after each batch. The learning rate controls the size of that adjustment.

There is no formula that tells you which value is right for a given dataset, so the practical answer is to try a few and look.

What early stopping does

Version 8 always completed 100 epochs and used the model from the final epoch, even if the error on the validation days had started increasing earlier. One reason is that the training batches are randomly shuffled, so some weight adjustments can make predictions on the validation days worse. Version 9 checks that error after every epoch and saves the model whenever it reaches a new lowest error. If the error does not improve for 10 epochs in a row, training stops. The program then restores the saved model with the lowest validation error instead of using the model from the final epoch.

This has two benefits. Training can finish sooner when more epochs are no longer improving the model. The program also keeps the model that performed best on the validation days, not simply the model from the final epoch.

How the program works

1. Wrap one run in a function

def train_one_model(lr):
    model     = TimeSeriesModel(window_size).to(device)
    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=lr)

Everything inside repeats version 7's training, with one change: lr, short for learning rate, is now a parameter, so the same function can be called with different learning rates. A fresh model is built on each call, so the three runs start from independent random weights and do not contaminate one another.

2. Keep the best weights

    best_val_loss = float('inf')
    best_state    = {k: v.clone() for k, v in model.state_dict().items()}
    patience_ctr  = 0

best_val_loss starts at infinity, so the first epoch always counts as an improvement. best_state holds a copy of the weights at that moment. The .clone() matters: without it the copy would keep changing as the model changed, and there would be nothing to go back to. patience_ctr counts how many consecutive epochs have passed without an improvement in the validation error. It starts at zero because no epochs have run yet.

3. Stop when improvement stops

        if v_loss < best_val_loss:
            best_val_loss = v_loss
            best_state    = {k: v.clone() for k, v in model.state_dict().items()}
            patience_ctr  = 0
        else:
            patience_ctr += 1
            if patience_ctr >= PATIENCE:
                break

A new best resets the counter, patience_ctr, to zero. A worse epoch adds one to it. When ten worse epochs have piled up in a row, break leaves the loop early. Afterwards model.load_state_dict(best_state) puts the best weights back, so the returned model is the one that scored best on the validation days, not the one that happened to be last.

4. Run the three learning rates

learning_rates = [0.001, 0.01, 0.1]

for lr in learning_rates:
    model, t_losses, v_losses, epochs = train_one_model(lr)
    r2, rmse, mae, preds, actuals = evaluate_on_test(model)

Each pass of the loop trains a complete model and grades it on the test days with version 8's four measures. The results are collected in a list so they can be compared and sorted afterwards.

5. Keep the winner

best = max(results, key=lambda r: r["r2"])
torch.save(best["model"].state_dict(), "best_model.pt")

max selects a result from the results list. The key=lambda r: r["r2"] part tells Python to compare the results by their R² values, so the result with the highest R² is stored in best. torch.save then writes that model's learned weights to a file called best_model.pt. A later program can load this file and reuse the model without training it again.

What you should see

      lr   epochs        R²        RMSE         MAE
--------------------------------------------------
   0.001       68    0.4384      5.3538      4.2973
    0.01       37    0.4394      5.3491      4.2547
     0.1       12    0.0881      6.8221      5.3624

Best learning rate: 0.01  (R² = 0.4394)
Best model saved → best_model.pt

Your numbers will differ, but the pattern is what matters. The two smaller learning rates reach a similar quality, and 0.01 reaches it in 37 epochs rather than 68. Both give a similar result, but 0.01 needs fewer epochs. The large rate of 0.1 stops after 12 epochs with an R² of 0.09, because each adjustment is too large for the errors to become smaller steadily.

Two things improved since version 8. R² moved from 0.24 to about 0.44, and much of that improvement came from early stopping rather than from the learning rate: keeping the best weights instead of the final ones raises the score noticeably. Training also finishes in 37 epochs instead of 100.

Loss curves for the best run, falling and then flattening, with training and validation lines close together
Loss curves of the winning run, stopped at epoch 37
Actual test-day sales in grey and predicted values in red, the red line tracking the grey more closely than before
Actual (grey) and predicted (red) values for the test days, best run
Bar chart of R squared for the three learning rates, the middle bar green as the best
R² for the three learning rates, green for the winner

The whole program

The pieces above, put back together. This is the complete file:

import os
os.environ["KMP_DUPLICATE_LIB_OK"] = "TRUE"

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import TensorDataset, DataLoader, Subset
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

# ── Data preparation (same as time_08) ────────────────────────────────────────
def create_sliding_windows_with_target(data, window_size, shift=1):
    windows = data.unfold(0, window_size, shift)
    features = windows[:, :-1]
    targets = windows[:, -1:]
    return features, targets

df = pd.read_csv('time_series.csv')
series = df['series'].values
series_tensor = torch.tensor(series, dtype=torch.float32)
window_size = 30
features, targets = create_sliding_windows_with_target(series_tensor, window_size=window_size, shift=1)
full_dataset = TensorDataset(features, targets)

total_windows = len(full_dataset)
train_size = int(total_windows * 0.7)
val_size = int(total_windows * 0.15)
test_size = total_windows - train_size - val_size

train_indices = list(range(train_size))
val_indices = list(range(train_size, train_size + val_size))
test_indices = list(range(train_size + val_size, total_windows))

train_loader = DataLoader(Subset(full_dataset, train_indices),
                         batch_size=32, shuffle=True)
val_loader   = DataLoader(Subset(full_dataset, val_indices),
                         batch_size=32, shuffle=False)
test_loader  = DataLoader(Subset(full_dataset, test_indices),
                         batch_size=32, shuffle=False)

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# ── Visualize train/val/test split (same as time_08) ─────────────────────────
val_start = train_size
test_start = train_size + val_size
plt.figure(figsize=(12, 4))
plt.plot(range(val_start), series[:val_start], 'b-', label='Training Data')
plt.plot(range(val_start, test_start), series[val_start:test_start], 'r-', label='Validation Data')
plt.plot(range(test_start, len(series)), series[test_start:], 'g-', label='Test Data')
plt.axvline(x=val_start, color='k', linestyle='--', label='Train/Val Split')
plt.axvline(x=test_start, color='purple', linestyle='--', label='Val/Test Split')
plt.title('Time Series Train/Validation/Test Split')
plt.xlabel('Time'); plt.ylabel('Value')
plt.legend(); plt.grid(True); plt.show()

# ── Model (same as time_07/08) ────────────────────────────────────────────────
class TimeSeriesModel(nn.Module):
    def __init__(self, window_size):
        super(TimeSeriesModel, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(window_size - 1, 10),
            nn.ReLU(),
            nn.Linear(10, 10),
            nn.ReLU(),
            nn.Linear(10, 1)
        )

    def forward(self, x):
        return self.network(x)

# ── NEW: training with early stopping ────────────────────────────────────────
# Early stopping: if val loss does not improve for PATIENCE consecutive epochs,
# stop training early and restore the best weights seen so far.
PATIENCE   = 10
MAX_EPOCHS = 100

def train_one_model(lr):
    model     = TimeSeriesModel(window_size).to(device)
    criterion = nn.MSELoss()
    optimizer = optim.Adam(model.parameters(), lr=lr)

    best_val_loss = float('inf')
    best_state    = {k: v.clone() for k, v in model.state_dict().items()}
    patience_ctr  = 0
    train_losses, val_losses = [], []

    for epoch in range(MAX_EPOCHS):
        # ── train ──
        model.train()
        t_loss = 0
        for batch_features, batch_targets in train_loader:
            batch_features = batch_features.to(device)
            batch_targets  = batch_targets.to(device)
            optimizer.zero_grad()
            loss = criterion(model(batch_features), batch_targets)
            loss.backward()
            optimizer.step()
            t_loss += loss.item()
        t_loss /= len(train_loader)

        # ── validate ──
        model.eval()
        v_loss = 0
        with torch.no_grad():
            for batch_features, batch_targets in val_loader:
                batch_features = batch_features.to(device)
                batch_targets  = batch_targets.to(device)
                v_loss += criterion(model(batch_features), batch_targets).item()
        v_loss /= len(val_loader)

        train_losses.append(t_loss)
        val_losses.append(v_loss)

        # ── early stopping check ──
        if v_loss < best_val_loss:
            best_val_loss = v_loss
            best_state    = {k: v.clone() for k, v in model.state_dict().items()}
            patience_ctr  = 0
        else:
            patience_ctr += 1
            if patience_ctr >= PATIENCE:
                break

    # restore the best weights
    model.load_state_dict(best_state)
    actual_epochs = len(train_losses)
    return model, train_losses, val_losses, actual_epochs

# ── NEW: evaluate on test set (returns R², RMSE, MAE) ────────────────────────
def evaluate_on_test(model):
    model.eval()
    preds, actuals = [], []
    with torch.no_grad():
        for batch_features, batch_targets in test_loader:
            preds.extend(model(batch_features.to(device)).cpu().numpy())
            actuals.extend(batch_targets.numpy())

    p = torch.tensor(np.array(preds))
    a = torch.tensor(np.array(actuals))

    mae  = torch.mean(torch.abs(p - a)).item()
    rmse = torch.sqrt(torch.mean((p - a) ** 2)).item()
    ss_res = torch.sum((a - p) ** 2)
    ss_tot = torch.sum((a - torch.mean(a)) ** 2)
    r2 = (1 - ss_res / ss_tot).item()
    return r2, rmse, mae, preds, actuals

# ── NEW: manually try 3 learning rates ────────────────────────────────────────
# Instead of picking one optimizer and one learning rate, we run the same
# training procedure three times with different learning rates and compare.
learning_rates = [0.001, 0.01, 0.1]
results = []

print(f"{'lr':>8}  {'epochs':>7}  {'R²':>8}  {'RMSE':>10}  {'MAE':>10}")
print("-" * 50)

for lr in learning_rates:
    model, t_losses, v_losses, epochs = train_one_model(lr)
    r2, rmse, mae, preds, actuals = evaluate_on_test(model)

    results.append({
        "lr": lr, "epochs": epochs,
        "r2": round(r2, 4), "rmse": round(rmse, 4), "mae": round(mae, 4),
        "model": model, "t_losses": t_losses, "v_losses": v_losses,
        "preds": preds, "actuals": actuals,
    })
    print(f"{lr:>8}  {epochs:>7}  {r2:>8.4f}  {rmse:>10.4f}  {mae:>10.4f}")

# ── Pick the best trial and save ──────────────────────────────────────────────
best = max(results, key=lambda r: r["r2"])
print(f"\nBest learning rate: {best['lr']}  (R² = {best['r2']:.4f})")

torch.save(best["model"].state_dict(), "best_model.pt")
print("Best model saved → best_model.pt")

# ── Plots ──────────────────────────────────────────────────────────────────────
# 1. Loss curves for the best trial
plt.figure(figsize=(10, 5))
plt.plot(best["t_losses"], label="Train Loss")
plt.plot(best["v_losses"], label="Val Loss")
plt.title(f"Best Model — Loss Curves\nlr={best['lr']}  (stopped at epoch {best['epochs']})")
plt.xlabel("Epoch"); plt.ylabel("MSE Loss")
plt.legend(); plt.grid(True); plt.tight_layout(); plt.show()

# 2. Predictions vs actual on test set (best trial)
plt.figure(figsize=(12, 5))
plt.plot(best["actuals"], label="Actual",    color="lightgrey", linewidth=1.5)
plt.plot(best["preds"],   label="Predicted", color="red",       linewidth=1.5)
plt.title(f"Best Model — Predictions vs Actual (Test Set)\nR²={best['r2']:.4f}  RMSE={best['rmse']:.4f}")
plt.xlabel("Time"); plt.ylabel("Value")
plt.legend(); plt.grid(True); plt.tight_layout(); plt.show()

# 3. R² bar chart comparing all 3 trials
plt.figure(figsize=(8, 4))
lrs = [str(r["lr"]) for r in results]
r2s = [r["r2"] for r in results]
colors = ["#4472C4" if r["lr"] != best["lr"] else "#70AD47" for r in results]
plt.bar(lrs, r2s, color=colors)
plt.axhline(best["r2"], color="red", linestyle="--", linewidth=1, label=f"Best R²={best['r2']:.4f}")
plt.title("R² Across 3 Learning Rates (green = best)")
plt.xlabel("Learning Rate"); plt.ylabel("R²")
plt.legend(); plt.grid(axis="y", alpha=0.4); plt.tight_layout(); plt.show()

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the programs in this series so far, the time_series.csv data file, and pyproject.toml. This note uses time_09.py. Open the folder in Visual Studio Code, then type these three lines in the terminal:

uv python pin 3.12.10
uv sync
uv run time_09.py

The first line records which Python version this folder uses, and the second installs every package that pyproject.toml lists. The third line trains three models in turn and prints the comparison table, then opens the charts; close each window to continue.

Those three lines are needed only once. The next time you want to run the program again, type just uv run time_09.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

What comes next

Trying three values by hand is already repetitive. The learning rate is only one setting among many — the number of hidden layers, their width, the optimizer and the window length are all choices too. The next version lets the program try many combinations by itself and keep the best.