How to measure the model on data it has never seen (Forecast, version 8)?

Howard Pong Yuen Lam · AI Notes · November 13, 2026 · 6 min read

English 繁體中文 简体中文

This is version 8 of the forecasting series. Version 7 judged the model on validation days, but those days were also used, epoch after epoch, to decide when training looked good. That makes them no longer fully fair. Version 8 therefore cuts the data into three parts instead of two — training, validation and test — keeps the test part sealed until the very end, and reports four numbers on it.

Why three parts, not two

The validation days are used while training is still going on. After every epoch, the program measures their loss, so you can see whether the model is improving or beginning to memorise. These results also guide your decisions: whether to train for more epochs and, in later versions, which settings to choose. Because the validation results influence those decisions, they stop being a completely independent check. It is like using the same practice exam repeatedly: a good score may reflect familiarity with those particular questions rather than how well you would do on new ones.

The test part deals with that problem by being kept out of the room. It is sealed before training starts, opened once after training has finished, and used for nothing except the final report. No decision along the way was made with it in view, so whatever it says is an honest answer to the question how good is this model on days it has never seen?

Where the three cuts fall

train_size = int(total_windows * 0.7)
val_size = int(total_windows * 0.15)
test_size = total_windows - train_size - val_size

1,431 examples give train_size of 1,001 (0.7 of 1,431 is 1,001.7, and int() drops the decimal) and val_size of 214. The test part takes whatever is left rather than its own percentage, so the three parts always add back to 1,431: 1,001 + 214 + 216.

The three lists of positions are again three unbroken stretches of time, earliest first, so the model trains on the oldest days, is watched on the middle ones, and is graded on the newest.

Sales line split into blue training days, red validation days and green test days, with two dashed vertical divider lines
Training, validation and test parts, drawn by time_08.py

The four measures

All four compare the model's guesses with the real days in the test part, and each answers a slightly different question.

R² of 0.24 here is the number worth staring at. The model is clearly better than guessing the average, but it accounts for only about a quarter of the movement in the test days. This is an honest, modest result, and versions 9 and 10 try to improve it.

How the program works

1. Build three loaders

train_indices = list(range(train_size))
val_indices = list(range(train_size, train_size + val_size))
test_indices = list(range(train_size + val_size, total_windows))

train_dataset = Subset(full_dataset, train_indices)
val_dataset = Subset(full_dataset, val_indices)
test_dataset = Subset(full_dataset, test_indices)

batch_size = 32
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

Version 6's two-way split becomes three, with the same shape: three lists of positions, three Subsets, three loaders. Training shuffles; the other two do not, so their measurements stay comparable from run to run.

2. Train exactly as before

The network, the loss, the optimizer and the 100-epoch loop are unchanged from version 7. Only the split above and the measurement below are new. That is worth noticing: once the training loop exists, adding a fair test does not require touching it.

3. Measure once, after training

model.eval()
test_predictions = []
test_targets = []
with torch.no_grad():
    for batch_features, batch_targets in test_loader:
        batch_features = batch_features.to(device)
        outputs = model(batch_features)
        test_predictions.extend(outputs.cpu().numpy())
        test_targets.extend(batch_targets.numpy())

The two lists collect the guesses and the real answers over the whole test part. .extend adds every batch's contents to the list, so that at the end the two lists hold 216 numbers each, in order.

4. Turn the lists into the four numbers

mae = torch.mean(torch.abs(test_predictions_tensor - test_targets_tensor))
mse = torch.mean((test_predictions_tensor - test_targets_tensor) ** 2)
rmse = torch.sqrt(mse)
ss_res = torch.sum((test_targets_tensor - test_predictions_tensor) ** 2)
ss_tot = torch.sum((test_targets_tensor - torch.mean(test_targets_tensor)) ** 2)
r2 = 1 - ss_res / ss_tot

Each line is the definition written out. MAE averages the absolute differences; MSE averages the squared differences; RMSE takes the square root of MSE.

R² compares the model with a simple alternative: if we did not have a useful model, we could use the average sales value of the test days as the prediction for every day. ss_res is the total squared error from the model, and ss_tot is the total squared error from this average-based prediction. Dividing one by the other shows how much error remains compared with the simple average, and subtracting from 1 shows how much error the model removes.

If the model performs no better than the average-based prediction, the two totals are equal, the ratio is 1, and R² is 0. If the model is perfect, ss_res is 0 and R² is 1.

What you should see

Epoch [10/100], Train Loss: 45.0302, Val Loss: 53.8099
Epoch [20/100], Train Loss: 45.5068, Val Loss: 51.7960
Epoch [30/100], Train Loss: 43.4795, Val Loss: 54.2412
Epoch [40/100], Train Loss: 42.1594, Val Loss: 50.4357
Epoch [50/100], Train Loss: 40.0044, Val Loss: 47.8687
Epoch [60/100], Train Loss: 38.7391, Val Loss: 45.2560
Epoch [70/100], Train Loss: 37.1991, Val Loss: 44.5767
Epoch [80/100], Train Loss: 36.1731, Val Loss: 41.6744
Epoch [90/100], Train Loss: 35.9192, Val Loss: 42.3977
Epoch [100/100], Train Loss: 33.9332, Val Loss: 45.6109

Test Set Metrics:
  MAE:  5.0289
  MSE:  38.7685
  RMSE: 6.2264
  R²:   0.2404

As in version 7, your numbers will differ from these because the weights start at random. The pattern to look for is that the training loss keeps falling while the validation loss flattens and wobbles, and that RMSE sits somewhat above MAE.

Loss curves over 100 epochs, training loss falling and validation loss flattening around the mid forties
Training and validation loss with the three-way split
Actual test-day sales in grey and predicted values in red, the red line following the overall shape but missing the peaks
Actual (grey) and predicted (red) values for the test days

In the second chart the red line follows the general rise but flattens out the peaks. A model that cannot reach the peaks is exactly what an R² of 0.24 looks like on a picture.

The whole program

The pieces above, put back together. This is the complete file:

import torch
from torch.utils.data import TensorDataset, DataLoader, Subset
import pandas as pd
import matplotlib.pyplot as plt

def create_sliding_windows_with_target(data, window_size, shift=1):
    windows = data.unfold(0, window_size, shift)
    features = windows[:, :-1]
    targets = windows[:, -1:]
    return features, targets

df = pd.read_csv('time_series.csv')
series = df['series'].values
series_tensor = torch.tensor(series, dtype=torch.float32)
window_size = 30
features, targets = create_sliding_windows_with_target(series_tensor, window_size=window_size, shift=1)
full_dataset = TensorDataset(features, targets)

total_windows = len(full_dataset)
train_size = int(total_windows * 0.7)
val_size = int(total_windows * 0.15)
test_size = total_windows - train_size - val_size

train_indices = list(range(train_size))
val_indices = list(range(train_size, train_size + val_size))
test_indices = list(range(train_size + val_size, total_windows))

train_dataset = Subset(full_dataset, train_indices)
val_dataset = Subset(full_dataset, val_indices)
test_dataset = Subset(full_dataset, test_indices)

batch_size = 32
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
val_loader = DataLoader(val_dataset, batch_size=batch_size, shuffle=False)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)

# Visualize train/val/test split
val_start = train_size
test_start = train_size + val_size
plt.figure(figsize=(12, 4))
plt.plot(range(val_start), series[:val_start], 'b-', label='Training Data')
plt.plot(range(val_start, test_start), series[val_start:test_start], 'r-', label='Validation Data')
plt.plot(range(test_start, len(series)), series[test_start:], 'g-', label='Test Data')
plt.axvline(x=val_start, color='k', linestyle='--', label='Train/Val Split')
plt.axvline(x=test_start, color='purple', linestyle='--', label='Val/Test Split')
plt.title('Time Series Train/Validation/Test Split')
plt.xlabel('Time')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()

import torch.nn as nn
import torch.optim as optim

class TimeSeriesModel(nn.Module):
    def __init__(self, window_size):
        super(TimeSeriesModel, self).__init__()
        self.network = nn.Sequential(
            nn.Linear(window_size - 1, 10),
            nn.ReLU(),
            nn.Linear(10, 10),
            nn.ReLU(),
            nn.Linear(10, 1)
        )

    def forward(self, x):
        return self.network(x)

model = TimeSeriesModel(window_size)
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters())

num_epochs = 100
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model.to(device)

train_losses = []
val_losses = []

for epoch in range(num_epochs):
    model.train()
    train_loss = 0
    for batch_features, batch_targets in train_loader:
        batch_features = batch_features.to(device)
        batch_targets = batch_targets.to(device)
        outputs = model(batch_features)
        loss = criterion(outputs, batch_targets)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        train_loss += loss.item()

    model.eval()
    val_loss = 0
    with torch.no_grad():
        for batch_features, batch_targets in val_loader:
            batch_features = batch_features.to(device)
            batch_targets = batch_targets.to(device)
            outputs = model(batch_features)
            val_loss += criterion(outputs, batch_targets).item()

    train_loss /= len(train_loader)
    val_loss /= len(val_loader)
    train_losses.append(train_loss)
    val_losses.append(val_loss)

    if (epoch + 1) % 10 == 0:
        print(f'Epoch [{epoch+1}/{num_epochs}], '
              f'Train Loss: {train_loss:.4f}, '
              f'Val Loss: {val_loss:.4f}')

# Plot training progress
plt.figure(figsize=(10, 6))
plt.plot(train_losses, label='Training Loss')
plt.plot(val_losses, label='Validation Loss')
plt.title('Model Loss Over Time')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.legend()
plt.grid(True)
plt.show()

# Evaluate on test set
model.eval()
test_predictions = []
test_targets = []
with torch.no_grad():
    for batch_features, batch_targets in test_loader:
        batch_features = batch_features.to(device)
        outputs = model(batch_features)
        test_predictions.extend(outputs.cpu().numpy())
        test_targets.extend(batch_targets.numpy())

# Plot predictions vs actual for test set
plt.figure(figsize=(10, 6))
plt.plot(test_targets, label='Actual', color='lightgrey')
plt.plot(test_predictions, label='Predicted', color='red')
plt.title('Predictions vs Actual Values (Test Set)')
plt.xlabel('Time')
plt.ylabel('Value')
plt.legend()
plt.grid(True)
plt.show()

# Metrics on test set
test_predictions_tensor = torch.tensor(test_predictions)
test_targets_tensor = torch.tensor(test_targets)

mae = torch.mean(torch.abs(test_predictions_tensor - test_targets_tensor))
mse = torch.mean((test_predictions_tensor - test_targets_tensor) ** 2)
rmse = torch.sqrt(mse)
ss_res = torch.sum((test_targets_tensor - test_predictions_tensor) ** 2)
ss_tot = torch.sum((test_targets_tensor - torch.mean(test_targets_tensor)) ** 2)
r2 = 1 - ss_res / ss_tot

print(f"\nTest Set Metrics:")
print(f"  MAE:  {mae:.4f}")
print(f"  MSE:  {mse:.4f}")
print(f"  RMSE: {rmse:.4f}")
print(f"  R²:   {r2:.4f}")

Running it on your own computer

Download the zip file and unzip it to a folder. The zip holds the programs in this series so far, the time_series.csv data file, and pyproject.toml. This note uses time_08.py. Open the folder in Visual Studio Code, then type these three lines in the terminal:

uv python pin 3.12.10
uv sync
uv run time_08.py

The first line records which Python version this folder uses, and the second installs every package that pyproject.toml lists. The third line prints the epoch lines and the four test measures, and opens three chart windows in turn; close each one to continue.

Those three lines are needed only once. The next time you want to run the program again, type just uv run time_08.py.

If you have not set up uv yet, start with How to install Visual Studio Code? and then the uv note for Windows or Mac.

What comes next

An R² of 0.24 leaves plenty of room. Version 9 stops guessing at the settings and tries several learning rates by hand, with early stopping, and keeps whichever one scores best.