bo4mob-estimation — BO4Mob held-out-count OD estimation (D2 observational)

What. The T2 estimation track’s first observational certificate (adr-041). BO4Mob (Ryu, Kwon, Choi, Deshwal, Kang & Osorio 2025, arXiv:2510.18824) poses a San Jose freeway network as a black-box OD-estimation problem: choose a continuous OD vector over a fixed set of active (fromTaz, toTaz) pairs to fit real Caltrans PeMS link counts under a mesoscopic SUMO run. There is no declared BPR network, no true OD, and no equilibrium — so this is a NEW T2 sibling family (the adr-023 pattern), scored by re-running the pinned eclipse-sumo engine, not a bfw assignment.

Dual-benchmark honesty (the central constraint). BO4Mob is a UMN Choi Lab benchmark; TABenchmark hosts its instances as scenarios/data only (adr-034), not as validation of TABench methods, and does not reproduce BO4Mob’s published numbers. This estimation row additionally does not reproduce BO4Mob’s own SPSA/BO leaderboard rankings, and its D2 held-out NRMSE is not comparable to the static/dynamic T2 heldout_count_rmse scale (it is BO4Mob’s own n-scaled NRMSE, a same-lab scenario).

How this notebook is graded

A notebook never claims a number it does not compute in that cell. Every scored quantity below — od_feasible, obs_nrmse, heldout_nrmse — comes from the SAME Bo4MobODCertifier the harness uses: it re-runs the pinned od2trips + mesoscopic sumo pipeline ONCE on the emitted OD and compares the resulting link counts to real PeMS panels. Equilibrium is never claimed.

The ranking column is heldout_nrmse — the MEAN of BO4Mob’s count NRMSE over a temporally-disjoint, same-hour set of held-out DATES (the anchor hour 06-07, dates 221009–221021). The estimator emits ONE fixed OD from the TRAIN anchor (221008); the certifier scores it against each held-out date’s real counts. The held-out CSV bytes and dates never enter the task — only their heldout_digest does (P7). The engine version is pinned (assert_engine_pin RAISES on a mismatch), so a drifted box fails loudly rather than silently scoring.

This notebook needs the optional sumo extra (pip install 'tabench[sumo]').

# Headless inline backend (Agg): figures render into the notebook so CI can execute
# tutorials without a display. Never call matplotlib.use() after pyplot is imported.
import matplotlib
matplotlib.use("Agg")
import json
import matplotlib.pyplot as plt
import numpy as np

from tabench.core.budget import Budget
from tabench.data.bo4mob import (
    BO4MOB_HELDOUT_DATES, BO4MOB_HELDOUT_HOUR, BO4MOB_REGISTRY,
    bo4mob_pairs, bo4mob_prior_vector, fetch_bo4mob, fetch_bo4mob_heldout,
)
from tabench.estimation.bo4mob_base import Bo4MobPriorBaseline
from tabench.experiments.runner import run_bo4mob_estimation_experiment
from tabench.metrics.estimation_bo4mob import Bo4MobODCertifier

INSTANCE = "1ramp"  # the smallest instance: ~0.4 s meso, the CI liveness case
spec = BO4MOB_REGISTRY[INSTANCE]
print(f"{INSTANCE}: {spec.n_od} OD pairs, {spec.n_sensors_anchor} anchor sensors")

1ramp: 3 OD pairs, 3 anchor sensors

The held-out split (framing b)

The estimator emits one fixed OD vector; “holding out a date” only withholds which ground-truth CSVs are used for scoring, never which OD to submit. This closes a free-riding hole that per-date OD emission would open (copy the nearest train date’s OD) and costs one meso run per certify regardless of how many held-out dates are scored. PeMS counts vary enormously across the day, so the hour window is held FIXED — held-out probes same-hour, different-DATE generalisation only.

# Run the prior baseline (emit BO4Mob's example OD unchanged) and certify it.
result = run_bo4mob_estimation_experiment(
    INSTANCE, [Bo4MobPriorBaseline()], Budget(sp_calls=1), seed=0,
)
row = result.rows[0]
print(f"estimator      : {row['estimator']}")
print(f"od_feasible    : {row['od_feasible']}")
print(f"obs_nrmse      : {row['obs_nrmse']:.6f}   (in-sample, TRAIN anchor 221008)")
print(f"heldout_nrmse  : {row['heldout_nrmse']:.6f}   <- ranking column (mean over "
      f"{int(row['n_heldout_dates'])} held-out dates)")
print(f"  min / max    : {row['heldout_nrmse_min']:.6f} / {row['heldout_nrmse_max']:.6f}")
# The prior OD is BO4Mob's low-demand anchor day, so its held-out MEAN (over higher-
# demand days) differs from its in-sample fit -- an honest generalisation signal.

estimator      : bo4mob-prior
od_feasible    : 1.0
obs_nrmse      : 2.432471   (in-sample, TRAIN anchor 221008)
heldout_nrmse  : 1.697988   <- ranking column (mean over 13 held-out dates)
  min / max    : 0.563911 / 3.297711

The anti-laundering control: the OD-window fill

The one real laundering vector, found and fixed once in stage 1 (adr-034 Decision 3), is the OD-window fill: the emitted vector must fill od.xml with the interval end rewritten to the instance’s od_end_time. On 1ramp (od_end_time=3300) keeping the template’s 3600 releases ~5% of demand past the OD window and silently biases the count NRMSE. fill_od_from_vector inherits this rewrite, so the prior baseline’s certified obs_nrmse reproduces the stage-1 faithful pipeline exactly (2.432471…); the unfixed value would be 2.314704….

# The per-held-out-date NRMSE (one pipeline run, K cheap comparisons). This is the
# spread the ranking `heldout_nrmse` averages over -- never a black box.
paths = fetch_bo4mob(spec)
heldout = fetch_bo4mob_heldout(INSTANCE)
cfg = json.loads(paths["config"].read_text())
pairs = bo4mob_pairs(paths["od"])
prior = bo4mob_prior_vector(paths["od"], paths["single_od"])
cert = Bo4MobODCertifier(
    instance_key=INSTANCE, pairs=pairs, train_sensor=paths["sensor"], heldout_sensors=heldout,
    paths=paths, od_end_time=int(cfg["od_end_time"]), sim_end_time=float(cfg["sim_end_time"]),
    sensor_start_time=float(cfg["sensor_start_time"]), sensor_end_time=float(cfg["sensor_end_time"]),
    engine_version="1.27.1", certificate={"seed": 0},
)
per_date = cert.per_date_heldout_nrmse(prior)
dates = list(per_date)
vals = [per_date[d] for d in dates]
fig, ax = plt.subplots(figsize=(8, 3))
ax.bar(range(len(dates)), vals, color="#4477aa")
ax.axhline(np.mean(vals), color="#cc3311", ls="--", label=f"heldout_nrmse = {np.mean(vals):.3f}")
ax.set_xticks(range(len(dates)))
ax.set_xticklabels([d[-4:] for d in dates], rotation=45, ha="right")
ax.set_ylabel("count NRMSE")
ax.set_title(f"{INSTANCE}: prior-OD NRMSE per held-out date (hour {BO4MOB_HELDOUT_HOUR})")
ax.legend()
plt.tight_layout()
print("held-out dates:", ", ".join(BO4MOB_HELDOUT_DATES))
print(f"mean held-out NRMSE = {np.mean(vals):.6f}")

held-out dates: 221009, 221010, 221011, 221012, 221013, 221014, 221015, 221016, 221017, 221018, 221019, 221020, 221021
mean held-out NRMSE = 1.697988

Reading the row

heldout_nrmse ranks the family: a prior OD already fitting the held-out days is unbeatable; one far off certifies a terrible (honest) held-out count-fit. A future black-box estimator (an SPSA over this pipeline) would run the engine as its own inner oracle and try to lower heldout_nrmse — improving out-of-sample count fit, never memorising a single day.

Honesty, restated. BO4Mob is a UMN Choi Lab benchmark; TABenchmark hosts its instances as scenarios/data only (adr-034), not as validation of TABench methods, and does not reproduce BO4Mob’s published numbers — including BO4Mob’s own SPSA/BO leaderboard rankings. Equilibrium is never claimed, and this D2 held-out NRMSE is not comparable to the static/dynamic T2 heldout_count_rmse scale.