ADR-034: bo4mob-scenarios — the BO4Mob San Jose freeway OD-estimation instances (stage 1)

Status: accepted (implemented) Date: 2026-07-16 Deciders: non-solver roadmap — the bo4mob-scenario queue item (ROADMAP.md:110) File: docs/design/adr-034-bo4mob-scenarios.md

Context

BO4Mob (Ryu, Kwon, Choi, Deshwal, Kang & Osorio, 2025, arXiv:2510.18824, NeurIPS 2025 Datasets & Benchmarks; canon ryu2025bo4mob, tier 1) poses five San Jose freeway networks as high-dimensional black-box OD-estimation problems: minimise the NRMSE between mesoscopic-SUMO link counts and real Caltrans PeMS sensor data over a continuous OD vector (3 → 10,100 OD pairs). It is an inverse problem with a stochastic, non-differentiable, expensive objective and — the fact that shapes everything below — no ground-truth OD: truth is the real sensor panel (14 dates × 3 hour windows = 210 CSVs in-repo).

The dual-benchmark sensitivity. BO4Mob is the lab’s own benchmark (github.com/UMN-Choi-Lab/BO4Mob, MIT). Integrating a benchmark the lab authored into a benchmark the lab authors is a standing honesty hazard, and the contract that governs it (below) is the central content of this ADR, not a footnote. The canon already carries the cross-benchmark pointer (ryu2025bo4mob, MODELS.md roadmap card); this ADR adds the instances.

This is stage 1 only: data availability + pipeline liveness. It ships the P9 data/scenario family (the four small instances, checksummed download-on-demand) and a guarded smoke test proving the mesoscopic-SUMO evaluation pipeline runs end-to-end under the shipped eclipse-sumo wheel. It ships no task family, no certificate, no estimator — those are a named stage-2 follow-up with its own future ADR: a bo4mob_estimation T2 task with a pinned-engine held-out-date observational certificate (the honest certifier shape given there is no true OD and no declared BPR network — see “Stage 2” below).

The dual-benchmark honesty contract (the load-bearing section)

The lab owns both benchmarks. The only honest claim shape:

• ALLOWED: “TABenchmark hosts BO4Mob’s instances as scenarios/data (same-lab provenance disclosed in notes and bo4mob_citation); any future results on them are scored by TABench’s own certificate discipline.”

• FORBIDDEN, and guarded against here:

  1. “TABench methods validated on the external BO4Mob benchmark” — BO4Mob is not external, and its instances are scenarios, never validation of TABench methods. Stage 1 ships no method scored on BO4Mob at all; the smoke test is a liveness check of the engine pipeline, explicitly not an oracle (VALIDATION.md tier 5, below everything).

  2. Any claim of reproducing BO4Mob’s published numbers. The shipped eclipse-sumo wheel is 1.27.1; BO4Mob’s paper ran SUMO 1.12. A measured schema drift (Decision 4) makes the simulated values non-reproducible here — the instances transfer, the numbers do not — so paper numbers are never claimed, in prose or in a pin.

  3. Using TABench runs as “independent replication” of BO4Mob’s published strategy rankings (SPSA/SAASBO/TuRBO), or vice versa. TABench does not re-host BO4Mob’s BO leaderboard; those comparisons remain BO4Mob’s.

  4. Tuning TABench estimators on BO4Mob anchors and reporting it as cross-benchmark generalisation.

The disclosure ships in-artifact, not only in docs: every registry entry’s notes and bo4mob_citation() carry the affiliation, the MIT license, the PeMS provenance, the “scenarios/data only” scope, and the “does not reproduce BO4Mob’s published numbers” clause. Tests pin all four strings.

Decision 1 — P9 checksummed fetcher family; a separate registry

Data are fetched, never vendored (P9). BO4MOB_REGISTRY (data/bo4mob.py) holds the five instances; fetch_bo4mob fetches each file from a commit-pinned raw URL (UMN-Choi-Lab/BO4Mob@ef571e68), verifies it against a pinned per-file SHA-256 on every load, caches it under ~/.cache/tabench/bo4mob/<key>/, and evicts + ChecksumErrors on any mismatch — the TNTP fetcher mechanics, extended with the xu2024 .part hygiene (try/finally-unlinked on a mid-download failure). The body is streamed with a cap at each file’s pinned byte size (+4 KB slack): an oversized upstream body raises Bo4MobUpstreamError mid-stream, before it is materialised (a hostile 256 MB body no longer fills RSS/disk ahead of the checksum — the fresh adr-033 standard, restored after review). The .part carries a per-process suffix (.part.<pid>) so concurrent cold-start fetches never collide; fetcher.py / xu2024.py share the latent single-.part race (a documented follow-up, deliberately not touched this sprint).

The four small instances (1ramp, 2corridor, 3junction, 4smallRegion) each ship a single-evaluation bundle: net.xml, taz.xml, od.xml, additional.xml, routes_single.csv, the od_for_single_run OD vector, the config JSON, and the one canonical sensor CSV (221008 06-07). Bundle sizes measured at the pin: 1ramp 27,327 B, 2corridor 146,814 B, 3junction 361,055 B, 4smallRegion 643,697 B — 1,178,893 B total, all download-on-demand. The full 14-date × 3-hour PeMS panel and the routes_multiple stochastic route table are available upstream but not pinned in stage 1; stage 2’s held-out-date certificate pins them. (routes_single is used precisely because it is deterministic — one route per OD pair — so the meso run is seed-stable; routes_multiple draws routes with np.random.choice and is out of scope.)

The registry is deliberately separate from the CI-prefetched TNTP REGISTRY (the xu2024 lesson): a core CI run never fetches any BO4Mob file on its own. Only the sumo job prefetches the 1ramp bundle (the smoke test’s input, < 30 KB). The registry keys are the bare instance names (1ramp, 2corridor, 3junction, 4smallRegion, 5fullRegion), and — unlike xu2024 — they are not load_scenario scenarios: a BO4Mob instance is a mesoscopic-SUMO net with no BPR network and no true OD, so no Scenario (Network/Demand/ReferenceSolution) can be built. That is not a gap; it is the honest statement that these are data + an engine, not equilibrium instances. A test pins that load_scenario raises on both a bare key (1ramp) and a bo4mob--prefixed key.

5fullRegion is registered metadata-only and refuses to fetch. At 10,100 OD pairs, 74 MB, and ~11 h/eval it is never a CI input, never a default, never an anchor. The chosen mechanism is an explicit hpc_only=True flag whose fetch raises Bo4MobHpcOnlyError with a clear message — a named refusal, not a silent exclusion (an HPC run fetches it deliberately by other means). This is cleaner than excluding it: the instance’s existence and cost stay visible in the registry and its metadata (dimension, sensor count) is recorded, while the refusal makes the “never in CI / never a laptop” boundary executable.

Decision 2 — the evaluation convention lives in the module; the engine run lives in the test

data/bo4mob.py ships the pure, pandas-free (numpy/stdlib) transforms of BO4Mob’s own pipeline: fill_single_od (fill the count=0 od.xml template from the single-run OD CSV and rewrite the interval end to the config’s od_end_time — load-bearing, Decision 3), fix_routes_single (BO4Mob’s update_trip_routes for routes_per_od='single' — rewrite each trip’s from/to to the route table’s start/last edge, sort by departure, departLane='best'), edgedata_counts (the arrived + left count convention, link_flow_analysis.py), bo4mob_nrmse (the count NRMSE √(n·Σ(gt−sim)²)/Σgt, sim filled 0 for absent sensors), and local_edgedata_additional (redirect the meso edgeData output to a per-run workdir file while keeping the DEFAULT_VEHTYPE IDM/speedDev=0 attribute-exact — semantically identical; ElementTree normalises serialisation whitespace/quoting, so this is not a byte-level claim). The benchmark core stays pandas-free (BO4Mob’s own pipeline uses pandas; this is a faithful numpy/stdlib reimplementation, validated against the upstream result to the digit — Decision 3).

The subprocess orchestration (the od2trips and mesoscopic sumo invocations) lives in the guarded smoke test, exactly as the sumo-marouter adapter keeps its _run_marouter helper in test_sumo_marouter.py (adr-027). This keeps the data layer free of subprocess/sumo — the module imports numpy/stdlib only and is importable on a core install — while the engine run is exercised behind the sumo extra. Binary discovery is sumo.SUMO_HOME-only (never PATH / the ambient stale SUMO_HOME=/opt/sumo-1.12 this box ships), each subprocess gets stdin=DEVNULL and a workdir-local output, and one wall deadline threads both subprocesses (the adr-027/028 wrapper discipline); a nonzero exit or timeout raises loudly (a liveness smoke must fail, not censor). Stage 2’s estimator will reuse the module’s pure transforms behind a proper guarded adapter.

Decision 3 — pipeline liveness: the measured 1ramp run (provenance, pinned loose)

Reproduced end-to-end on this box (eclipse-sumo==1.27.1 wheel), at the pinned commit, through the pandas-free transforms above:

• NRMSE = 2.432471221214843 at the shipped od_for_single_run OD, date 221008, hour 06-07, over the 3 GT sensors — byte-identical across seeds 0/1/2 (max−min = 0.0). 1ramp is uncongested and speedDev=0, so the meso run is deterministic; the paper treats the objective as stochastic in general, so no cross-seed byte-identity is claimed beyond 1ramp (P8 macroreps stay mandatory for any future scored row). 3,087 trips kept, 10 edges emitted, ~0.41 s per pipeline run (paper reports 0.80 ± 0.29 s for 1ramp).

This is the value after the od_end_time fill fix (below). The first draft kept the shipped od.xml template’s interval end=3600 instead of rewriting it to 1ramp’s od_end_time=3300 (BO4Mob’s own create_od_tazrelation_xml always rewrites it), which released ~5% of demand after the OD window — vehicles that never reach the sensors — and returned 2.314704, a value that mis-attributed a TABench od-fill bug to engine drift. The corrected transform reproduces the upstream-faithful pipeline to the digit (cross-checked against BO4Mob’s own pandas update_trip_routes + parse_link_flow_xml_to_pandas on the identical edge_data.xml); the mismatch is 1ramp-only — the other three small instances have od_end_time == 3600 == the template end, so the rewrite is a no-op there (measured deltas exactly 0.0).

The smoke test pins these loose: the NRMSE in a wide band 1.5 < x < 3.5 (the measured 2.4325 with generous headroom — a version-robust ceiling, never an exact decimal, since the meso values shift with the engine build), seed-stability as max−min < 1e-3 (measured 0.0, the tol survives cross-platform float drift), and the pipeline emitting per-edge counts with trips kept. A regression pins that fill_single_od rewrites the interval end to od_end_time regardless of the template end. The 2.432471221214843 is recorded here as provenance only — it is the value under 1.27.1, explicitly not BO4Mob’s published number under 1.12.

Decision 4 — the measured SUMO 1.12 → 1.27.1 schema drift (documented, tested)

Under the shipped 1.27.1 wheel, the mesoscopic edgeData output carries no nVehContrib attribute — a naive parser keying on it silently reads 0.0 for every edge (a silent-corruption surface). BO4Mob’s own count convention is arrived + left, whose attributes still exist under 1.27.1, so the evaluation convention transfers unchanged while the simulated values do not match the 1.12 paper values. edgedata_counts therefore uses arrived + left (with a code comment naming the drift), and edgedata_has_nvehcontrib is a witness the smoke test asserts False — pinning the drift as an executable fact, not a claim. (The GT CSV column is named interval_nVehContrib; that is the PeMS-side count and is unrelated to the SUMO-side attribute drift — bo4mob_nrmse documents this so the two nVehContrib names are not conflated.)

Stage 2 (named follow-up — NOT shipped here)

A bo4mob_estimation T2 task family, its own ADR + adversarial review. The ADR-002 pinned-bfw certifier cannot apply: there is no declared BPR network (meso SUMO) and no true OD (so od_rmse / OD-fit columns and the linear identifiability report are undefined). The honest certificate is a pinned-engine observational one: re-run the pinned, seeded engine at the emitted OD and recompute NRMSE on held-out dates/hours/sensors (the 14×3 panel gives natural splits; siouxfalls-t2 already ranks on held-out count fit), macroreplicated (P8). It carries its own non-comparable scale (disclosed as not comparable to the bfw-certified families — a separate leaderboard table), because no shared-scale alternative exists (ADR-028 rejected a sumo-pinned certifier only where bfw exists; here it does not). First estimators would be prior (the Improvement% anchor) + an SPSA over the BO4Mob pipeline reusing ADR-028’s SPSAEstimator hooks; BoTorch baselines stay out (TABench does not re-host BO4Mob’s BO leaderboard). This is the estimation-side sibling of adr-030’s deferred observational-certificate ADR and should be co-designed with it. Do not start it until the certificate design is written.

Consequences

• New: src/tabench/data/bo4mob.py (the five-instance registry with a per-file SHA-256 and byte size pinned at ef571e68; the commit-pinned checksummed fetcher — streamed with a size cap, per-process .part, named HPC-only refusal; the pandas-free BO4Mob evaluation transforms; the citation + in-artifact provenance line); tests/test_bo4mob.py (14 tests — 13 core registry + fetcher-hardening + the fill_single_od od_end_time regression + the oversized- body refusal, 1 sumo-gated pipeline-liveness smoke). The six symbols BO4MOB_REGISTRY, BO4MOB_SMOKE, Bo4MobSpec, Bo4MobHpcOnlyError, fetch_bo4mob, bo4mob_citation are exported from the data layer (Bo4MobUpstreamError is available on the module).

• CI: the existing sumo job gains the 1ramp prefetch (< 30 KB) and tests/test_bo4mob.py in its pytest list; no sixth job. The data cache key hashes bo4mob.py (the xu2024 lesson) across all jobs. The registry + fetcher tests run in the core matrix (no sumo, no network — the fetcher-hardening tests mock urlopen); the smoke test importorskip('sumo')-skips there.

• Unchanged: no new dependency (numpy/stdlib core; the smoke uses the existing sumo extra — the wheel already ships od2trips and the meso-capable sumo binary); the Evaluator, the fairness gate, the TNTP/xu2024 fetchers, REGISTRY, load_scenario, and every content hash including the golden Braess hash cf00f411… (re-asserted in test_bo4mob.py). docs/model-specs.json is untouched — a data family is not a solver (the TNTP / xu2024 precedent), so tools/generate_models.py is not run.

• Gaps deliberately left (stage 2): the bo4mob_estimation task + its pinned-engine held-out-date observational certificate + estimators; the full 14×3 PeMS panel and routes_multiple; 4smallRegion’s pipeline run (only 1ramp is the CI liveness anchor); the four small instances beyond 1ramp are fetchable + checksummed (data availability) but their engine pipeline is exercised in stage 2; 5fullRegion at HPC scale.

Adversarial review

Three independent lenses (fetcher/P9 integrity, pipeline fidelity, infra), each executing repros against the uncommitted tree and the real upstream artifact.

MATERIAL (pipeline fidelity): the od.xml interval end was not rewritten to od_end_time. The first draft’s fill_single_od kept the shipped template’s end=3600; BO4Mob’s own create_od_tazrelation_xml always rewrites it to the config od_end_time. On 1ramp (od_end_time=3300) this released ~5% of demand after the OD window, so the recorded provenance NRMSE 2.314704 mis-attributed a TABench od-fill bug to engine drift; the BO4Mob-faithful value under 1.27.1 is 2.432471221214843 (bit-identical across seeds 0/1/2). The reviewer’s variant matrix isolated it: departSpeed has zero effect, the od_end_time fill is the whole delta, and the shipped numpy transforms match BO4Mob’s own pandas pipeline to the digit once fixed. The mismatch is 1ramp-only (the other three small instances have od_end_time == 3600 == the template end, deltas exactly 0.0). FIXED in fill_single_od (interval end := od_end_time), re-recorded here + in VALIDATION + ROADMAP, and pinned by an offline regression (template-end-agnostic, mutation-verifiable) plus the corrected smoke band.

Hygiene/precision, all fixed: (a) the fetcher buffered resp.read() unbounded (a hostile 256 MB body drove +256 MB RSS / cache disk before the checksum evicted it — a regression vs the fresh adr-033 standard) → per-file byte sizes pinned in the registry and a capped stream that raises Bo4MobUpstreamError before an oversized body materialises; (b) the .part-hygiene test was vacuous (its mock raised at urlopen(), before any .part existed, so it passed under a finally-removal mutant) → the mock now fails mid-body and the test is mutation-verified; (c) a concurrent-fetch .part race (loud flakes at ~25–37% under 8-way cold start) → per-process .part.<pid> suffix, with a note that fetcher.py/xu2024.py share the latent pattern (a follow-up, untouched here); (d) doc precision — the cache path (bo4mob/<key>/, not bo4mob-*), the bare registry keys (not bo4mob-<inst>), and DEFAULT_VEHTYPE “attribute-exact” (not byte-exact, since ElementTree normalises serialisation); (e) per-seed edge_data filenames in the smoke loop so a future engine exiting 0 without writing cannot re-read a stale file.

Repo-hygiene (pre-existing, surfaced this sprint): tools/generate_references.py’s internal SHIPPED dict had drifted far behind the hand-flipped ROADMAP [x] rows, so one accidental python tools/generate_references.py would have unchecked every shipped row whose bibkey was absent. Fixed defensively without running the generator: every currently-shipped bibkey was added to SHIPPED (verified by ast.literal_eval back-parse against the live ROADMAP [x] rows), and the ROADMAP footer now states the actual practice (rows are hand-maintained on shipping; the generator carries SHIPPED forward and is only for adding new canon entries).

Stage 1 still asserts no oracle (no best-known-flow claim, no certified gap): the honest-oracle surface arrives with the stage-2 certificate ADR.