TABenchmark Angle A: A Scenario × Model Cross-Evaluation Matrix with SimOpt-Style Experiment Machinery

TABenchmark — Angle A: The Scenario × Model Matrix

Organize the benchmark as a cross-evaluation matrix Scenario × Model × Task, adapting SimOpt’s Model/Problem/Solver split: a Scenario is the problem instance (network + demand + cost config + observation config), an AssignmentModel is the method under test (1956 incremental loading through 2025 GNN surrogates), and the harness — never the model — computes all metrics.

Core abstractions

class Scenario:                      # frozen, content-hashed, declarative (no code)
    network: Network                 # capacity, fft, bpr_b, bpr_power, toll, first_thru_node
    demand: DemandMatrix
    cost_config: CostConfig          # per-network toll/distance weights (not in .tntp!)
    ref: ReferenceSolution | None    # best-known flows + objective + provenance/hash

class AssignmentModel(ABC):
    capabilities: Caps   # deterministic, seedable, iterative, emits_path_flows,
                         # self_reports_gap, external_engine, needs_training
    factors: dict[str, FactorSpec]   # typed hyperparameters with defaults (SimOpt-style)
    def setup(self, scenario: Scenario, rngs: RNGBundle) -> None: ...
    def run(self, budget: Budget) -> Iterator[FlowState]: ...   # yield at checkpoints

@dataclass(frozen=True)
class FlowState:
    link_flows: np.ndarray
    path_flows: PathFlows | None
    coords: BudgetCoords             # iters, model_wall_ms, sp_calls, engine_evals
    self_report: dict                # model's own gap/objective — recorded, never trusted

class Evaluator:                     # harness-side, model-blind
    def score(self, s: Scenario, st: FlowState) -> Metrics:
        # AON shortest paths at costs(st.link_flows) -> relative gap, AEC,
        # Beckmann objective, RMSE vs s.ref, TSTT; SUE-gap for SUE tasks

The load-bearing decision: every convergence metric is recomputed by the Evaluator from emitted link flows (relative gap = 1 − AON-cost/current-cost, AEC, Beckmann). This is SimOpt’s “postreplicate, never trust in-run estimates” principle applied to the deterministic setting, and it is what lets a convex-programming solver and an opaque simulator share one results schema: anything that emits link flows gets an externally verified gap. Self-reports are logged and diffed against harness values as an honesty check.

Randomness. One MRG32k3a generator, SimOpt’s fixed (stream, substream, subsubstream) = (macrorep m, declared randomness-source i, replication r) schema, identical stream universe for all (scenario, model) pairs so results are order-independent. Each stochastic model declares its sources of randomness (probit draws, simulator seed, NN init); CRN is switchable across models-on-a-scenario (default on) and across macroreps (default off). Dedicated streams for evaluation, post-evaluation, and bootstrap. External engines must accept a seed passthrough (--seed à la BO4Mob’s SUMO adapter) or be declared seedable=False, which forces larger M.

Budgets. Three budget classes — B-iter(K), B-time(seconds), B-eval(sp_calls or engine_evals) — but every checkpoint records all four coordinates, so any curve can be re-sliced post hoc. Wall-clock is normalized by a per-machine calibration constant (time a standard AON pass on SiouxFalls) and the clock stops during harness evaluation, so checkpointing cadence (geometric in budget) doesn’t distort timing.

Scenario & data model

Scenario = data directory + JSON config, no code (BO4Mob convention). Tiered ladder: 0braess (hand-checkable), 1siouxfalls, 2anaheim/easternmass, 3barcelona/winnipeg/chicagosketch, 4chicagoregional/…. TNTP data is downloaded on demand with per-file checksums and auto-generated citations (repo terms forbid vendoring); a defensive parser handles first_thru_node semantics, ~ comments, and trailing ;. Per-network metadata modules encode unit conventions (SiouxFalls 0.01-h times), generalized-cost weights, and known defects (Austin duplicate links, Chicago-Regional ramp errors) machine-readably. The six published best-known UE solutions (SiouxFalls objective 42.31335287107440, etc.) are regression oracles stored as ReferenceSolution with provenance. Observation-level factors (count coverage, noise, periods N, trajectories) are first-class scenario factors, reserved as dials for the estimation-task track; sensor sets carry the distinct-nonzero-columns identifiability flag.

Task & evaluation protocol

Tasks bind a scenario to a target and metric set: TASK-UE (relative gap + link-flow RMSE vs best-known), TASK-SUE(θ) (SUE gap / fixed-point residual), TASK-PREDICT (amortized flow prediction, scored identically to UE at zero iteration budget), with estimation tasks (OD-from-counts) schema-compatible later. Compatibility metadata (deterministic? provides paths? needs training?) auto-filters the matrix, SimOpt-style.

Protocol per (scenario, model, task):

  1. Run stage: M macroreps (M=1 on the deterministic track — no wasted replication machinery); each yields a checkpoint sequence.

  2. Post stage: Evaluator scores every checkpoint; metrics that are themselves stochastic (simulator TSTT) get N post-evaluations on the dedicated stream.

  3. Aggregation: progress curves (log relative gap vs each budget coordinate, mean/quantile bands over macroreps); α-solve-time solvability curves for α ∈ {1e-2, 1e-4, 1e-6} (CDF of first budget where gap ≤ α); Moré–Wild data/performance profiles aggregated over the scenario tier; AUC scatter; all CIs by bootstrapping macrorep outputs (dedicated stream), never parametric.

  4. Non-convergent models are first-class: models that cannot reach equilibrium (DTA simulators, surrogates) are reported as gap-at-budget points and appear in solvability profiles as censored (never-solve) entries — honest, not excluded. Results land in one parquet schema: (scenario_hash, model_id, model_version, task, m, coords, metrics{...}, self_report{...}, env{python, package versions, hardware, calibration_constant}).

Frank-Wolfe (1975) and a GNN surrogate (2025) in one harness

Frank-Wolfe: capabilities = {deterministic, iterative, self_reports_gap}. run() yields a FlowState per iteration with sp_calls incremented; harness gap should match its self-report to numerical precision (a built-in correctness test). M=1; its curve is the classical gap-vs-iteration profile, now also available vs normalized time and SP calls.

GNN surrogate: capabilities = {seedable, needs_training, emits_path_flows=False}. It registers a training-data card declaring exactly which scenario hashes and demand perturbations it trained on; the harness enforces disjointness from evaluation scenarios (frozen published splits + a demand-perturbation generator for held-out instances). run() emits one checkpoint (or a few, if it does iterative refinement); coords records inference wall-time and zero SP calls. The Evaluator computes its relative gap and flow RMSE exactly as for FW — so the headline comparison “GNN reaches gap 3e-3 in 0.1 normalized seconds; FW needs 40 iterations to match” is native output, and its inability to reach 1e-6 shows up honestly in solvability profiles. Seeds over weight init/training give M macroreps capturing training variance.

External binaries (SUMO, MATSim, TAP-B) plug in via subprocess adapters exposing the same setup/run contract, file-based I/O, and seed passthrough.

Repo layout

tabenchmark/
  core/          # factors, rng (MRG32k3a bundle), budget, flowstate, capabilities
  scenarios/     # tntp parser, fetcher(+checksums), registry, network_notes/, configs/*.json
  models/        # base.py, fw.py, msa.py, bfw.py, algorithm_b.py, sue/*, 
                 # adapters/{subprocess_base, sumo, matsim}, surrogates/
  evaluation/    # gap.py, metrics.py, profiles.py, bootstrap.py
  experiments/   # runner.py (parallel macroreps), grid.py, postprocess.py
  cli/           # evaluate_once.py, run_benchmark.py   (BO4Mob's two-mode pattern)
demos/           # laddered: demo_scenario, demo_model, demo_experiment
tests/  docs/    # DATASHEET.md, extension guides, per-network defect notes

Proper pyproject.toml pip package with pinned deps + Docker + CI (fixing BO4Mob’s gaps); deterministic run-folder naming encoding all factors.

Top-5 design risks & mitigations

  1. Metric gaming / self-report drift. Models could tune to their own gap formula. → Harness recomputes everything from flows; self-report vs harness discrepancies flagged in report cards; reference-solution regression tests on evaluator code.

  2. Unfair cross-vintage timing (C engines vs Python, 1975 vs 2025 hardware). → Multi-axis budgets with SP-call and iteration counts as hardware-free primaries; calibration-normalized wall-clock as secondary; pinned Docker environment recorded per run.

  3. Black boxes without checkpoints or seeds. → Capability metadata degrades gracefully: final-state-only scoring, gap-at-budget, censored solvability entries; unseedable engines require larger M and are labeled as such.

  4. Data/reference rot (TNTP unit chaos, wrong best-knowns, license limits). → No vendoring; checksummed fetcher; per-network metadata + defect notes; validate parsers against the six published objectives at import time.

  5. ML train/test contamination. → Mandatory training-data cards, scenario content-hashing, frozen splits, harness-side disjointness enforcement, and held-out demand-perturbed scenarios generated from dedicated RNG streams.