TABenchmark Design Proposal — Contract/Plugin-First Extensibility (Angle C)

TABenchmark: Contract-First Design Proposal (Angle C)

Core abstractions

Everything hangs on one minimal contract plus declared capabilities. The harness — not the model — computes all scored metrics.

@dataclass(frozen=True)
class Capabilities:
    paradigm: Literal["static_ue","static_so","sue","dta","simulation","learned"]
    determinism: Literal["deterministic","stochastic"]
    inputs_required: frozenset[str]     # {"od_matrix"} | {"link_counts"} | ...
    outputs: frozenset[str]             # {"link_flows","path_flows","flow_dist","od_estimate"}
    provides_gap: bool                  # can self-certify equilibrium gap
    seedable: bool
    trained_on: tuple[str, ...]         # scenario-family lineage IDs (learned models)

class TrafficAssignmentModel(ABC):
    capabilities: ClassVar[Capabilities]
    factors: ClassVar[dict[str, FactorSpec]]   # SimOpt-style: defaults, type, bounds

    @abstractmethod
    def solve(self, scenario: Scenario, budget: Budget,
              rng: RngBundle, callback: Trace | None = None) -> ResultBundle: ...

class WhiteBoxMixin:                    # optional richer surface
    def link_cost(self, v): ...
    def link_cost_jacobian(self, v): ...
    def objective(self, v): ...         # e.g. Beckmann; enables duality-gap checks

Budget counts scenario evaluations / iterations, never wall clock (wall clock is recorded, not ranked). Trace receives (budget_fraction, solution) pairs, feeding progress curves. ResultBundle = link flows (+ optional path flows / distributions / OD estimate), the trace, self-reported gap, seeds consumed, and engine versions. Self-reported numbers are provenance only — the harness recomputes every metric.

Key soundness move: for static tasks with known cost functions, relative gap = (TSTT − SPTT)/SPTT is a property of (link_flows, scenario). The harness computes it externally for any model whose output passes a demand-feasibility audit (flow conservation, OD totals). So a black-box GNN still receives a certified gap; models producing infeasible flows are flagged, never silently scored.

Adapters subclass the same ABC: SubprocessAdapter / DockerAdapter (DTALite, SUMO, MATSim, arbitrary binaries — BO4Mob’s seeded-subprocess pattern: write inputs, shell out with explicit --seed, parse outputs) and CallableAdapter (torch/GNN callables). Unseedable engines must declare seedable=False and get macroreplicated instead.

Registry: models register via Python entry points ([project.entry-points."tabench.models"]), so third parties pip install tabench-mymodel and tabench run --model mymodel discovers it — BO4Mob’s string registry, made pip-native.

Task layer (SimOpt’s Model/Problem split): a Task wraps a Scenario + ObservationSpec + objective (solve-UE, OD-estimation-from-counts, calibration-from-trajectories). Task↔model compatibility is enforced automatically from Capabilities (e.g., a counts-only task refuses a model requiring full OD), so one network spawns many problems cheaply.

Scenario & data model

A Scenario is an immutable, content-addressed spec, not code: canonical network arrays (defensively parsed TNTP/GMNS, first_thru_node centroid semantics honored in every shortest path), demand, cost parameters including per-network toll/distance weights and units metadata (SiouxFalls’s 0.01-hour times documented, not guessed), and a machine-readable known-defects registry (Austin duplicate links, Chicago-Regional ramp errors). scenario_hash = SHA256(canonical serialization); leaderboard entries pin the hash, so a silently edited network can never masquerade as the benchmark scenario.

Data are fetched on demand, never vendored (TNTP terms are academic-only, non-OSI): cached downloads with per-file checksums and auto-generated citation strings. Best-known UE solutions (SiouxFalls 42.31335287…, Barcelona, Winnipeg, Chicago-Sketch/Regional, Anaheim) become regression oracles.

Observation levels derive from stored ground truth, Hazelton-style: synthetic scenarios store realized integer route flows x; every data level is a projection — full OD, per-period link counts y=Aₘx (number of periods N and monitored-link mask are dials; per-period counts distributed, never time-averages), noisy counts, trajectory samples, stale-prior OD (Gamma pseudo-counts). Each (network, sensor-set) config reports the identifiability condition — distinct nonzero columns of the masked incidence matrix — and the suite deliberately includes violating configs. Incidence matrices, masks, and priors ship as first-class artifacts.

Scenario ladder: 0braess (hand-checkable), 1siouxfalls, 2anaheim/2eastmass, 3barcelona/3winnipeg/3chicagosketch, 4chicagoregional+, each with a declarative YAML budget table (BO4Mob Table-3 style).

Task & evaluation protocol

Two tracks. Deterministic: M=1, no postreplications; certified relative gap vs budget as Moré–Wild data profiles; convergence regression against best-known objectives. Stochastic (SUE, simulators, ML): full SimOpt machinery — M macroreps, N independent postreplications for unbiased objective estimates at recommended solutions, normalization against shared x₀ and x*/best-known, MRG32k3a with the fixed (macrorep, randomness-source, replication) stream schema, switchable CRN layers, bootstrap CIs, mean/quantile progress curves and α-solve-time solvability profiles.

A single metrics/ module defines each metric once: relative gap, link-flow NRMSE vs oracle, OD/route-flow error, and for distributional outputs empirical coverage of 95% intervals (point accuracy and uncertainty calibration scored separately — overdispersion misspecification corrupts the latter first). Feasibility-aware reporting: count-matching error and equilibrium gap are reported as a pair, never collapsed to one scalar.

Honest non-equilibrium handling: models without a gap (DTA heuristics, microsimulators, surrogates) appear in a “no equilibrium certificate” leaderboard column with externally computed gap where feasible-flow output permits; otherwise scored only on observable-fit metrics. No imputed zeros, no shared single ranking across tracks.

Fairness gates the harness enforces: refuse any run where trained_on intersects the evaluation scenario’s family lineage; identical budgets in evaluations; warm starts are declared factors; every run folder (deterministic BO4Mob-style naming) carries manifest.json (scenario hash, package+engine versions, seeds, env, git commit) and the leaderboard rejects incomplete manifests. CI conformance suite for contributions: solve 0braess to the known answer, budget respected, seeded-twice determinism, declared outputs actually returned, adapter smoke test in Docker.

1975 Frank–Wolfe and a 2025 GNN, side by side

Frank–Wolfe: ~200-line NumPy/SciPy class with WhiteBoxMixin; Capabilities(paradigm="static_ue", deterministic, provides_gap=True, inputs={"od_matrix"}). solve() iterates AON + line search, emitting flows per iteration to Trace. Deterministic track: gap-vs-iteration profile, SiouxFalls objective as a CI regression test.

GNN surrogate: CallableAdapter around a torch model; Capabilities(paradigm="learned", provides_gap=False, trained_on=("tntp-small-v1",)). solve() is one forward pass (budget = 1 evaluation) returning link flows; the harness audits feasibility, computes the external gap and NRMSE, and the fairness gate blocks evaluation on its training families. Optionally macroreplicated over seeds for distributional scoring. Both yield ResultBundles consumed by identical profile machinery — the leaderboard legitimately shows “gap 1e-14 in 200 evaluations” vs “gap 3e-2 in 1 evaluation,” which is the scientific comparison.

Repo layout

tabench/            # installable package (pyproject.toml — fixes BO4Mob gap)
  core/             # model.py task.py scenario.py factors.py results.py rng.py budget.py
  models/           # fw.py msa.py path_based.py sue_logit.py ...
  adapters/         # subprocess.py docker.py callable.py; dtalite/ sumo/ matsim/
  data/             # fetchers (tntp.py, gmns.py), checksums, citations, defects.yaml
  observe/          # count/trajectory/noise samplers, identifiability checks
  metrics/  experiments/  cli.py
scenarios/*.yaml    # hashed declarative specs + budget tables
conformance/        # plugin contract test suite (`tabench validate`)
demos/              # demo_model.py demo_task.py demo_experiment.py (laddered)
docs/               # DATASHEET.md, extending.md, per-network unit notes
tests/  Dockerfile  .github/workflows/   # per-adapter pinned engine images

Top-5 design risks & mitigations

  1. External-engine drift (SUMO/DTALite versions change results) → per-adapter Docker images pinned by digest; engine version in every manifest; CI smoke tests; unpinned runs flagged non-leaderboard.

  2. Cross-vintage budget incomparability (an FW iteration ≠ a simulation ≠ a forward pass) → budget = scenario evaluations with wall clock reported separately; multi-column leaderboard segmented by track/certificate, never one global ranking.

  3. Learned-model test leakage → trained_on lineage declarations enforced by the harness, required training-data manifests, plus a held-out generator (demand/topology perturbations of base scenarios) so there is no fixed test set to memorize.

  4. Uncomputable certificates for black boxes → feasibility audit before external gap; explicit “no certificate” labeling; pair-reported fit-vs-gap metrics rather than imputation.

  5. Contract friction deterring contributors → one abstract method + one declaration; cookiecutter template, laddered demos, and tabench validate giving actionable conformance errors before any PR review.