ADR-033: xu2024-dataset — the Xu et al. (2024) 20-US-city cross-domain axis

Status: accepted (implemented) Date: 2026-07-15 Deciders: non-solver roadmap — redeeming the adr-006 cross-domain-axis promise File: docs/design/adr-033-xu2024-dataset.md

Context

Every scored instance in TABenchmark so far is either a hand-built analytic anchor or one of the four donated TNTP networks (Sioux Falls, Anaheim, Barcelona, Winnipeg). adr-006 (learned-model certification) named the missing piece explicitly: “The Xu et al. 2024 real-city dataset, disjoint from TNTP and CC-BY, is a natural future cross-domain test set; it is referenced, not vendored.” This ADR redeems that promise — a benchmark that claims to test whether a method (learned or classical) generalizes needs instances off the standard four.

The artifact. Xu, Zheng, Hu, Feng & Ma (2024), A unified dataset for the city-scale traffic assignment model in 20 U.S. cities, Scientific Data 11:325, DOI 10.1038/s41597-024-03149-8 (xu2024unified, canon reference 276). The data live on figshare (10.6084/m9.figshare.24235696, v4) as one 276 MB zip (file id 48908890, whole-zip md5 3f7632e00599588abecbcfc488f862b2), CC BY 4.0. Each of the 20 cities ships an OSM-derived network, an OD demand matrix, and computed equilibria from two solvers (TransCAD and AequilibraE). The integration target is the per-city AequilibraE trio: network.csv (link table + per-city-calibrated BPR α/β), od_demand.aem (the OD matrix), and assignment_result.csv (the published UE link flows).

The discovery (the load-bearing finding). The dataset’s published AequilibraE runs injected the OD demand at node ids 1..Z — which are not the tract centroids the dataset intends. This is not an inference: it is visible in the authors’ own published AequilibraE_assignment.py, which calls g.prepare_graph(np.arange(zones)+1). Measured consequence: with centroids taken as 1..Z, the shipped AequilibraE flows conserve machine-exactly (max node residual ~1e-11); with the intended tract nodes (10000000+k) conservation fails wildly, and those nodes sit a median 6.4 km from the corresponding tract. The dataset’s TransCAD side used the correct centroids (and a different connector speed), so the two shipped solver results are different instances. Any “cross-solver agreement” claim on the shipped artifacts would therefore be false, and this ADR never makes one.

Decision 1 — Ship Variant A (AS-PUBLISHED), defer Variant B

We ship the as-published AequilibraE instances: node ids 1..Z are the zone centroids, exactly as the published run treated them. These are self-consistent and machine-verifiable — the published flows conserve and (approximately) equilibrate on this graph — and need only the numpy/stdlib parsing of the trio. first_thru_node=1: the published run allowed through-centroid flow and the low-id “centroids” are real intersections, not pure zone stubs.

The corrected tract-centroid variant (Variant B) — demand at the true tract centroids, cross-checkable against the TransCAD reference — is deferred. It needs DBF/SHP/TransCAD-convention parsing, its reference is only ~1e-2 accurate, and it carries the connector-speed divergence; it is named future work, not part of this sprint.

Decision 2 — The wrong-centroid defect is a documented known-defect (P2)

The defect is recorded prominently, in the same honest spirit as the TNTP unit quirks: in the module docstring (data/xu2024.py), in each scenario card’s known_defect field, and in every scenario’s ReferenceSolution.note. The shipped instances are labelled AS-PUBLISHED throughout. The one claim the data cannot support — cross-solver TransCAD↔AequilibraE agreement — is stated as off-limits everywhere it might be tempting.

Decision 3 — P9 integration by HTTP-range extraction, on a separate registry

Data are fetched, never vendored (P9). The primary fetch path opens the figshare zip over HTTP byte ranges (_HttpRangeFile + zipfile): the central directory is read from the zip’s 276 MB tail, and only the three needed members are inflated — a single-city load transfers ~1–4 MB, never the whole archive (measured: Honolulu’s trio downloads + builds in ~12 s over ~12 range requests). Each extracted file is verified against a pinned SHA-256; a documented fallback (whole-zip download + md5 verify + extract) covers the case where a server refuses ranges. The version-immutable figshare file id keeps the URL alive across future deposit versions.

The 17 cities live in a separate XU2024_REGISTRY, deliberately not in the TNTP REGISTRY that CI prefetches (.github/workflows/ci.yml runs [fetch(s) for s in REGISTRY.values()]). So a CI run never pulls 276 MB on its own: the two rungs’ trios are fetched lazily only when their tests run, and the other 15 cities are never fetched in CI at all. load_scenario("xu2024-<city>") routes to the builder; the family key xu2024-<city> gives the P7 fairness gate the cross-domain axis for free.

Decision 4 — Honest validation: the published flows are a LOOSE reference

The published AequilibraE flows are not a best-known oracle like the TNTP *_flow.tntp solutions (those certify at AEC ~1e-15). Their own recomputed relative gap is ~1e-3 — roughly 11 orders looser — because the paper solved to a gap target of 0.001. They are used only as a provenance cross-check, never as a regression oracle. The real validation is four measured facts:

1. Builder invariants on all 20 cities (the audit below): unique links, fft > 0, cap > 0, BPR α,β ≥ 0, node-id ranges, link_id = 1..L, the AEM header size identity, and the 1..Z-centroid convention.

2. Published-flow conservation, recomputed from the flows: max node-balance residual ≤ 3e-10 (worst: New York ~2e-10; Honolulu 1.1e-11) across the 17 shipped cities. The prose bound is deliberately loose of the measurement — the residual is summation-order sensitive (two review lenses measured New York at 2.00e-10 and 2.04e-10), so a zero-headroom “≤ 2.0e-10” pin would be fragile; the test asserts the far looser < 1e-9.

3. The repo’s own bfw certifying a tighter gap on the identical instance — genuine cross-implementation agreement (AequilibraE bfw ↔ tabench bfw). On Honolulu, BiconjugateFrankWolfeModel reaches relative gap 1.07e-4 in 400 iterations (~100 s, measured 104.5 s on the dev box), with correlation 0.99992 to the published flows and Σ|Δv|/Σv = 0.96%. (The CI anchor uses a small budget and checks feasibility + correlation > 0.99; the tight number is this in-sprint measurement, not a CI pin.)

4. Correlation with the published flows as provenance (Honolulu 0.99992).

Decision 5 — The 20-city audit: 17 ship, 3 excluded and named

The builder + invariants + conservation battery was run on all 20 cities in-sprint. 17 pass and ship. 3 are excluded and named: Washington, Pittsburgh, Phoenix. Their failure is itself further evidence of the wrong-centroid defect — a handful of centroid ids in 1..Z are absent from the network as low-id nodes (they exist only as their 10000000+k tract ids) yet still carry demand (Washington 2 of 179; Pittsburgh 5 of 149; Phoenix 4 of 378), so no valid Network can be built on the 1..Z convention. They are candidates for the deferred Variant B. BPR α/β vary by city (per-city grid-search calibration, paper Table 5) but are uniform within each city; published gaps span 7.6e-4–1.37e-3.

#	City	Zones	Nodes	Links	BPR α	BPR β	Demand	Published rgap	Status
01	San Francisco	194	4986	18002	0.5	1.8	168,828	1.27e-03	ship
02	Seattle	139	6891	27361	0.6	3.0	133,900	9.18e-04	ship
03	Portland	157	8245	31939	0.5	1.2	136,471	8.97e-04	ship
04	Las Vegas	175	7823	28831	0.5	1.3	76,532	1.02e-03	ship
05	Chicago	819	14434	54469	0.5	1.2	468,616	1.21e-03	ship
06	New Orleans	185	7217	24073	0.6	1.8	59,875	7.78e-04	ship
07	Austin	199	10717	40158	0.5	1.5	202,867	1.03e-03	ship
08	Minneapolis	130	4004	15363	0.15	1.8	60,167	7.58e-04	ship
09	Dallas	328	21389	77818	0.6	1.3	200,434	1.04e-03	ship
10	Milwaukee	234	8521	30747	0.5	1.5	98,220	9.37e-04	ship
11	New York	2005	28626	99408	0.25	1.5	1,782,020	9.47e-04	ship
12	Washington	179	6136	23573	0.5	1.5	123,084	—	excluded
13	Boston	191	5542	20487	0.25	2.0	119,746	1.37e-03	ship
14	Philadelphia	389	10410	38641	0.5	1.2	233,554	9.49e-04	ship
15	Pittsburgh	149	3532	13662	0.5	2.0	48,436	—	excluded
16	Miami	108	4121	15108	0.5	1.5	39,412	8.92e-04	ship
17	Atlanta	141	5207	20243	0.2	1.5	61,069	8.10e-04	ship
18	Phoenix	378	15324	58070	0.15	1.2	284,808	—	excluded
19	Denver	175	9205	34724	0.5	1.5	120,434	1.00e-03	ship
20	Honolulu	117	2982	11205	0.5	1.5	107,515	1.04e-03	ship

Rungs

Honolulu (rung 5, 11,205 links) and San Francisco (rung 6, 18,002 links) are the CI-sized rungs (scenarios/5honolulu-xu2024.yaml, scenarios/6sanfrancisco-xu2024.yaml): small enough to fetch (~1 MB trio) and certify quickly. The other 15 cities build via the identical path but are local-only by download/solve cost (New York is ~10× Honolulu). New York, at 2,005 zones and 99,408 links, is the largest instance in the entire benchmark.

Consequences

• New: src/tabench/data/xu2024.py (registry of 17 cities; HTTP-range fetcher — primary ranged path with a narrowed whole-zip fallback that fires only on a transport error, and a pre-transfer size guard against the pinned v4 artifact on both paths; AEM/network/assignment parsers with drift guards on the 1..Z zone index and the fully-directed convention; scenario builder; citation); two scenario cards; tests/test_xu2024.py (20 tests). load_scenario routes xu2024-<city>; xu2024_scenario, XU2024_REGISTRY, XU2024_RUNGS, Xu2024UpstreamError are exported from the package root / data layer.

• Unchanged: no new dependency (numpy/stdlib only — no pandas); the Evaluator, the fairness gate, the CI prefetch, and every content hash, including the golden Braess hash cf00f411… (re-asserted in test_xu2024.py). The xu2024 data node in docs/model-specs.json stays shipped:false/not-a-solver, matching the TNTP (stabler2016transportation) precedent — a dataset is not a solver.

• Cache-eviction skip semantics (inherited from the TNTP fetcher): a checksum mismatch evicts the cached file and raises ChecksumError; a subsequent offline load in the same session then reports “data unavailable” and skips (via the house load_or_skip) — that skip reflects the eviction, not a missing initial download. CI closes this window with TABENCH_REQUIRE_DATA=1, under which any data failure is a hard error, never a skip.

• Gaps deliberately left: the corrected tract-centroid Variant B (with a TransCAD cross-check) and its three currently-excluded cities; the 15 non-rung cities are available but not CI-run. A learned-model cross-domain train/test split (TNTP → xu2024) is now possible but is future work, not shipped here.

Adversarial review

Three independent lenses (fetcher/P9 integrity, builder correctness + honest- validation claims, numerics/infra/integration), each executing repros against the uncommitted tree and the REAL figshare artifact. The first sprint of the review program with zero confirmed material defects (21 sprints reviewed, 20 with at least one confirmed material defect): every corruption, tamper, truncation, and upstream-v5-swap attack ended in a loud ChecksumError with nothing unverified left in the cache — the measure-first, integrate-second dossier (which had already solved Honolulu and found the upstream defect before any integration code existed) is the plausible cause, and the pattern is worth keeping.

Hygiene/precision findings, all fixed: the “≤ 2.0e-10” conservation prose bound had zero headroom (one lens recomputed New York at 2.037e-10 while another measured exactly 2.0e-10 — summation-order sensitivity; the BLAS lesson in prose form; restated with headroom); a failed whole-zip fallback stranded an unbounded .part file in the cache (now try/finally-unlinked); the whole-zip fallback triggered on ANY ranged-path exception including permanent structural errors, contradicting the ci.yml “never the 276 MB zip” comment (now transport-only, with ZIP_SIZE as a live pre-transfer guard raising Xu2024UpstreamError on an upstream artifact change BEFORE any bulk transfer); adr-006 still called the set “future work” (touched up); the .aem zone index and direction column are now asserted at parse time (verified true on all 17 cities — upstream drift becomes loud, not silent); wall-time claims harmonized; the checksum-eviction offline-skip semantics disclosed.

Attacked and survived (highlights): per-file sha256 checked on EVERY load, post-decompress (byte flips in all three file kinds → loud, evicted, re-fetch heals); a tampered zip member with a VALID CRC still refused; truncated ranged responses and mixed 206/200 servers never silently parse; a v5 upstream re-upload cannot poison either fetch path; the CI byte-budget claim measured TRUE against real figshare (both rungs = 1,027,033 bytes in 24 range reads + 2 probes — 0.37% of the 276 MB zip); warm-cache tests fully offline, cold-cache skips clean, TABENCH_REQUIRE_DATA=1 turns skips loud; CLI byte-parity for every existing scenario (all 12 existing content hashes identical, run/fetch outputs byte-identical against HEAD); two cold builds byte-identical (hash 1baa0c04… stable across processes and caches); the excluded-city analysis reproduced exactly (the missing low-id centroids still carry demand — further evidence of the wrong-centroid defect); the Honolulu bfw anchor reproduced to every printed digit (rgap 1.074e-4 @ 400 iters, corr 0.999919); no cross-solver claim vs TransCAD anywhere (repo-wide sweep — every mention is a disclaimer); tools/generate_models.py reproduces the shipped MODELS.md byte-identically; the golden Braess hash untouched.