`sumo-duaiterate` — the first external-dynamic (EDOC-1) row¶

What. sumo-duaiterate wraps SUMO’s dynamic user-assignment driver duaIterate.py (eclipse-sumo 1.27.1) — an iterated duarouter best-response over a mesoscopic dynamic network load. Unlike the static sumo-marouter row, there is no declared cost law to certify a gap against, so the certificate is observational (EDOC-1, ADR-036 / ADR-037): the engine is the instance, and the harness re-derives every scored number by re-running the pinned engine on the model’s emitted plans.

Score. The frozen-field best-response gap RG_D1 = Σ(c_driven − c_BR) / Σc_driven, a door-to-door quantity on its own scale — not the Wardrop relative_gap of the static rows, so it lives on a separate leaderboard table (R8 non-comparability).

How this notebook is graded¶

A notebook never claims a number it does not compute in that cell. Every quantity below — the certified RG_D1, feasibility, the negative-control separation, the duarouter cross-check — is recomputed here by the same EdocEvaluator the benchmark uses. The model emits artifacts; the certifier, model-blind, re-runs the pinned replay and recomputes the score (a doctored experienced record diverges from the replay and is censored).

%matplotlib inline
# Setup. `sumo-duaiterate` is behind the OPTIONAL `eclipse-sumo` extra
# (`pip install tabench[sumo]`); this guard raises politely when it is missing rather than
# failing with an ImportError deep in a later cell.
import importlib.util

if importlib.util.find_spec("sumo") is None:
    raise RuntimeError("this tutorial needs the optional 'sumo' extra: pip install tabench[sumo]")

from collections import Counter

import matplotlib.pyplot as plt

from tabench.metrics.edoc_gaps import EdocEvaluator
from tabench.models.adapters.sumo_duaiterate import (
    SumoDuaIterateAdapter,
    duarouter_recost_crosscheck,
    make_replay_runner,
    reference_scenario,
)

The instance¶

The pinned reference is a diamond: two O→D routes, with a 1-lane capacity drop on the route-distinguishing edge a2 (route A is free-flow-shorter, so an all-or-nothing assignment piles onto the bottleneck). The engine identity + version + seed + the meso config + Δ + the lane/geometry dials are all inside the content hash (P2) — the engine is part of the instance.

scenario = reference_scenario()

print(f"instance       : {scenario.name}")
print(f"content hash   : {scenario.content_hash()}")
print(f"engine (pinned): {scenario.engine} {scenario.engine_version}  seed={scenario.seed}")
print(f"network        : {scenario.n_edges} edges, {scenario.n_agents} agents (all O->D)")
print(f"lanes          : {scenario.lanes_of()}   (a2 is the 1-lane bottleneck)")
print(f"declared       : separation >= {scenario.separation_factor:g}x, "
      f"floor {scenario.floor_seconds:g}s, Delta = {scenario.dt:g}s x {scenario.n_intervals}")

instance       : sumo-duaiterate-ref
content hash   : b502f21880a28ff8e4e20531d294069cc63f7d47427bd3c148db8a807b467f29
engine (pinned): eclipse-sumo 1.27.1  seed=42
network        : 4 edges, 720 agents (all O->D)
lanes          : {'a1': 2, 'a2': 1, 'b1': 2, 'b2': 2}   (a2 is the 1-lane bottleneck)
declared       : separation >= 5x, floor 15s, Delta = 300s x 16

Emit — plans `P`, and `X` from the adapter’s own pinned replay¶

The model runs duaIterate (an MSA-style best-response loop) to produce the plans P. The adapter then runs its own pinned meso replay of P to produce the door-to-door experienced record X and the frozen cost field: X and the field are defined by the pinned replay map (a hashed instance field), not scraped from the solver’s internals — those are provenance (ADR-037). All ADR-027 subprocess discipline applies (wheel SUMO_HOME, one wall deadline, teleport off so gridlock censors, RuntimeError on any engine failure).

adapter = SumoDuaIterateAdapter(iterations=12)
emitted = adapter.emit(scenario, wall_seconds=300.0)

split = Counter(" -> ".join(route) for route, _dep in emitted.plans.values())
print(f"emitted plans  : {len(emitted.plans)} agents")
print(f"route split    : {dict(split)}")
print(f"provenance     : engine {emitted.engine_version}, seed {emitted.seed}")

emitted plans  : 720 agents
route split    : {'a1 -> a2': 358, 'b1 -> b2': 362}
provenance     : engine 1.27.1, seed 42

Certify (G0–G4 + `RG_D1`)¶

The certifier is model-blind: it checks the engine pin (G0), re-runs the pinned engine in zero-replanning replay twice (G1 — replay fidelity + a determinism double), verifies the plan↔demand bijection (G2), the two-sided delivery census with departDelay in every cost (G3), and conservation (G4); then it rebuilds the frozen field and recomputes RG_D1 itself.

runner = make_replay_runner(deadline=None)
metrics = EdocEvaluator(scenario, runner).certify(emitted)

print(f"feasible       : {metrics['feasible']}")
print(f"RG_D1 (gap)    : {metrics['rg_d1']:.5f}   ({int(metrics['n_improvers'])} strict improvers)")
print(f"resolution     : delta {metrics['delta']:.2f}s vs floor {scenario.floor_seconds:g}s "
      f"-> floor_gap {metrics['floor_gap']:.4f}, sub_floor={int(metrics['sub_floor'])}")
print(f"delivery       : max backlog {metrics['max_backlog']:.1f}s "
      f"(bound {scenario.backlog_bound:g}s), BR-path coverage {metrics['br_coverage']:.2f}")

feasible       : 1.0
RG_D1 (gap)    : 0.02703   (366 strict improvers)
resolution     : delta 8.75s vs floor 15s -> floor_gap 0.0911, sub_floor=1
delivery       : max backlog 0.0s (bound 600s), BR-path coverage 1.00

The negative control — the metric has attributable signal¶

RG_D1 must separate a bad assignment from a good one. An all-or-nothing control (duaIterate -l 1, free-flow shortest path) piles everyone onto the free-flow-shorter route, congesting the bottleneck → a large gap; the converged assignment above balances → a small gap. Their ratio is the attributable negative control ADR-030 said the dynamic track lacked.

aon = SumoDuaIterateAdapter(iterations=1).emit(scenario, wall_seconds=300.0)
aon_metrics = EdocEvaluator(scenario, runner).certify(aon)
separation = aon_metrics["rg_d1"] / metrics["rg_d1"]

print(f"AON control    : RG_D1 {aon_metrics['rg_d1']:.5f}")
print(f"converged      : RG_D1 {metrics['rg_d1']:.5f}")
print(f"separation     : {separation:.1f}x   (declared >= {scenario.separation_factor:g}x)")

AON control    : RG_D1 0.13921
converged      : RG_D1 0.02703
separation     : 5.1x   (declared >= 5x)

Cross-check (R3) — the field the certifier scores == the field the engine reads¶

The pinned duarouter re-costs the driven plans on the frozen field and must agree with the substrate’s own field arithmetic within the declared tolerance; a disagreement RAISES (a harness correctness failure, not a censor).

replay = runner(scenario, emitted.plans)
r3 = duarouter_recost_crosscheck(
    scenario, emitted.plans, replay, deadline=None, tolerance_s=scenario.r3_tolerance_s,
)
print(f"R3 duarouter vs substrate: max {r3['r3_max_s']:.2f}s, mean {r3['r3_mean_s']:.2f}s "
      f"(tolerance {r3['r3_tolerance_s']:g}s)")

R3 duarouter vs substrate: max 5.10s, mean 1.34s (tolerance 15s)

Visualize¶

One figure, every bar a number certified above: the negative-control separation on the RG_D1 scale, with the Δ=300 resolution floor for reference. The converged gap sitting below the floor is the honest disclosure that a well-converged solution’s residual is below the aggregation resolution — the metric still cleanly separates it from the AON control.

tabench.viz is a road-network flow visualizer; RG_D1 is a scalar gap on its own leaderboard scale, not a per-link flow on a road Network, so this notebook plots the certified quantities directly (a house bar chart, not tabench.viz) — the same rule the bottleneck/newell rows follow (adr-035).

fig, ax = plt.subplots(figsize=(6.2, 3.2))
bars = {
    "AON control": aon_metrics["rg_d1"],
    "converged": metrics["rg_d1"],
    "resolution floor": metrics["floor_gap"],
}
ax.bar(list(bars), list(bars.values()), color=["#c44e52", "#4c72b0", "#9a9a9a"])
ax.set_ylabel("RG_D1  (frozen-field BR gap)")
ax.set_title(f"{scenario.name}: negative control separates ({separation:.1f}x)")
for i, v in enumerate(bars.values()):
    ax.text(i, v, f"{v:.3f}", ha="center", va="bottom", fontsize=9)
fig.tight_layout()
fig

../../_images/3b18a8b5666684e9bfd9533a534900cadd8fa09f93ed0b2a8299267a97640477.png

Takeaways & pointers¶

Observational, not cost-matched. With no declared latency function, the matched object is the engine itself under G1 replay fidelity; the certified claim is P is best-response stable under the pinned replay map.
The engine is the instance. Engine identity + version + seed + config are inside the hash; a version drift RAISES at certify time (G0). The solver that produced P is as model-blind as every other benchmark model.
RG_D1 is not Wardrop relative_gap. A separate leaderboard table (R8); never compared to the static rows’ gap.
Provenance vs score. duaIterate’s own convergence print and its internal experienced record are provenance only — X and the field are defined by the pinned replay (ADR-037), which is why a self-reported gap is never gated.

See ADR-037 for the measured family constants and the deterministic-track (single-seed) decision.

sumo-duaiterate — the first external-dynamic (EDOC-1) row¶