`matsim` — the first agent-based, stochastic-track EDOC row¶

What. matsim wraps the MATSim 2025.0 co-evolutionary agent-based simulator (Horni et al. 2016) — QSim queue dynamics plus route/selection-only replanning (ReRoute + ChangeExpBeta) — as the second EDOC-1 observational row (ADR-036 / ADR-039). Like sumo-duaiterate there is no declared cost law: the engine is the instance, and the certifier re-derives every scored number by re-running the pinned engine in zero-replanning replay (lastIteration = firstIteration) on the model’s emitted plans (G1). The engine is Java-only — no pip extra — addressed via TABENCH_MATSIM_HOME + TABENCH_JAVA_HOME (pinned matsim-2025.0 on Temurin 21.0.11+10).

Score. MATSim is genuinely stochastic (global.randomSeed moves the converged state), so this is the benchmark’s first stochastic-track external row (ADR-036 R5): the score is the mean frozen-field best-response gap RG_D1 over a pinned 5-seed macrorep list, with a percentile-bootstrap CI — a door-to-door quantity on its own scale, not the Wardrop relative_gap of the static rows, so it lives on a separate leaderboard table (R8 non-comparability). Single-seed readouts are structurally impossible: certify_row is the row’s only score entry point.

How this notebook is graded¶

A notebook never claims a number it does not compute in that cell. Every quantity below — the per-seed certified RG_D1, the row mean and its bootstrap CI, the negative-control separation — is recomputed here by the same substrate the benchmark uses (EdocEvaluator per macrorep, tabench.edoc.macrorep for the row). The model emits artifacts; the certifier, model-blind, re-runs the pinned replay per seed and recomputes the score (a doctored experienced record diverges from the replay and is censored).

%matplotlib inline
# Setup. `matsim` is a JAVA-ONLY engine behind NO pip extra (adr-039): the toolchain is
# addressed via TABENCH_MATSIM_HOME (the matsim-2025.0 release layout) and TABENCH_JAVA_HOME
# (the pinned Temurin 21.0.11+10 JDK). This guard raises politely when the toolchain is
# unaddressed rather than failing with a G0 error deep in a later cell.
from tabench.models.adapters._matsim_io import matsim_available

if not matsim_available():
    raise RuntimeError(
        "this tutorial needs the pinned MATSim toolchain: set TABENCH_MATSIM_HOME to the "
        "matsim-2025.0 release dir and TABENCH_JAVA_HOME to the Temurin 21.0.11+10 JDK "
        "(see docs/design/adr-039-matsim.md for the exact download pins)"
    )

import matplotlib.pyplot as plt

from tabench.models.adapters.matsim_edoc import (
    certify_row,
    matsim_reference_scenario,
    negative_control_separation,
)

The instance¶

The pinned reference is a diamond behind single entry/exit plumbing edges (MATSim activities sit on links): route A (a1,a2) is free-flow-shorter, and the 1-lane capacity drop on a2 is where MATSim’s outflow queue physically sits — the cost signal lands exactly on the route-distinguishing edge. Engine identity + version (jar md5 + JDK major), the full-JDK pin, the numberOfThreads=1 determinism pins, the pinned seed list (R5), Δ, and every floor/deadline constant are all inside the content hash (P2) — the engine is part of the instance.

scenario = matsim_reference_scenario()

print(f"instance       : {scenario.name}")
print(f"content hash   : {scenario.content_hash()}")
print(f"engine (pinned): {scenario.engine} {scenario.engine_version}")
print(f"seed list (R5) : {scenario.seed_list}   (P8 macroreps; a model never picks its seed)")
print(f"network        : {scenario.n_edges} edges, {scenario.n_agents} agents (all O0 -> D2)")
print(f"lanes          : {scenario.lanes_of()}   (a2 is the 1-lane drop, 600 veh/h)")
print(f"declared       : separation >= {scenario.separation_factor:g}x, "
      f"floor {scenario.floor_seconds:g}s, Delta = {scenario.dt:g}s x {scenario.n_intervals}")

The negative control — mean-vs-mean over the SAME pinned seeds¶

RG_D1 must separate a bad assignment from a good one as a mean, not per lucky seed (ADR-039 ruling: single-seed anchors would violate R5’s single-readout ban). The control is the engine-routed free-flow AON state (iterations=0, deterministic); the converged state co-evolves for 10 iterations under each pinned seed. The gate compares floor-displayed values, so a topology whose anchors both sit at RG_D1 = 0 (the shared-edge drop, which self-certifies) separates 1.0x and is refused at construction — this cell also separation-vets the family, which certify_row requires. (~2 min: 2 states x 5 macroreps, each one JVM co-evolution + two JVM replays.)

anchors = negative_control_separation(scenario, wall_seconds=900.0)

print(f"AON control    : mean RG_D1 {anchors['control_rg_d1']:.5f}   (deterministic at it.0)")
print(f"converged      : mean RG_D1 {anchors['converged_rg_d1']:.5f}")
print(f"separation     : {anchors['separation']:.2f}x   (displayed values; "
      f"declared >= {anchors['separation_factor']:g}x)")

Certify the row — P8 macroreps + bootstrap CI (R5)¶

Each pinned seed gets its own emission (a fresh co-evolution under that seed) and its own full certificate: G0 pins (engine + full JDK), G1 replay fidelity twice (the determinism double on the tie-sorted canonical hash), G2 demand bijection with exact departures, G3 delivery census with insertion wait in every cost, G4 conservation, the resolution-floor gate, and RG_D1 against the certifier-owned time-dependent shortest path. The row score is the mean with a percentile-bootstrap CI on the reserved SOURCE_BOOTSTRAP stream — and ANY censored macrorep censors the whole row (a subset mean would re-open seed shopping).

row = certify_row(scenario, iterations=10, wall_seconds=900.0)
m = row.metrics

print(f"feasible       : {m['feasible']:.0f}   ({m['n_seeds']:.0f} macroreps; "
      "ANY censored seed censors the row)")
for seed in scenario.seed_list:
    ps = row.per_seed[seed]
    print(f"  seed {seed:>6}: RG_D1 {ps['rg_d1']:.5f}   delta {ps['delta']:.2f}s   "
          f"max backlog {ps['max_backlog']:.0f}s")
print(f"row score      : mean RG_D1 {m['rg_d1_mean']:.5f}   "
      f"{m['ci_level']:.0%} CI [{m['rg_d1_ci_lo']:.5f}, {m['rg_d1_ci_hi']:.5f}]")
print(f"resolution     : floor_gap {m['floor_gap']:.4f} -> sub_floor={m['sub_floor']:.0f}")

Visualize¶

One figure, every mark a number certified above: the five per-seed gaps, the row mean with its bootstrap CI band, the resolution floor, and the AON control mean. The spread of the dots is the seed lottery R5 exists to average over — which is why a single-seed readout is never the score. (tabench.viz is a road-network flow visualizer; RG_D1 is a scalar gap on its own leaderboard scale, so this notebook plots the certified quantities directly — a house chart, not tabench.viz — the same rule the bottleneck/newell rows follow, adr-035.)

fig, ax = plt.subplots(figsize=(6.6, 3.4))
seeds = list(scenario.seed_list)
per = [row.per_seed[s]["rg_d1"] for s in seeds]
ax.scatter(range(len(seeds)), per, color="#4c72b0", zorder=3, label="per-seed RG_D1")
ax.axhline(m["rg_d1_mean"], color="#4c72b0", lw=1.5, label=f"mean {m['rg_d1_mean']:.3f}")
ax.axhspan(m["rg_d1_ci_lo"], m["rg_d1_ci_hi"], color="#4c72b0", alpha=0.15,
           label=f"{m['ci_level']:.0%} bootstrap CI")
ax.axhline(m["floor_gap"], color="#9a9a9a", ls="--", lw=1.2,
           label=f"resolution floor {m['floor_gap']:.3f}")
ax.axhline(anchors["control_rg_d1"], color="#c44e52", lw=1.5,
           label=f"AON control {anchors['control_rg_d1']:.3f}")
ax.set_xticks(range(len(seeds)))
ax.set_xticklabels([str(s) for s in seeds])
ax.set_xlabel("pinned macrorep seed (R5)")
ax.set_ylabel("RG_D1  (frozen-field BR gap)")
ax.set_title(f"{scenario.name}: {anchors['separation']:.1f}x separation, "
             "mean +/- CI over 5 seeds")
ax.legend(fontsize=8, loc="center right")
fig.tight_layout()
fig

Takeaways & pointers¶

Stochastic track, structurally. The seed list is a hashed instance field; per-seed instances are derived by the harness; the score exists only as the macrorep mean + CI (tabench.edoc.macrorep). A model never chooses its seed, and one censored seed censors the row.
The corrected R10 record. The same-timestamp event-tie permutation is a multithreading artifact (measured at numberOfThreads=8), not a replay effect — so threads=1 is pinned in global/qsim/eventsManager (hashed), the G1 hash is the tie-sorted canonical stream, and raw-byte replay identity at threads=1 is a bonus check, never the gate.
The replay map is seed-independent (measured: replaying the same plans under a different pinned seed is canonically identical). Reusing one seed’s artifacts across macroreps therefore collapses to legal optimization — the mean is still harness-computed per seed against seed-dependent emissions.
RG_D1 is not Wardrop relative_gap. A separate leaderboard table (R8); never compared to the static rows.
Provenance vs score. MATSim’s scores/scorestats and its internal 15-min replanning field are provenance only — X and the frozen field are defined by the pinned replay map (the ADR-037 clarification), which is why a self-reported record is never gated.

See ADR-039 for the measured family constants (census, seed-dependence, R4 anchors) and ADR-036 for the certificate.

matsim — the first agent-based, stochastic-track EDOC row¶