{ "cells": [ { "cell_type": "markdown", "id": "matsim-00", "metadata": {}, "source": [ "# `matsim` \u2014 the first agent-based, stochastic-track EDOC row\n", "\n", "**What.** `matsim` wraps the MATSim 2025.0 co-evolutionary agent-based simulator (Horni et al.\n", "2016) \u2014 QSim queue dynamics plus route/selection-only replanning (`ReRoute` + `ChangeExpBeta`) \u2014\n", "as the second **EDOC-1** observational row ([ADR-036](../../docs/design/adr-036-external-dynamic-observational-certificate.md) /\n", "[ADR-039](../../docs/design/adr-039-matsim.md)). Like `sumo-duaiterate` there is **no declared\n", "cost law**: *the engine is the instance*, and the certifier re-derives every scored number by\n", "re-running the pinned engine in zero-replanning replay (`lastIteration = firstIteration`) on the\n", "model's emitted plans (G1). The engine is Java-only \u2014 no pip extra \u2014 addressed via\n", "`TABENCH_MATSIM_HOME` + `TABENCH_JAVA_HOME` (pinned `matsim-2025.0` on Temurin 21.0.11+10).\n", "\n", "**Score.** MATSim is genuinely stochastic (`global.randomSeed` moves the converged state), so\n", "this is the benchmark's first **stochastic-track** external row (ADR-036 R5): the score is the\n", "**mean frozen-field best-response gap `RG_D1`** over a pinned 5-seed macrorep list, with a\n", "percentile-bootstrap CI \u2014 a door-to-door quantity on its own scale, **not** the Wardrop\n", "`relative_gap` of the static rows, so it lives on a separate leaderboard table (R8\n", "non-comparability). Single-seed readouts are structurally impossible: `certify_row` is the row's\n", "only score entry point.\n" ] }, { "cell_type": "markdown", "id": "matsim-01", "metadata": {}, "source": [ "## How this notebook is graded\n", "\n", "**A notebook never claims a number it does not compute in that cell.** Every quantity below \u2014\n", "the per-seed certified `RG_D1`, the row mean and its bootstrap CI, the negative-control\n", "separation \u2014 is recomputed here by the same substrate the benchmark uses (`EdocEvaluator` per\n", "macrorep, `tabench.edoc.macrorep` for the row). The model emits artifacts; the certifier,\n", "model-blind, re-runs the pinned replay per seed and recomputes the score (a doctored experienced\n", "record diverges from the replay and is censored).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "matsim-02", "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "# Setup. `matsim` is a JAVA-ONLY engine behind NO pip extra (adr-039): the toolchain is\n", "# addressed via TABENCH_MATSIM_HOME (the matsim-2025.0 release layout) and TABENCH_JAVA_HOME\n", "# (the pinned Temurin 21.0.11+10 JDK). This guard raises politely when the toolchain is\n", "# unaddressed rather than failing with a G0 error deep in a later cell.\n", "from tabench.models.adapters._matsim_io import matsim_available\n", "\n", "if not matsim_available():\n", " raise RuntimeError(\n", " \"this tutorial needs the pinned MATSim toolchain: set TABENCH_MATSIM_HOME to the \"\n", " \"matsim-2025.0 release dir and TABENCH_JAVA_HOME to the Temurin 21.0.11+10 JDK \"\n", " \"(see docs/design/adr-039-matsim.md for the exact download pins)\"\n", " )\n", "\n", "import matplotlib.pyplot as plt\n", "\n", "from tabench.models.adapters.matsim_edoc import (\n", " certify_row,\n", " matsim_reference_scenario,\n", " negative_control_separation,\n", ")" ] }, { "cell_type": "markdown", "id": "matsim-03", "metadata": {}, "source": [ "## The instance\n", "\n", "The pinned reference is a **diamond** behind single entry/exit plumbing edges (MATSim activities\n", "sit *on* links): route A (`a1`,`a2`) is free-flow-shorter, and the **1-lane capacity drop on\n", "`a2`** is where MATSim's outflow queue physically sits \u2014 the cost signal lands exactly on the\n", "route-distinguishing edge. Engine identity + version (jar md5 + JDK major), the **full-JDK pin**,\n", "the `numberOfThreads=1` determinism pins, the pinned seed **list** (R5), \u0394, and every\n", "floor/deadline constant are **all inside the content hash** (P2) \u2014 the engine is part of the\n", "instance.\n" ] }, { "cell_type": "code", "execution_count": null, "id": "matsim-04", "metadata": {}, "outputs": [], "source": [ "scenario = matsim_reference_scenario()\n", "\n", "print(f\"instance : {scenario.name}\")\n", "print(f\"content hash : {scenario.content_hash()}\")\n", "print(f\"engine (pinned): {scenario.engine} {scenario.engine_version}\")\n", "print(f\"seed list (R5) : {scenario.seed_list} (P8 macroreps; a model never picks its seed)\")\n", "print(f\"network : {scenario.n_edges} edges, {scenario.n_agents} agents (all O0 -> D2)\")\n", "print(f\"lanes : {scenario.lanes_of()} (a2 is the 1-lane drop, 600 veh/h)\")\n", "print(f\"declared : separation >= {scenario.separation_factor:g}x, \"\n", " f\"floor {scenario.floor_seconds:g}s, Delta = {scenario.dt:g}s x {scenario.n_intervals}\")" ] }, { "cell_type": "markdown", "id": "matsim-05", "metadata": {}, "source": [ "## The negative control \u2014 mean-vs-mean over the SAME pinned seeds\n", "\n", "`RG_D1` must *separate* a bad assignment from a good one **as a mean**, not per lucky seed\n", "(ADR-039 ruling: single-seed anchors would violate R5's single-readout ban). The control is the\n", "engine-routed free-flow **AON** state (`iterations=0`, deterministic); the converged state\n", "co-evolves for 10 iterations under each pinned seed. The gate compares **floor-displayed**\n", "values, so a topology whose anchors both sit at `RG_D1 = 0` (the shared-edge drop, which\n", "self-certifies) separates 1.0x and is **refused at construction** \u2014 this cell also\n", "separation-vets the family, which `certify_row` requires. (~2 min: 2 states x 5 macroreps, each\n", "one JVM co-evolution + two JVM replays.)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "matsim-06", "metadata": {}, "outputs": [], "source": [ "anchors = negative_control_separation(scenario, wall_seconds=900.0)\n", "\n", "print(f\"AON control : mean RG_D1 {anchors['control_rg_d1']:.5f} (deterministic at it.0)\")\n", "print(f\"converged : mean RG_D1 {anchors['converged_rg_d1']:.5f}\")\n", "print(f\"separation : {anchors['separation']:.2f}x (displayed values; \"\n", " f\"declared >= {anchors['separation_factor']:g}x)\")" ] }, { "cell_type": "markdown", "id": "matsim-07", "metadata": {}, "source": [ "## Certify the row \u2014 P8 macroreps + bootstrap CI (R5)\n", "\n", "Each pinned seed gets its **own** emission (a fresh co-evolution under that seed) and its own\n", "full certificate: G0 pins (engine + full JDK), G1 replay fidelity **twice** (the determinism\n", "double on the tie-sorted canonical hash), G2 demand bijection with exact departures, G3\n", "delivery census with insertion wait in every cost, G4 conservation, the resolution-floor gate,\n", "and `RG_D1` against the certifier-owned time-dependent shortest path. The row score is the\n", "**mean** with a percentile-bootstrap CI on the reserved `SOURCE_BOOTSTRAP` stream \u2014 and ANY\n", "censored macrorep censors the whole row (a subset mean would re-open seed shopping).\n" ] }, { "cell_type": "code", "execution_count": null, "id": "matsim-08", "metadata": {}, "outputs": [], "source": [ "row = certify_row(scenario, iterations=10, wall_seconds=900.0)\n", "m = row.metrics\n", "\n", "print(f\"feasible : {m['feasible']:.0f} ({m['n_seeds']:.0f} macroreps; \"\n", " \"ANY censored seed censors the row)\")\n", "for seed in scenario.seed_list:\n", " ps = row.per_seed[seed]\n", " print(f\" seed {seed:>6}: RG_D1 {ps['rg_d1']:.5f} delta {ps['delta']:.2f}s \"\n", " f\"max backlog {ps['max_backlog']:.0f}s\")\n", "print(f\"row score : mean RG_D1 {m['rg_d1_mean']:.5f} \"\n", " f\"{m['ci_level']:.0%} CI [{m['rg_d1_ci_lo']:.5f}, {m['rg_d1_ci_hi']:.5f}]\")\n", "print(f\"resolution : floor_gap {m['floor_gap']:.4f} -> sub_floor={m['sub_floor']:.0f}\")" ] }, { "cell_type": "markdown", "id": "matsim-09", "metadata": {}, "source": [ "## Visualize\n", "\n", "One figure, every mark a number certified above: the five per-seed gaps, the row mean with its\n", "bootstrap CI band, the resolution floor, and the AON control mean. The spread of the dots *is*\n", "the seed lottery R5 exists to average over \u2014 which is why a single-seed readout is never the\n", "score. (`tabench.viz` is a road-network flow visualizer; `RG_D1` is a scalar gap on its own\n", "leaderboard scale, so this notebook plots the certified quantities directly \u2014 a house chart, not\n", "`tabench.viz` \u2014 the same rule the bottleneck/newell rows follow, adr-035.)\n" ] }, { "cell_type": "code", "execution_count": null, "id": "matsim-10", "metadata": {}, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(6.6, 3.4))\n", "seeds = list(scenario.seed_list)\n", "per = [row.per_seed[s][\"rg_d1\"] for s in seeds]\n", "ax.scatter(range(len(seeds)), per, color=\"#4c72b0\", zorder=3, label=\"per-seed RG_D1\")\n", "ax.axhline(m[\"rg_d1_mean\"], color=\"#4c72b0\", lw=1.5, label=f\"mean {m['rg_d1_mean']:.3f}\")\n", "ax.axhspan(m[\"rg_d1_ci_lo\"], m[\"rg_d1_ci_hi\"], color=\"#4c72b0\", alpha=0.15,\n", " label=f\"{m['ci_level']:.0%} bootstrap CI\")\n", "ax.axhline(m[\"floor_gap\"], color=\"#9a9a9a\", ls=\"--\", lw=1.2,\n", " label=f\"resolution floor {m['floor_gap']:.3f}\")\n", "ax.axhline(anchors[\"control_rg_d1\"], color=\"#c44e52\", lw=1.5,\n", " label=f\"AON control {anchors['control_rg_d1']:.3f}\")\n", "ax.set_xticks(range(len(seeds)))\n", "ax.set_xticklabels([str(s) for s in seeds])\n", "ax.set_xlabel(\"pinned macrorep seed (R5)\")\n", "ax.set_ylabel(\"RG_D1 (frozen-field BR gap)\")\n", "ax.set_title(f\"{scenario.name}: {anchors['separation']:.1f}x separation, \"\n", " \"mean +/- CI over 5 seeds\")\n", "ax.legend(fontsize=8, loc=\"center right\")\n", "fig.tight_layout()\n", "fig" ] }, { "cell_type": "markdown", "id": "matsim-11", "metadata": {}, "source": [ "## Takeaways & pointers\n", "\n", "- **Stochastic track, structurally.** The seed list is a hashed instance field; per-seed\n", " instances are derived by the harness; the score exists only as the macrorep mean + CI\n", " (`tabench.edoc.macrorep`). A model never chooses its seed, and one censored seed censors the\n", " row.\n", "- **The corrected R10 record.** The same-timestamp event-tie permutation is a **multithreading**\n", " artifact (measured at `numberOfThreads=8`), not a replay effect \u2014 so threads=1 is pinned in\n", " `global`/`qsim`/`eventsManager` (hashed), the G1 hash is the tie-sorted canonical stream, and\n", " raw-byte replay identity at threads=1 is a bonus check, never the gate.\n", "- **The replay map is seed-independent** (measured: replaying the same plans under a different\n", " pinned seed is canonically identical). Reusing one seed's artifacts across macroreps therefore\n", " collapses to *legal* optimization \u2014 the mean is still harness-computed per seed against\n", " seed-dependent emissions.\n", "- **`RG_D1` is not Wardrop `relative_gap`.** A separate leaderboard table (R8); never compared\n", " to the static rows.\n", "- **Provenance vs score.** MATSim's scores/`scorestats` and its internal 15-min replanning field\n", " are provenance only \u2014 `X` and the frozen field are defined by the pinned replay map (the\n", " ADR-037 clarification), which is why a self-reported record is never gated.\n", "\n", "See [ADR-039](../../docs/design/adr-039-matsim.md) for the measured family constants (census,\n", "seed-dependence, R4 anchors) and [ADR-036](../../docs/design/adr-036-external-dynamic-observational-certificate.md)\n", "for the certificate.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "name": "python" }, "tabench": { "covers": [], "requires_extra": "matsim", "track": "external", "unit": "matsim" } }, "nbformat": 4, "nbformat_minor": 5 }