{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "intro",
   "metadata": {},
   "source": [
    "# `bo4mob-estimation` — BO4Mob held-out-count OD estimation (D2 observational)\n",
    "\n",
    "**What.** The T2 estimation track's first *observational* certificate (`adr-041`).\n",
    "BO4Mob (Ryu, Kwon, Choi, Deshwal, Kang & Osorio 2025, `arXiv:2510.18824`) poses a\n",
    "San Jose freeway network as a black-box OD-estimation problem: choose a continuous\n",
    "OD vector over a fixed set of active `(fromTaz, toTaz)` pairs to fit **real Caltrans\n",
    "PeMS link counts** under a mesoscopic SUMO run. There is **no** declared BPR network,\n",
    "**no** true OD, and **no** equilibrium — so this is a NEW T2 sibling family (the\n",
    "`adr-023` pattern), scored by re-running the pinned `eclipse-sumo` engine, not a\n",
    "`bfw` assignment.\n",
    "\n",
    "**Dual-benchmark honesty (the central constraint).** BO4Mob is a UMN Choi Lab\n",
    "benchmark; TABenchmark hosts its instances as scenarios/data only (`adr-034`), not\n",
    "as validation of TABench methods, and does not reproduce BO4Mob's published numbers.\n",
    "This estimation row additionally does **not** reproduce BO4Mob's own SPSA/BO\n",
    "leaderboard rankings, and its D2 held-out NRMSE is **not comparable** to the\n",
    "static/dynamic T2 `heldout_count_rmse` scale (it is BO4Mob's own n-scaled NRMSE, a\n",
    "same-lab scenario)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "grading",
   "metadata": {},
   "source": [
    "## How this notebook is graded\n",
    "\n",
    "**A notebook never claims a number it does not compute in that cell.** Every scored\n",
    "quantity below — `od_feasible`, `obs_nrmse`, `heldout_nrmse` — comes from the SAME\n",
    "`Bo4MobODCertifier` the harness uses: it re-runs the pinned od2trips + mesoscopic\n",
    "`sumo` pipeline ONCE on the emitted OD and compares the resulting link counts to\n",
    "real PeMS panels. **Equilibrium is never claimed.**\n",
    "\n",
    "The **ranking column** is `heldout_nrmse` — the MEAN of BO4Mob's count NRMSE over a\n",
    "temporally-disjoint, **same-hour** set of held-out DATES (the anchor hour `06-07`,\n",
    "dates `221009`–`221021`). The estimator emits ONE fixed OD from the TRAIN anchor\n",
    "(`221008`); the certifier scores it against each held-out date's real counts. The\n",
    "held-out CSV bytes and dates never enter the task — only their `heldout_digest`\n",
    "does (P7). The engine version is pinned (`assert_engine_pin` RAISES on a mismatch),\n",
    "so a drifted box fails loudly rather than silently scoring.\n",
    "\n",
    "This notebook needs the optional `sumo` extra (`pip install 'tabench[sumo]'`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "setup",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-21T13:47:58.152710Z",
     "iopub.status.busy": "2026-07-21T13:47:58.152364Z",
     "iopub.status.idle": "2026-07-21T13:48:00.115679Z",
     "shell.execute_reply": "2026-07-21T13:48:00.114323Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1ramp: 3 OD pairs, 3 anchor sensors\n"
     ]
    }
   ],
   "source": [
    "# Headless inline backend (Agg): figures render into the notebook so CI can execute\n",
    "# tutorials without a display. Never call matplotlib.use() after pyplot is imported.\n",
    "import matplotlib\n",
    "matplotlib.use(\"Agg\")\n",
    "import json\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "from tabench.core.budget import Budget\n",
    "from tabench.data.bo4mob import (\n",
    "    BO4MOB_HELDOUT_DATES, BO4MOB_HELDOUT_HOUR, BO4MOB_REGISTRY,\n",
    "    bo4mob_pairs, bo4mob_prior_vector, fetch_bo4mob, fetch_bo4mob_heldout,\n",
    ")\n",
    "from tabench.estimation.bo4mob_base import Bo4MobPriorBaseline\n",
    "from tabench.experiments.runner import run_bo4mob_estimation_experiment\n",
    "from tabench.metrics.estimation_bo4mob import Bo4MobODCertifier\n",
    "\n",
    "INSTANCE = \"1ramp\"  # the smallest instance: ~0.4 s meso, the CI liveness case\n",
    "spec = BO4MOB_REGISTRY[INSTANCE]\n",
    "print(f\"{INSTANCE}: {spec.n_od} OD pairs, {spec.n_sensors_anchor} anchor sensors\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "split",
   "metadata": {},
   "source": [
    "## The held-out split (framing b)\n",
    "\n",
    "The estimator emits **one** fixed OD vector; \"holding out a date\" only withholds\n",
    "which ground-truth CSVs are used for scoring, never which OD to submit. This closes\n",
    "a free-riding hole that per-date OD emission would open (copy the nearest train\n",
    "date's OD) and costs **one** meso run per certify regardless of how many held-out\n",
    "dates are scored. PeMS counts vary enormously across the day, so the hour window is\n",
    "held FIXED — held-out probes same-hour, different-DATE generalisation only."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "run",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-21T13:48:00.121011Z",
     "iopub.status.busy": "2026-07-21T13:48:00.120701Z",
     "iopub.status.idle": "2026-07-21T13:48:00.578357Z",
     "shell.execute_reply": "2026-07-21T13:48:00.576985Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "estimator      : bo4mob-prior\n",
      "od_feasible    : 1.0\n",
      "obs_nrmse      : 2.432471   (in-sample, TRAIN anchor 221008)\n",
      "heldout_nrmse  : 1.697988   <- ranking column (mean over 13 held-out dates)\n",
      "  min / max    : 0.563911 / 3.297711\n"
     ]
    }
   ],
   "source": [
    "# Run the prior baseline (emit BO4Mob's example OD unchanged) and certify it.\n",
    "result = run_bo4mob_estimation_experiment(\n",
    "    INSTANCE, [Bo4MobPriorBaseline()], Budget(sp_calls=1), seed=0,\n",
    ")\n",
    "row = result.rows[0]\n",
    "print(f\"estimator      : {row['estimator']}\")\n",
    "print(f\"od_feasible    : {row['od_feasible']}\")\n",
    "print(f\"obs_nrmse      : {row['obs_nrmse']:.6f}   (in-sample, TRAIN anchor 221008)\")\n",
    "print(f\"heldout_nrmse  : {row['heldout_nrmse']:.6f}   <- ranking column (mean over \"\n",
    "      f\"{int(row['n_heldout_dates'])} held-out dates)\")\n",
    "print(f\"  min / max    : {row['heldout_nrmse_min']:.6f} / {row['heldout_nrmse_max']:.6f}\")\n",
    "# The prior OD is BO4Mob's low-demand anchor day, so its held-out MEAN (over higher-\n",
    "# demand days) differs from its in-sample fit -- an honest generalisation signal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laundering",
   "metadata": {},
   "source": [
    "## The anti-laundering control: the OD-window fill\n",
    "\n",
    "The one real laundering vector, found and fixed once in stage 1 (`adr-034`\n",
    "Decision 3), is the OD-window fill: the emitted vector must fill `od.xml` with the\n",
    "interval `end` rewritten to the instance's `od_end_time`. On `1ramp`\n",
    "(`od_end_time=3300`) keeping the template's `3600` releases ~5% of demand past the\n",
    "OD window and silently biases the count NRMSE. `fill_od_from_vector` inherits this\n",
    "rewrite, so the prior baseline's certified `obs_nrmse` reproduces the stage-1\n",
    "faithful pipeline exactly (`2.432471…`); the unfixed value would be `2.314704…`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "perdate",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2026-07-21T13:48:00.582733Z",
     "iopub.status.busy": "2026-07-21T13:48:00.582342Z",
     "iopub.status.idle": "2026-07-21T13:48:01.115678Z",
     "shell.execute_reply": "2026-07-21T13:48:01.114547Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "held-out dates: 221009, 221010, 221011, 221012, 221013, 221014, 221015, 221016, 221017, 221018, 221019, 221020, 221021\n",
      "mean held-out NRMSE = 1.697988\n"
     ]
    }
   ],
   "source": [
    "# The per-held-out-date NRMSE (one pipeline run, K cheap comparisons). This is the\n",
    "# spread the ranking `heldout_nrmse` averages over -- never a black box.\n",
    "paths = fetch_bo4mob(spec)\n",
    "heldout = fetch_bo4mob_heldout(INSTANCE)\n",
    "cfg = json.loads(paths[\"config\"].read_text())\n",
    "pairs = bo4mob_pairs(paths[\"od\"])\n",
    "prior = bo4mob_prior_vector(paths[\"od\"], paths[\"single_od\"])\n",
    "cert = Bo4MobODCertifier(\n",
    "    instance_key=INSTANCE, pairs=pairs, train_sensor=paths[\"sensor\"], heldout_sensors=heldout,\n",
    "    paths=paths, od_end_time=int(cfg[\"od_end_time\"]), sim_end_time=float(cfg[\"sim_end_time\"]),\n",
    "    sensor_start_time=float(cfg[\"sensor_start_time\"]), sensor_end_time=float(cfg[\"sensor_end_time\"]),\n",
    "    engine_version=\"1.27.1\", certificate={\"seed\": 0},\n",
    ")\n",
    "per_date = cert.per_date_heldout_nrmse(prior)\n",
    "dates = list(per_date)\n",
    "vals = [per_date[d] for d in dates]\n",
    "fig, ax = plt.subplots(figsize=(8, 3))\n",
    "ax.bar(range(len(dates)), vals, color=\"#4477aa\")\n",
    "ax.axhline(np.mean(vals), color=\"#cc3311\", ls=\"--\", label=f\"heldout_nrmse = {np.mean(vals):.3f}\")\n",
    "ax.set_xticks(range(len(dates)))\n",
    "ax.set_xticklabels([d[-4:] for d in dates], rotation=45, ha=\"right\")\n",
    "ax.set_ylabel(\"count NRMSE\")\n",
    "ax.set_title(f\"{INSTANCE}: prior-OD NRMSE per held-out date (hour {BO4MOB_HELDOUT_HOUR})\")\n",
    "ax.legend()\n",
    "plt.tight_layout()\n",
    "print(\"held-out dates:\", \", \".join(BO4MOB_HELDOUT_DATES))\n",
    "print(f\"mean held-out NRMSE = {np.mean(vals):.6f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "close",
   "metadata": {},
   "source": [
    "## Reading the row\n",
    "\n",
    "`heldout_nrmse` ranks the family: a prior OD already fitting the held-out days is\n",
    "unbeatable; one far off certifies a terrible (honest) held-out count-fit. A future\n",
    "black-box estimator (an SPSA over this pipeline) would run the engine as its own\n",
    "inner oracle and try to lower `heldout_nrmse` — improving out-of-sample count fit,\n",
    "never memorising a single day.\n",
    "\n",
    "**Honesty, restated.** BO4Mob is a UMN Choi Lab benchmark; TABenchmark hosts its\n",
    "instances as scenarios/data only (`adr-034`), not as validation of TABench methods,\n",
    "and does not reproduce BO4Mob's published numbers — including BO4Mob's own SPSA/BO\n",
    "leaderboard rankings. Equilibrium is never claimed, and this D2 held-out NRMSE is\n",
    "not comparable to the static/dynamic T2 `heldout_count_rmse` scale."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "tabench": {
   "covers": [],
   "requires_extra": "sumo",
   "track": "estimation",
   "unit": "bo4mob-estimation"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}