profiles — SimOpt-style progress curves and solvability profiles

What. tabench.experiments.profiles is the diagnostics half of the SimOpt design (Eckman, Henderson & Shashaani 2023, [eckman2023simopt], docs/REFERENCES.md): progress curves, α-solve-time cdf/quantile solvability profiles, and Moré-Wild data profiles, all as pure post-hoc arithmetic over already-certified rows — no solver, certifier, or the runner changes. See docs/design/adr-032-simopt-profiles.md for the full derivation, the ten disclosed deviations from a literal SimOpt port (D1-D10), and every closed-form anchor.

Why it is in the benchmark. It redeems the last unshipped P5/P6 promise (docs/ARCHITECTURE.md: progress curves and solvability profiles for the deterministic/stochastic tracks). Because it reads certified CSV rows rather than touching a solver, there is no new trust surface — correctness is pinned by closed-form hand-derivations, not a new certifier.

Scope. Running a real grid on Braess (mirroring demos/demo_profiles.py), the certified α-solve-time closed-form anchor, cdf-solvability and Moré-Wild data profiles, the honest censoring of a black-box surrogate, and TWO sharp API edges worth knowing before you reach for this module yourself: load_run takes the literal {stem}.csv path, and write_profiles needs protocol=/provenance= — neither is optional or defaulted.

How this notebook is graded

A notebook never claims a number it does not compute in that cell. The closed-form α-solve-time anchor from adr-032 ({msa: 5, fw: 24, bfw: 4}) is asserted against a REAL grid run in this notebook, not quoted from the ADR. Profiles add no new certificate — every underlying metric was already certified by the P1 Evaluator inside run_experiment (README, Certified, not self-reported).

# Setup. `experiments.profiles` is numpy/scipy core -- no optional extra, so no
# guard cell. The inline backend is Agg-based: figures render headlessly into
# the notebook, so CI can execute tutorials without a display. NEVER
# matplotlib.use("Agg") in-kernel -- it silently suppresses inline capture.
%matplotlib inline
import tempfile
from pathlib import Path

import numpy as np

from tabench import (
    AllOrNothingModel,
    BiconjugateFrankWolfeModel,
    Budget,
    CallableModel,
    ConjugateFrankWolfeModel,
    FrankWolfeModel,
    MSAModel,
    braess_scenario,
    run_experiment,
)
from tabench.experiments.profiles import (
    Run,
    cdf_solvability,
    data_profile,
    load_run,
    progress_curves,
    read_profiles,
    run_provenance,
    solve_times,
    write_profiles,
)

ALPHA = 1e-4  # certified-gap solve target (Boyce et al. 2004 convergence target)
TAU = 1e-3  # Moré-Wild convergence-test level

Running a real grid, to disk

A five-solver Braess grid plus a toy-surrogate black box (a CallableModel that emits a noisy guess with no shortest-path work at all, sp_calls=0) — the honest-censoring story below needs a model that FAILS the audit. run_experiment(..., out_dir=...) writes the certified {stem}.csv + {stem}.manifest.json pair; the stem is auto-generated from the scenario hash + model names + budget + seed, so distinct runs can never silently overwrite each other — and, notably, so a caller can never choose the exact filename up front (the first sharp edge, made concrete below).

scenario = braess_scenario()


def naive_surrogate(s, rng):
    base = s.demand.total / 2.0
    return np.abs(base + rng.normal(0.0, 0.5, s.network.n_links))


models = [
    AllOrNothingModel(),
    MSAModel(),
    FrankWolfeModel(),
    ConjugateFrankWolfeModel(),
    BiconjugateFrankWolfeModel(),
    CallableModel(fn=naive_surrogate, name="toy-surrogate", seedable=True),
]

out_dir = Path(tempfile.mkdtemp(prefix="tabench-profiles-"))
result = run_experiment(scenario, models, Budget(iterations=50), seed=0, out_dir=out_dir)
csv_path = next(out_dir.glob("*.csv"))
print(f"scenario      : {scenario.name}  (hash {scenario.content_hash()[:16]}…)")
print(f"models        : {[m.name for m in models]}")
print(f"written stem  : {csv_path.stem}")

scenario      : braess  (hash cf00f411cdccec88…)
models        : ['aon', 'msa', 'fw', 'cfw', 'bfw', 'toy-surrogate']
written stem  : braess-cf00f411_aon-msa-fw-cfw-bfw-toy-surrogate_it50_seed-0

The closed-form anchor (adr-032): α-solve times {msa: 5, fw: 24, bfw: 4}

progress_curves turns certified rows into per-(model, macrorep) step curves on the certified metric (relative_gap here, the manifest’s default); solve_times reads the first work coordinate where the curve strictly crosses below α — SimOpt’s strict-< crossing rule (D3). All three share the AON start; msa’s crossing at sp_calls=5 is a genuine first crossing even though its trace ends unconverged — first-crossing, not sustained convergence.

curves = progress_curves(result, axis="sp_calls")  # accepts the in-memory ExperimentResult directly
times = {model: t for (model, _macrorep), t in solve_times(curves, ALPHA).items()}
for model in sorted(times, key=lambda m: times[m]):
    t = times[model]
    print(f"  {model:<14}{'inf' if t == float('inf') else int(t):>6}")

assert times["msa"] == 5
assert times["fw"] == 24
assert times["bfw"] == 4
assert times["toy-surrogate"] == float("inf")  # censored: never crosses (see below)

  cfw                4
  bfw                4
  msa                5
  fw                24
  aon              inf
  toy-surrogate    inf

cdf-solvability and Moré-Wild data profiles

cdf_solvability is the fraction of the run solved to α vs normalized budget fraction; data_profile is Moré & Wild’s d_s(κ), the fraction solved within κ all-or-nothing passes (κ = sp_calls / n_origins; Braess has one origin, so κ = raw sp_calls here). Both keep censored problems IN the denominator (D4) — a black box that never solves never inflates the profile by disappearing from it.

run = Run.from_result(result)
demand = scenario.demand
n_origins = int((demand.matrix.sum(axis=1) > 0).sum())
assert n_origins == 1  # Braess: a single origin

cdf = cdf_solvability(run, ALPHA, axis="sp_calls")
data = data_profile(run, tau=TAU, axis="sp_calls", work_unit=float(n_origins))

print("cdf-solvability at budget fraction 1.0 (terminal):")
for model, curve in sorted(cdf.items(), key=lambda kv: -kv[1].lookup(1.0)):
    print(f"  {model:<14}{curve.lookup(1.0):.3f}")

print("\ndata-profile fraction solved at κ=25:")
for model, curve in sorted(data.items(), key=lambda kv: -kv[1].lookup(25.0)):
    print(f"  {model:<14}{curve.lookup(25.0):.3f}")

assert cdf["bfw"].lookup(1.0) == 1.0
assert cdf["toy-surrogate"].lookup(1.0) == 0.0  # censored -> never in the numerator

cdf-solvability at budget fraction 1.0 (terminal):
  msa           1.000
  fw            1.000
  cfw           1.000
  bfw           1.000
  aon           0.000
  toy-surrogate 0.000

data-profile fraction solved at κ=25:
  msa           1.000
  fw            1.000
  cfw           1.000
  bfw           1.000
  aon           0.000
  toy-surrogate 0.000

Two sharp API edges (this is why they are documented in-cell)

Both refuse LOUDLY rather than doing something surprising:

1. load_run takes the literal {stem}.csv path, not a stem or a directory. Pass the run’s directory, or the stem without .csv, and it raises ValueError immediately — it never guesses.

2. write_profiles(out_path, profiles, protocol, provenance) has no defaults for protocol/provenance. They are not optional metadata: the artifact’s whole honesty story (adr-032 D8) is that a profile without its protocol constants and provenance is unauditable, so Python’s own TypeError enforces it before a single byte is written.

try:
    load_run(out_dir)  # a directory, not the {stem}.csv file
    raise AssertionError("expected load_run(directory) to raise")
except ValueError as exc:
    print(f"load_run(directory) refused : {str(exc).replace(out_dir.name, 'tabench-profiles-<tmp>')}")

try:
    load_run(str(csv_path)[: -len(".csv")])  # the stem, without .csv
    raise AssertionError("expected load_run(stem) to raise")
except ValueError as exc:
    print(f"load_run(stem) refused      : {exc}")

try:
    write_profiles(out_dir / "profiles.json", {"cdf_solvability": cdf})  # missing protocol/provenance
    raise AssertionError("expected write_profiles(...) without protocol/provenance to raise")
except TypeError as exc:
    print(f"write_profiles(no protocol/provenance) refused : {exc}")

# The CORRECT usage of both, round-tripped through the certified artifact.
# (rows differ in TYPE -- csv.DictReader yields strings, the in-memory
# ExperimentResult yields floats; _to_float collapses both, so it is the
# CURVES that round-trip identically, not the raw row dicts -- adr-032 D8.)
run_from_disk = load_run(csv_path)
assert progress_curves(run_from_disk, axis="sp_calls") == curves

protocol = {
    "metric": "relative_gap", "axis": "sp_calls", "alpha": ALPHA, "tau": TAU,
    "crossing": "strict-<", "censoring": "in-denominator", "aon_work_unit": n_origins,
}
artifact_path = out_dir / "profiles.json"
write_profiles(
    artifact_path, {"cdf_solvability": cdf, "data_profile": data},
    protocol, run_provenance(run_from_disk),
)
doc, profiles_back = read_profiles(artifact_path)
print(f"\nartifact written and round-tripped: schema={doc['schema']!r}, "
      f"kinds={sorted(profiles_back)}")
assert doc["protocol"] == protocol
assert profiles_back["cdf_solvability"]["bfw"].lookup(1.0) == 1.0

load_run(directory) refused : expected a .csv run file, got 'tabench-profiles-<tmp>'
load_run(stem) refused      : expected a .csv run file, got 'braess-cf00f411_aon-msa-fw-cfw-bfw-toy-surrogate_it50_seed-0'
write_profiles(no protocol/provenance) refused : write_profiles() missing 2 required positional arguments: 'protocol' and 'provenance'

artifact written and round-tripped: schema='tabench-profiles-v1', kinds=['cdf_solvability', 'data_profile']

Visualize

Profile curves (fraction-solved vs a normalized budget fraction / work units) are not road link flows, so the viz rule (adr-035) calls for plain matplotlib here, not tabench.viz — reasoned explicitly, not defaulted.

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(10, 3.2))
for ax, (title, profile, xlabel) in zip(
    axes,
    [
        ("cdf-solvability", cdf, "normalized budget fraction"),
        ("Moré-Wild data profile", data, f"κ (sp_calls / {n_origins} AON-pass)"),
    ],
):
    for model, curve in profile.items():
        ax.step(curve.x, curve.y, where="post", label=model)
    ax.set_xlabel(xlabel)
    ax.set_ylabel("fraction solved")
    ax.set_ylim(-0.02, 1.02)
    ax.set_title(title)
    ax.legend(fontsize=6)
fig.tight_layout()
display(fig)
plt.close(fig)

Takeaways & pointers

• No new trust surface. Every curve here is a deterministic pure function of rows the P1 Evaluator already certified inside run_experiment — profiles add reporting, not scoring.

• Censoring stays in the denominator. toy-surrogate never crosses α (+inf, cdf=0.0 at budget fraction 1.0) but it never vanishes from a profile the way a solver that silently skipped a hard scenario would (D10, D4) — garbage is a first-class row, never dropped.

• Two sharp edges, now known. load_run wants the exact .csv path; write_profiles wants its protocol and provenance every time — both refuse loudly rather than guessing.

• Where next. The ten disclosed SimOpt deviations (D1-D10), the β-quantile parity fix, and the Moré-Wild work-unit convention in docs/design/adr-032-simopt-profiles.md; a CLI-free runnable script at demos/demo_profiles.py.