ADR-032: simopt-profiles — SimOpt-style progress curves and solvability profiles

Status: accepted (implemented) Date: 2026-07-15 Deciders: non-solver roadmap — redeeming the P5/P6 progress-curve / solvability-profile promise File: docs/design/adr-032-simopt-profiles.md

Context

SimOpt (Eckman, Henderson & Shashaani 2023, SimOpt: A testbed for simulation-optimization experiments, INFORMS J. on Computing 35(2):495-508, eckman2023simopt, tier-1 metric/protocol in docs/references.json) is one of the three named design sources in docs/ARCHITECTURE.md. Its diagnostics — progress curves, α-solve-time solvability profiles, Moré-Wild data profiles — are defined not in the testbed paper but in its companion, Eckman, Henderson & Shashaani (2023), “Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms”, INFORMS J. on Computing 35(2):350-367, DOI 10.1287/ijoc.2022.1261 and, for data profiles, Moré & Wild (2009), “Benchmarking Derivative-Free Optimization Algorithms”, SIAM J. Optim. 20(1):172-191, DOI 10.1137/080724083. Neither is in the verified canon (which holds only the testbed paper, eckman2023simopt); this ADR cites both freely in text — ADRs cite non-canon works — and does not grow the canon (it stays at 246 references).

The architecture already promises these deliverables in normative language: P5 (lines 98-114) says the deterministic track is “reported as progress curves and Moré-Wild-style data profiles”, the stochastic track carries “mean/quantile progress curves and α-solve-time solvability profiles”, and no-certificate models appear “as censored entries in solvability profiles”; the v0.x roadmap lists “progress-curve/solvability-profile plotting” as still open. This is the last unredeemed piece of that text.

The experiment half of SimOpt already ships and was verified live against the working tree: macroreplications with the fixed (macrorep, source, replication) stream schema (P8, core/rng.py), hardware-free budget coordinates on every checkpoint (BudgetCoords, P6), one certified CSV row per checkpoint (the post-replication-equivalent P1 certificate — deterministic metrics recomputed exactly, the probit residual certified through a pinned common Monte-Carlo sample, adr-003), row-level censoring of infeasible flows, and the terminal percentile bootstrap across macroreps (experiments/bootstrap.py). What was missing is the diagnostics half: curve objects, α-solve times, solvability/data profiles, difference profiles, functional bootstrap bands, and a profile artifact. That missing piece is pure post-hoc arithmetic over already-certified rows, so it needs no new certifier and no change to any solver, certifier, or the runner.

Decision

Ship a new module src/tabench/experiments/profiles.py — at the same altitude as experiments/bootstrap.py (aggregation over certified outputs), not in metrics/ (those are P1 certifiers over (scenario, flows); profiles consume already-certified rows). It reads either an in-memory ExperimentResult or the runner’s on-disk {stem}.csv + {stem}.manifest.json pair (load_run, which owns the typed parse and the total-budget/scenario-hash the rows lack). Curves are immutable StepCurve objects; profiles are plain data; the certified artifact is profiles.json. No file in models/, metrics/, core/, or runner.py changes; the runner keeps writing the same CSV+manifest, so the golden Braess hash cf00f411cdccec88019979e8cfbf3d8014ba590688b285a1db667315ac96762d is byte-identical (re-asserted in tests/test_profiles.py).

Ten protocol decisions are deliberate, disclosed deviations from a literal SimOpt port:

D1 — the curve y-axis is the certified ranking metric, not SimOpt’s (f-f*)/(f(x0)-f*). SimOpt normalizes because raw objectives are incomparable across problems and f is noisy; TABenchmark’s certified relative gap (UE), SUE fixed-point residual, or SO relative gap is already an absolute, scale-free, harness-computed optimality measure with a true zero. Adopting it makes α-solve mean “certified gap < α” — the repo’s existing convergence-target language (Boyce et al. 2004, Budget.target_relative_gap) — and avoids estimating f(x0)/f(x*) entirely. The metric defaults to the track’s ranking column read from the manifest: sue_fixed_point_residual when sue_theta is set; so_relative_gap only when every model in the grid is system-optimum — a single static_so model in a mixed grid must NOT flip the UE solvers beside it onto a column they do not populate (that would score their transient crossings as SO convergence); mixed grids keep relative_gap and the SO column is requested explicitly. A SimOpt-faithful Beckmann-normalized variant stays possible later (the beckmann_objective column + best-known anchors exist); it is not v1 scope.

D2 — the budget axis is a declared hardware-free coordinate (P6): default sp_calls, iterations accepted; wall_ms is allowed for descriptive progress curves but refused by every ranked profile (cdf/quantile solvability, data profiles). The α-solve time is reported in the axis’s own work units, not hidden behind a budget fraction. An axis that is 0 at every checkpoint of a model that nonetheless produced a genuine (feasible, finite-metric) checkpoint is refused (a learned model / spsa-sumo discloses sp_calls=0 while doing real work, so ranking it on sp_calls would be a degenerate curve, not a fast one) — but an already-censored model at sp_calls=0 is exempt, since it is +inf regardless of the axis. The zero-axis guard is judged against the caller’s metric, not the default: an alternate metric can make an otherwise-censored model finite (and vice versa), so the guard must see the same metric the curve will. wall_ms reaches progress_curves/solve_times for descriptive curves and raw α-solve times only — every ranked profile refuses it (P6: wall-clock is recorded, never ranked). When the manifest budget does not constrain the chosen axis (an iterations-only budget leaves sp_calls=None), the normalizer is the realized budget envelope (the maximum observed axis value). This differs from SimOpt, whose problem.factors['budget'] is a declared factor fixed up front: the realized-max fallback means adding a longer-running model to a grid rescales the normalized solvability x-axis, so cdf/quantile profiles are comparable only within a fixed grid composition — the raw-work-unit α-solve times (solve_times) are not rescaled and stay directly comparable.

D3 — strict < crossing. StepCurve.crossing_time(α) is the first knot whose value is strictly below α (SimOpt Curve.compute_crossing_time parity), +inf when never crossed. Pinned by a knot-exact test.

D4 — censoring stays in the denominator. A checkpoint with feasible=0 or a non-finite metric becomes y=+inf (censored); a never-solved run has an +inf solve time; censored entries stay in every cdf denominator so they never inflate a profile (SimOpt parity; P5’s “censored entries in solvability profiles” made literal). A black box that emits infeasible flows is thus a first-class censored row, never an error and never a leaderboard-topper.

D5 — step semantics. The last recommendation is carried to the budget end (curve_to_full_curve); before a model’s first checkpoint the curve is +inf (censored), so there is no fictitious t=0 solve. An early-converged trace (bfw reaches its target in 3 checkpoints on Braess) carries its final value forward to the realized budget end.

D6 — the data-profile work unit is one all-or-nothing pass. Moré-Wild’s d_s(κ) is the fraction of problems solved within κ work units, where one unit is n_p+1 function evaluations (a simplex gradient). The TA analog is one AON pass = n_origins shortest-path trees, so κ = sp_calls / n_origins (Braess 1, Sioux Falls 24). Because n_origins is a scenario property not carried by the certified row schema (and the runner is unchanged), it is supplied by the caller (default 1 ⇒ κ = raw work); a work_unit dict with a scenario key missing is refused, never silently defaulted to 1 (that would mix raw sp_calls with per-pass units in one profile). The convergence test is the certified metric <= τ (a gap-based MW variant; MW’s f-based test f(x) <= f_L + τ(f(x0)-f_L) needs a common x0, which non-iterative paradigms do not share — disclosed). Each (scenario, macrorep) pair is one equally-weighted problem: a deterministic model contributes a single rep per scenario while a stochastic model contributes M, so a mixed deterministic/stochastic grid weights the stochastic model’s scenarios more heavily in the pooled data profile — disclosed, not corrected (collapsing macroreps to a per-scenario summary would discard the sampling spread the stochastic track exists to show).

D7 — one-level functional bootstrap (macroreps only). SimOpt’s second resampling level draws post-replications; here the deterministic certificate has zero estimation noise and the probit certificate is a pinned protocol constant (adr-003: a fixed r_cert sample, CRN across everything in the run), so there is no post-replication noise to resample. bootstrap_curve_band extends bootstrap.py’s discipline — percentile, never parametric (P5), on the reserved SOURCE_BOOTSTRAP stream, byte-reproducible from root_seed alone (P8).

D8 — the certified artifact is profiles.json. It carries the profile curves, the protocol constants (metric, axis, α/τ/β, crossing rule, censoring rule, mesh), and full provenance (scenario hashes, manifest budget blocks, seeds, tabench version, git commit). Plots are rendering, never the artifact. A censored +inf curve value is written as the JSON string "Infinity" (not a bare Infinity token), so the artifact is strict RFC-8259 that any conformant consumer (jq, a browser) parses; the reader restores it. A NaN never reaches the file — the curve encoder refuses it and json.dumps(allow_nan=False) is the belt-and-suspenders. The P1 story: profiles are deterministic pure functions of already-certified rows, so there is no new trust surface and therefore no new certifier — correctness is pinned by the closed-form tests below, provenance by the artifact schema.

D9 — the β-quantile is SimOpt’s exact estimator, made inf-aware. The default (quantile_method="simopt") is byte-parity with SimOpt’s quantile_cross_jump / quantile_of_curves: statistics.quantiles(values, n=100)[int(β·99)] (exclusive-interpolated). Because that estimator interpolates, a window that reaches a censored +inf returns a non-finite quantile — which for a solvability jump is the flat-zero (unsolvable) curve and for a progress-curve quantile is a censored +inf; it is made inf-aware so it never poisons to NaN. Note this means {0.2, 0.5, ∞} at β=0.5 is flat-zero (the median lands in the censored tail under the exclusive interpolation), not a jump at 0.5 — the earlier draft’s “parity” claim while shipping the type-1 estimator was false and always flattered the solver. The type-1 lower inverted-cdf sorted[⌈β·n⌉−1], which never interpolates a censored +inf, is retained as an opt-in quantile_method="censoring-robust" — the honest alternative when interpolating into the censored tail is undesirable.

D10 — solvability and data profiles require the full cross design. SimOpt’s profiles assume every solver is run on every problem. A model absent from a scenario would shrink only its own per-model denominator, so a solver that skipped the hard scenarios could top the profile a solver run everywhere honestly trails. cdf_solvability / quantile_solvability / data_profile therefore refuse (naming the offending models and the scenarios they are missing from) when the per-run model sets are incongruent; a single run is trivially congruent.

Closed-form anchors

Every profile value is derivable by hand and recomputed in tests/test_profiles.py (no BLAS-sensitive assertions):

• α-solve regression (integration). Braess msa/fw/bfw under Budget(iterations=50), α=1e-4, on sp_calls: solve times {msa: 5, fw: 24, bfw: 4} (measured directly from the certified rows). All three share the AON start (iteration 1, sp_calls=2, gap 1.9118e-1, P7). msa’s crossing at sp_calls=5 is a genuine first crossing even though its trace ends unconverged (6.5e-3) — the strict-< first-crossing semantics, not sustained convergence. bfw’s 3-checkpoint trace (sp_calls 2,3,4) carries forward to the realized budget end 51 (D5).

• Strict-< at a knot (D3). On x=(1,2,3), y=(0.5,0.2,0.1): crossing_time(0.2) strict = 3 (the knot valued exactly 0.2 does not count), non-strict = 2.

• Censored denominator (D4). cdf of solve times {0.2, 0.5, ∞}: terminal value 2/3, the ∞ counted in the denominator.

• Quantile parity (D9), checked against statistics.quantiles. Under the default SimOpt estimator: {0.2, 0.5, ∞} at β=0.5 ⇒ flat-zero (the exclusive interpolation reaches the censored tail); {0.2, 0.5, ∞, ∞} at β=0.5 ⇒ flat-zero; {0.1, 0.2, 0.3, ∞} at β=0.5 ⇒ jump at 0.25; {0.2, 0.5, 0.9, ∞} at β=0.5 ⇒ jump at 0.7. The censoring-robust opt-in gives jump at 0.5 / 0.2 for the first and third — the divergence the shipped “parity” claim had hidden.

• Inf-honesty (M3). quantile_of_curves and the bootstrap band over curves that carry +inf yield +inf, never NaN; a both-censored difference is the zero curve, a one-censored difference is ±inf; a NaN reaching write_profiles raises.

• Full-cross-design refusal (D10). Two scenarios where model B is run only on the first ⇒ cdf_solvability raises naming B missing from the second (else B, tested on one easy scenario, would top the honest A tested on both).

• Overshoot censoring (M7). With a manifest sp_calls budget 10 and a crossing at sp_calls=11, the cdf and the quantile jump BOTH read it as censored (terminal 0), never one counting and the other dropping it.

• Moré-Wild staircase (D6). Two scenarios × two solvers with solve work A=(2,6), B=(4,∞), unit 1: d_A(2)=1/2, d_A(6)=1, d_B(4)=1/2, terminal d_B=1/2 (the never-solved problem stays in the denominator).

• Difference profile. A solver differenced against itself is the identically zero curve.

• Bootstrap band. Byte-deterministic in root_seed; identical macroreps give a zero-width band; the band brackets the mean curve; M < 2 refuses.

• Artifact round-trip. In-memory rows and load_run of the written CSV+manifest pair yield identical profiles; the artifact is strict RFC-8259 (censored +inf as the string "Infinity", no bare token) and carries the protocol constants and the scenario hash. A T2 estimation CSV raises a clear, named limitation.

Alternatives considered

• Put profiles in metrics/: rejected — metrics/* are P1 certifiers over (scenario, flows); profiles consume already-certified rows and add no trust surface. experiments/ (the bootstrap altitude) is the honest home.

• SimOpt-faithful Beckmann-normalized progress curves: the (f-f*)/(f(x0)-f*) normalization needs estimated f(x0)/f(x*); the certified gap is already the scale-free measure with a true zero (D1). A Beckmann variant is a clean later addition, not v1 scope.

• A two-level bootstrap resampling post-replications: the certificate has no post-replication noise to resample (D7); the single macrorep level is the whole sampling story here.

• Normalizing every solve time to a budget fraction: rejected as the primary report — α-solve times are reported in hardware-free work units (P6/D2); the budget-fraction normalization is used only inside the cross-scenario cdf mean, where different scenarios have different budgets.

• A tabench profiles <run.csv>... CLI subcommand writing profiles.json: deferred to a follow-up; the demo exercises the module end-to-end and the artifact writer is public.

• Profiling the estimation (T2/T2d) track: a named follow-up, not free. The T2 CSVs key rows on estimator (not model) and censor on od_feasible (not feasible), and rank on heldout_count_rmse — so a T2 pair needs a column mapping and od_feasible censoring, not just the shared budget-coordinate shape. Until that ships, load_run and the profile functions detect the T2 schema and raise a clear, named limitation rather than dying with KeyError('model').

Consequences

The benchmark redeems the last P5/P6 promise: certified progress curves, α-solve-time cdf/quantile solvability profiles, Moré-Wild data profiles, difference profiles, and functional bootstrap bands, all as a profiles.json artifact — with zero changes to any solver, certifier, or the runner, no new runtime dependency (numpy/scipy core; the demo’s matplotlib is import-guarded), and no external data or spend (P9 untouched). The golden Braess hash is byte-identical. Follow-ups: the tabench profiles CLI, an estimation-track demo, and the Beckmann-normalized progress-curve variant.

Adversarial review

Three independent lenses (semantics-fidelity vs the fetched SimOpt master source, closed-form hand-derivations + real-data behavior, numerics/API/ integration), each executing repros; every finding CONFIRMED and fixed with a per-finding regression (streak: 20/20 sprints with at least one material defect; 48 tests after the fixes, from 31). For pure reporting code the false-certify analog is STATISTICAL DISHONESTY, and the review found four material ways this module could flatter models:

MAJORs, all fixed + pinned: (a) — found by ALL THREE lenses — missing (model, scenario) cells silently shrank per-model denominators, so a model that skipped a hard scenario topped the profile (measured: honest 0.5 vs selective 1.0, no warning; SimOpt’s full cross design cannot express the state) → cross-scenario profiles now REFUSE incongruent model sets (D10); (b) the β-quantile estimator was NOT SimOpt’s despite explicit parity claims, diverging always in the flattering direction (type-1 lower inverted-CDF vs statistics.quantiles(n=100)[int(β·99)] exclusive-interpolated with the NaN→flat-zero censored branch — the draft’s own closed-form anchor contradicted measured SimOpt) → SimOpt-exact is now the default, pinned in-test against statistics.quantiles itself, with the censoring-robust type-1 variant as an explicit opt-in (D9); (c) censored macroreps NaN-poisoned quantile curves and bootstrap bands (np’s lerp computes inf−inf) where the honest value is +inf → inf-aware percentiles throughout, and a NaN reaching write_profiles raises; (d) the degenerate-axis guard checked the DEFAULT metric rather than the REQUESTED one, so a zero-work model topped a ranked profile on an alternate metric — the exact flattery D2 promises to refuse → the metric is threaded through the guard, both directions pinned.

MEDIUMs/MINORs, fixed: one static_so model in a mixed grid flipped the default metric for every model (now: all-static_so only); T2 estimation runs crashed with a bare KeyError under a “generalizes for free” claim (now a clear named refusal; T2 profiles are a named follow-up); cdf and quantile disagreed on crossings past the realized budget envelope (both now censor); M=1 bootstrap “confidence” bands refused; profiles.json was not RFC-8259 under censoring (+inf now the string "Infinity", allow_nan=False, schema tabench-profiles-v1); work_unit key misses, duplicate-x row-order dependence, NaN x-knots, blank axis cells, and unknown metric names all refuse loudly instead of degrading silently; the ADR’s D-decisions were corrected where the code or SimOpt source contradicted them.

Survived (highlights): the strict-< first-crossing rule verified identical to the fetched Curve.compute_crossing_time; the re-crossing honesty of first-crossing solve times demonstrated on a REAL oscillating trace (braess msa: 3.6e-9 at sp=5 → 0.104 at sp=6 → terminal 6.5e-3) and disclosed; the cdf estimator arithmetic-equivalent to SimOpt’s bisect-over-sorted-crossings; the braess anchor {msa:5, fw:24, bfw:4} reproduced through the new code path against the scoping dossier’s measurement; left-endpoint step AUC, union-mesh means, carry-to-budget-end and the +inf-before-first-checkpoint convention all as disclosed; byte-determinism of the bootstrap band in root_seed; additive-only changes to bootstrap.py and the __init__ exports; the golden Braess hash byte-identical.

Sourcing

The SimOpt diagnostics semantics were measured against the library source (github.com/simopt-admin/simopt, master: simopt/curve.py, simopt/curve_utils.py, simopt/experiment/post_normalize.py, simopt/experiment/single.py, simopt/plots/solvability_profile.py) and the two definition papers above (Diagnostic Tools, DOI 10.1287/ijoc.2022.1261; Moré & Wild 2009, DOI 10.1137/080724083). The strict-< crossing, the censored-in-denominator cdf (bisect_right(sorted_crossings, t) / n_curves), the flat-zero quantile, the union-mesh mean_of_curves, and the left-endpoint step AUC are direct ports of that source, pinned by the closed-form anchors. The eight deviations (D1-D8) are TABenchmark’s, disclosed here; no SimOpt number is reproduced (their curves are over their own simulation-optimization problems).