tabench.metrics

The certifiers. Every scored metric — relative gaps, TSTT, system-optimum and day-to-day measures, estimation NRMSE, the EDOC observational gaps — is recomputed here from the flows a model emits, so a self-reported number can never reach the leaderboard.

Metric definitions: certified gaps, flow accuracy. Single source of truth.

tabench.metrics.nrmse(link_flows, oracle)[source]

RMSE normalized by the mean oracle flow (BO4Mob convention).

Parameters:

  • link_flows (ndarray)

  • oracle (ndarray)

Return type:

float

tabench.metrics.rmse(link_flows, oracle)[source]

Root-mean-square link-flow error.

Parameters:

  • link_flows (ndarray)

  • oracle (ndarray)

Return type:

float

class tabench.metrics.Evaluator(scenario, feasibility_tol=1e-06, so_metrics=False, root_seed=0, r_cert=2000)[source]

Bases: object

Model-blind scorer for one scenario. Reuse across checkpoints.

Parameters:

  • scenario (Scenario)

  • feasibility_tol (float)

  • so_metrics (bool)

  • root_seed (int)

  • r_cert (int)

evaluate(link_flows, class_link_flows=None)[source]

Certified metrics for one emitted flow state.

Infeasible or invalid flows are censored (feasible=0, NaN gaps), never scored and never raised out of the scoring loop — a black box emitting garbage must not crash the experiment nor top a leaderboard. Only a wrong-shaped array raises, since that is a programming error in the wrapper, not a property of the solution.

class_link_flows (optional (n_classes, n_links)) is required on a multiclass scenario and ignored otherwise; a multiclass scenario with no per-class flows is censored (an aggregate flow cannot certify a per-class equilibrium).

Parameters:

  • link_flows (ndarray)

  • class_link_flows (ndarray | None)

Return type:

dict[str, float]

class tabench.metrics.DNLEvaluator(scenario, tol=1e-09, kw_tol_factor=1.0)[source]

Bases: object

Model-blind DNL certifier. Pure function of (DynamicScenario, DNLOutput).

Master tolerances (constructor-overridable): with vehicle scale V = max(1, total demand) and per-link per-step flow scale F_a = q_max_a * dt,

  • eps_count = tol * V (absolute count comparisons),

  • eps_flow_a = tol * F_a + eps_count (per-step flow comparisons).

Rationale: float64 accumulates <= K ~ 1e4 additions of O(F) numbers, a relative error ~ K * 2^-52 ~ 2e-12; the default tol = 1e-9 sits three orders above rounding and six or more below any physical violation. kw_tol_factor sets the Tier-B kw_backward_at_resolution flag threshold in units of one step of capacity flow per link.

Parameters:

  • scenario (DynamicScenario)

  • tol (float)

  • kw_tol_factor (float)

evaluate(out)[source]

Certified metrics for one emitted DNL output.

Infeasible or invalid outputs are censored (dnl_feasible = 0, NaN scored quantities, residual columns populated), never scored and never raised out of the scoring loop. Only wrong-shaped arrays raise, since that is a programming error in the wrapper, not a solution property.

Parameters:

out (DNLOutput)

Return type:

dict[str, float]

class tabench.metrics.TransitEvaluator(scenario, feasibility_tol=1e-06)[source]

Bases: object

Model-blind certifier for one transit scenario.

Parameters:

  • scenario (TransitScenario)

  • feasibility_tol (float)

class tabench.metrics.BottleneckEvaluator(scenario, tol=1e-06)[source]

Bases: object

Model-blind bottleneck certifier: pure function of (scenario, schedule).

Parameters:

  • scenario (BottleneckScenario)

  • tol (float)

class tabench.metrics.SODTAEvaluator(scenario, tol=1e-06)[source]

Bases: object

Model-blind M-N certifier: pure function of (scenario, trajectory).

Raises ValueError at construction if the scenario’s canonical LP cannot be solved (horizon too short to clear the demand) — a configuration error, surfaced before any model output is scored.

Parameters:

  • scenario (SODTAScenario)

  • tol (float)

class tabench.metrics.CellSODTAEvaluator(scenario, tol=1e-06)[source]

Bases: object

Model-blind Ziliaskopoulos cell certifier: pure function of (scenario, trajectory) — same two-scale, weak-duality-backstopped design as SODTAEvaluator (adr-021 inherits the adr-020 review hardening). Gates recompute conservation (with the exogenous demand and the absorbing sink), the four aggregate CTM bound families, the x <= N envelope, the initial condition, terminal clearance, and delivery into the sink; scored quantities are the clamped-occupancy total cost, the gap against an eagerly harness-resolved LP optimum Z*, and the pure-arithmetic dual-certificate checks. holding_max is a Tier-B diagnostic — the largest headroom left unused by a connector whose tail still queues (LP “traffic holding”): legitimate on the optimal face for a single destination, never an error.

Raises ValueError at construction if the canonical LP is unsolvable (the horizon cannot clear the demand) — a configuration error, never a scoring-time crash.

Parameters:

  • scenario (CellSODTAScenario)

  • tol (float)

class tabench.metrics.DUEEvaluator(scenario, tol=1e-06)[source]

Bases: object

Model-blind SRDC-DUE certifier: pure function of (scenario, profile).

Parameters:

  • scenario (DUEScenario)

  • tol (float)

class tabench.metrics.TDTAEvaluator(scenario, tol=1e-06, n_levels=2000)[source]

Bases: object

Model-blind TD-UE / TD-SO certifier: a pure function of (scenario, TDPathFlows).

Raises ValueError at construction only if a single-destination SO cell LP is requested but unclearable; a multi-destination (UE-only) scenario simply reports so_bound_gap = NaN and never fakes a bound.

Parameters:

  • scenario (TDTAScenario)

  • tol (float)

  • n_levels (int)

class tabench.metrics.ThreeDetectorEvaluator(scenario, tol=1e-09, env_tol_factor=1.0)[source]

Bases: object

Model-blind three-detector certifier. Pure function of (scenario, field).

Reuse across submissions on one scenario: the reference field M and the clean branch structure are regenerated once at construction. tol sets the absolute count tolerance eps_count = tol * V (V = vehicle scale); env_tol_factor sets the Tier-B envelope_at_resolution threshold in units of one step of link capacity.

Parameters:

  • scenario (ThreeDetectorScenario)

  • tol (float)

  • env_tol_factor (float)

evaluate(field)[source]

Certified metric dict for one emitted interior field.

Parameters:

field (ThreeDetectorField)

Return type:

dict[str, float]

tabench.metrics.node_balance_residual(scenario, link_flows, demand_matrix=None)[source]

Maximum absolute demand-aware flow-conservation residual over all nodes.

Non-zone nodes must conserve flow exactly. Zone node i must satisfy inflow_i - outflow_i = attractions_i - productions_i where productions and attractions are the off-diagonal row/column sums of the OD matrix (intrazonal demand never enters the network).

demand_matrix overrides the scenario’s fixed OD matrix — used by the elastic-demand certificate, which audits conservation against the demand-consistent demand d* = D(u(v)) recomputed from the flows, not against a fixed matrix (adr-005).

Parameters:

  • scenario (Scenario)

  • link_flows (ndarray)

  • demand_matrix (ndarray | None)

Return type:

float

tabench.metrics.marginal_costs(network, link_flows)[source]

Marginal social cost t(v) + v t’(v) per link at the given flows.

Computed via the transformed network’s closed form (b -> b(1+p)), which is exact everywhere — the naive t + v * t' product is unreliable for 0 < p < 1 at subnormal flows, where the derivative clamps to float max.

Parameters:

  • network (Network)

  • link_flows (ndarray)

Return type:

ndarray

tabench.metrics.price_of_anarchy(scenario, ue_flows, so_flows)[source]

PoA = TSTT(UE flows) / TSTT(SO flows), recomputed from both flow vectors.

Roughgarden & Tardos (2002): >= 1 always; <= 4/3 for affine latencies. This is an experiment-level report comparing two certified solutions, not a per-model metric.

Parameters:

  • scenario (Scenario)

  • ue_flows (ndarray)

  • so_flows (ndarray)

Return type:

float

tabench.metrics.marginal_cost_tolls(network, so_flows)[source]

First-best (Pigouvian) tolls v* t’(v*) at system-optimal flows.

Charging these tolls makes the system optimum a user equilibrium (marginal-cost pricing; Yang & Huang 1998).

Parameters:

  • network (Network)

  • so_flows (ndarray)

Return type:

ndarray

tabench.metrics.tolled_network(network, tolls)[source]

The network with the given tolls added to the generalized cost.

Any pre-existing toll contribution is folded into a weight-1 toll column (fixed_cost_new = fixed_cost_old + tolls exactly), so the transform composes and works regardless of the base network’s toll convention.

Parameters:

  • network (Network)

  • tolls (ndarray)

Return type:

Network

class tabench.metrics.ODCertifier(scenario, obs_sensors, heldout_sensors, obs_counts, heldout_counts, oracle_flows, identifiability, certificate=None)[source]

Bases: object

Model-blind pinned-assignment scorer for one T2 task. Reuse across checkpoints.

Parameters:

  • scenario (Scenario)

  • obs_sensors (np.ndarray)

  • heldout_sensors (np.ndarray)

  • obs_counts (np.ndarray)

  • heldout_counts (np.ndarray)

  • oracle_flows (np.ndarray)

  • identifiability (Mapping[str, Any])

  • certificate (Mapping[str, Any] | None)

certify(od_matrix)[source]

Certified metric dict for one emitted OD estimate.

Parameters:

od_matrix (ndarray)

Return type:

dict[str, float]

class tabench.metrics.DynamicODCertifier(scenario, obs_sensors, heldout_sensors, obs_counts, heldout_counts, truth_profile, slice_length, n_lags, n_intervals, identifiability)[source]

Bases: object

Model-blind exact scorer for one within-day dynamic T2 task. Reuse across checkpoints.

The full-network lag tensor is regenerated here from (network, pairs, slice length, n_lags) — nothing model-attested is trusted, so map forgery is impossible. truth_profile is harness-only (the estimator never sees it).

Parameters:

  • scenario (Scenario)

  • obs_sensors (np.ndarray)

  • heldout_sensors (np.ndarray)

  • obs_counts (np.ndarray)

  • heldout_counts (np.ndarray)

  • truth_profile (np.ndarray)

  • slice_length (float)

  • n_lags (int)

  • n_intervals (int)

  • identifiability (Mapping[str, Any])

certify(profile)[source]

Certified metric dict for one emitted (H, Z, Z) profile.

Parameters:

profile (ndarray)

Return type:

dict[str, float]

class tabench.metrics.Bo4MobODCertifier(instance_key, pairs, train_sensor, heldout_sensors, paths, od_end_time, sim_end_time, sensor_start_time, sensor_end_time, engine_version, certificate=None, identifiability=None, installed_version=None, runner=None)[source]

Bases: object

Pinned-engine held-out-count scorer for one BO4Mob T2 task. Reuse across checkpoints.

train_sensor is the anchor GT CSV; heldout_sensors maps held-out date -> GT CSV path (fetched into the certifier’s closure ONLY — never exposed to the estimator, P7). paths are the instance’s static engine inputs (net / taz / od template / additional / routes_single). When runner is injected, paths may be None (engine-free tests).

Parameters:

  • instance_key (str)

  • pairs (tuple[tuple[str, str], ...])

  • train_sensor (str | Path)

  • heldout_sensors (Mapping[str, str | Path])

  • paths (Mapping[str, Path] | None)

  • od_end_time (int)

  • sim_end_time (float)

  • sensor_start_time (float)

  • sensor_end_time (float)

  • engine_version (str)

  • certificate (Mapping[str, Any] | None)

  • identifiability (Mapping[str, Any] | None)

  • installed_version (str | None)

  • runner (CountsRunner | None)

certify(od_vector)[source]

Certified metric dict for one emitted OD vector over self.pairs.

Parameters:

od_vector (ndarray)

Return type:

dict[str, float]

per_date_heldout_nrmse(od_vector)[source]

The per-held-out-date NRMSE for one OD vector (manifest transparency).

Runs the SAME single pipeline as certify() and reports each date’s NRMSE, so the aggregated heldout_nrmse is never a black box. Raises the same infrastructure errors; assumes a feasible (finite, non-negative) OD.

Parameters:

od_vector (ndarray)

Return type:

dict[str, float]