The model-trading benchmark

Headline — alpha dispersion is the proof

The benchmark measures alpha, and instrument expressiveness sets the alpha ceiling: near-zero room on equity buy-and-hold, large room on options — because a straddle can express “big move either way” and a defined-premium bet can carry a thesis that shares cannot represent.

Options — the expressive alpha axis

No fixed policy wins all three regimes, and the gap between the right and wrong read is large — the headroom is a market fact options open that the equity axis cannot. Grading is against the straddle baselines (not buy-and-hold — a long-underlier hold is not the counterfactual for a defined-premium play), over three real Databento OPRA tapes.

The headroom is a market fact — options open alpha the equity axis cannot

On a matched set, the reference-arm floors alone open a dispersion the equity axis cannot reach — an always-right regime and an always-wrong regime separated by the full straddle. That headroom is a property of the tape, not of any model; the per-tape breakdown is faceatable in the explorer.

Matched-set dispersion = $2,283/straddle, governed purely by regime discrimination: always-right-regime (hold the trends, TP the chop) = +$1,935; always-wrong-regime = −$348. The same read that is a −$1,757 loss in equity is a +$1,604 win in options.

Models pick the right structure — but mistime the catalyst, so realized capture is ≈$0

Structure present

6/6

Arm-rate (on replay)

4/6

Realized capture

≈$0

Best DOWN arm

+$66

vs headroom

+$1,604

Dominant cause — TIMING / JUDGMENT. Models author `WHEN phase regular` (enter at the 09:30 open) and an exit that fires before the catalyst: `EXIT held 90m` exits ~11:00 ET, hours before the 14:00 FOMC move, so they are simply not in the position during the move. The HOLD baseline that authors `WHEN time after 13:45` and holds to close banks the full +$1,604.

Secondary cause — residual GRAMMAR gaps. Two runs (2 & 5) stay de-armed on gaps PR #63 did not close (a range-breakout `crosses … outside`; a `cap`→`min` price anchor). Closing these would lift the arm-rate from 4/6 → 6/6 ON REPLAY — it stays a REPLAY figure, never fresh-authoring capability (the fresh non-CFG lane armed only 1/6; the clean CFG re-run is pending). They are separate features, not the time-exit.

The honest read: Models pick the right options STRUCTURE but MISTIME the catalyst (and hit residual grammar gaps) — a genuine, measurable JUDGMENT/timing gap the benchmark exposes, not a plumbing artifact. The headroom is genuinely capturable: the HOLD baseline proves +$1,604 is reachable.

The pair/watcher axis adds ≈0 on options too — structurally, not by failure

Where the headroom is actually captured — top option plays by α

Capable models DO reach into the options headroom on the cells where the catalyst is legible. Sorted by α (abstentions and ranges show their honest display string and sort last); the exhaustive per-cell, earn-trap, and open-reclaim tables are faceatable in the explorer.

See all 17 in the explorer →showing top 6 of 17 by α

The options roadmap — verified today vs needs-building

Equity control + capability

On equities, no single language model — run end-to-end, authoring its own plan AND managing every wake — beats buy-and-hold on any cell where the baseline made money. That is the CONTROL that proves the options point.

0 of 24 model×cell means and 0 of 48 individual replicates cleared buy-and-hold on the four profitable cells. Mean α vs the best reference arm is negative for all six models. Best single attempt in the whole run: qwen3-max on EQ-DAY-EARN rep0 at $860.34 — against buy-and-hold $2,954.00.

The null control is failed by everyone: 12 of 12 columns traded CTRL-QUIET-CHOP, an edgeless tape whose correct move is to place no order. Restraint is the cheapest thing the benchmark asks for and the roster does not have it.

Single-model axis — mean α vs the best reference arm (negative for all six)

Run end-to-end on the raw single-model axis, no watcher beats its best deterministic reference arm — mean α is negative across the board. That is the control: a saturated, low-expressiveness axis, not a model indictment. Ranked by α; the reference-arm floors and per-cell detail are faceatable in the explorer.

See all 6 in the explorer →

Fleet mean discretion α: OPEN $341.02 vs FRONTIER $871.75 — frontier leads overall. The “open matches frontier” result is axis-specific (managing someone else's well-formed plan is an easier job than authoring one) and does NOT replicate on the single-model axis, where qwen3-max's $0.00s are abstentions, not matched trades.

Capability — not size, and not openness

On the decisive EQ-DAY-EARN discretion trap (the stop-less gap-fill, frozen floor −$2,618), the same trap that separates capability from BOTH size and openness. The gap is judgment, not plumbing.

Capability separates from size and openness — the EARN-trap reclaim, by α

A small closed model and a current-gen open model both reclaim the earnings trap; a stale 7B and a 0.5B do not. Capability is judgment, not parameter count and not licence. Sorted by α (a passive `{}` no-action shows its honest $0.00 display string and sorts last, never as a $0 trade); the full earn-trap, size-sweep, CFG-ablation, and open-reclaim tables are faceatable in the explorer.

See all 13 in the explorer →showing top 6 of 13 by α

The strategist is the edge

The reachable headroom lives predominantly in the strategist's authored, defined-risk plan — before any in-loop management — so the plan, not downstream watcher P&L, is the thing to measure. This is the §5 verdict, stated as a DIRECTION rather than a precise fraction: an exact “% of headroom” is not reproduced from a committed computation, so we state the direction and cite its reproducible complement (the watcher-adds-≈$0 result) as the number.

When the plan carries its own stop, the watcher adds ≈$0. Mean watcher α across 33 fan runs is −$2.07; every disciplined cell sat at exactly $0.00, non-zero only because one over-trading cell lost money. The options pair axis (all 18 columns α=$0) is the same finding on the expressive instrument.

Exhibition / non-governed — Fable is a HOUSE-BENCH agent: house and Bench agents paper-trade only and never hold governed standings (PLAT-ADR-0035 §10/§11). Fable-authored plans are structurally disciplined by default — graded directly on the structural rubric (defined-risk EXIT / max-loss bounded / never-naked-clean / conditional entry / TP). No lever changed the structural grade: told to “ride the move and let winners run,” Fable still wrote a bounded EXIT — the §5 worry that aggressive framing drops the defined-risk exit did not reproduce. Plan noise floor is tiny: $0.39 on EARN, $12.74 on ORB across N=3 same-prompt re-samples.

Reasoning level changes plan CONSTRUCTION, not discrimination — `medium` is best AND cheapest

The strategist axis by α — `medium` is Pareto-dominant

`medium` reasoning has the best matched-set economics at less compute; the fine ordering is INDICATIVE. Latency is measured but UNPRICED (badged, and sortable, in the explorer) — a slow level never 'wins' here. Sorted by α (the degenerate/inert level and the pooled hill-climb FITNESS rows carry no dollar α and sort last, shown with their honest display string, never coerced to a number).

See all 11 in the explorer →showing top 6 of 11 by α

Reasoning level changes plan CONSTRUCTION (position sizing + stop placement), not discrimination — the matched-set twins are byte-identical at the honest vantage, so every deliberating level arms both poles identically (12/12). `medium` Pareto-dominates: best matched-set economics at LESS compute than high/max. If this benchmark drove the choice, ship the strategist at `medium` — best-quality and cheapest. `none` is degenerate (oversize plans that never fire), not restraint; `max` actively hurts (a very tight earn stop blows up on the gap-fill trap).

Methodology + caveats

The guarantees — why a do-nothing agent cannot score

1. 1
   No-lookahead vantage

   A model authors its plan at the cell's honest cutoff (`AUTHOR_CUTOFF` + `authorFromRange`), threaded into the live author only; the deterministic reference arms omit it so the calibration floor is byte-identical. Counterfactual-invariance is verified for every briefing.
2. 2
   Contamination holdout fence — structural, and proven WIRED

   `tests/bench/splits.ts` partitions every cell's seed space into disjoint-by-construction sets: HOLDOUT = certified, sha-pinned benchmark seeds (scored, never trained); TRAIN = a reserved 5000–5999 band (trained, never scored). The eval fails closed — `assertHoldoutSeed` throws `SeedSplitViolation` the instant it is asked to score a train seed — and `splitManifestSha` is stamped on every result record. This is not a green-by-construction unit test: the fence ships with a failing fixture THROUGH THE REAL DRIVER — a test that drives the actual grading path with a train seed and asserts the throw. Apply the decisive probe — DELETE the `assertHoldoutSeed` call site and the fixture goes red — so the firewall is proven WIRED into the shipped path, never merely present.
3. 3
   CFG-honest emission

   Grammar-constrained decoding (ADR-0030/0031) makes invalid Kestrel unrepresentable at the sampling step, so judgment is measured through the action, not through emission noise — necessary infrastructure, judgment-neutral (see the CFG ablation above).
4. 4
   Deterministic replay grade

   The frozen plan is replayed pure-algo (`blotter.totals.floor`, zero model calls); `SimRunId = sha256(gradedBus)` re-grades byte-identically. Local greedy decode is byte-reproducible (measured σ=$0.00); remote temp-0 lanes are NOT (the frontier lane diverged on 1/15 groups, the open lane 9/14) — measured, not assumed.

Caveats — read these before citing any number (NOT footnotes)

1. 1
   The open tier now includes current-gen frontier open models — and they RECLAIM; the remaining axis is RELIABILITY

   The locally-served Qwen2.5 entries (0.5B/1.5B/3B/7B) are small, older-generation CONTROLS used to separate capability from openness — not a claim about today's best open weights. On the decisive EARN trap all four current-gen open-frontier models (Kimi K2.6, DeepSeek-V4-Pro, GLM-5.1, GLM-5.2) act and reclaim, and Kimi matches the CLOSED frontier deterministically (+$1,596). The stale 7B's $0.00 was that model's judgment gap, not an open-tier ceiling. The honest open question is reliability — “CAN” is not “always.”
2. 2
   Where the watcher baseline is Opus, not Fable, we say so

   The strategist / plan-authoring boards use Fable (`claude-fable-5`, subagent lane) — that is where “baseline Fable is the champion” and the plan-grade findings apply. The watcher / execution boards (pair + single-model) are managed by Opus / Sonnet / Haiku / Gemini / open-weight models; there is no Fable watcher row. Do not read the watcher-board tiers as Fable results.
3. 3
   “Discretion” means backstopping an incomplete, stop-less plan — NOT a general skill

   The watcher's positive α on the discretion arm is reclaiming a plan deliberately stripped of its exit — a narrow backstop competence on a constructed trap, not evidence of broad trading ability. On real, well-stopped plans the same watchers add ≈$0.
4. 4
   The hill-climb is TRAIN-only

   The holdout (generalization) split was lost to a mid-run OOM crash (unrelated to the task); the train-vs-holdout gap is not computable. The verdict “no knob beats baseline” stands on the complete TRAIN matched set, but generalization is unmeasured.
5. 5
   The open-lane tier boundaries are NOT resolved

   Tier B of the pair board is the open lane at n=1, and both boundaries touching it are inside that lane's later-measured noise (mean σ $254.38, max σ $2,830.55). A replicated open-lane run is required before those tiers can be claimed.
6. 6
   The open lane's σ was NOT cache-fabricated — RESOLVED (PR #64)

   Historically the open lane's serving path set no cache-skip header, so a σ=$0 could in principle have been a cache hit. Re-measurement disproved it: every published σ=$0 group had divergent `busSha` + `outTok` across replicates (a cache hit is byte-identical by definition). An explicit cache-skip header plus a bounded-numerics INVALID guard now prevent any future fabrication.
7. 7
   Observability — invalid/parse-escaped authoring recorded as “authored” — RESOLVED (PR #61)

   Historically a plan that de-armed on an out-of-grammar clause was indistinguishable from a genuine stand-down without inspecting raw replies. Fixed: an empty/parse-escaped completion now classifies INVALID and is rejected-and-reprompted, never banked as a $0 “authored” stand-down. It did not corrupt any graded floor here.

Still open — honest remaining work

• Hill-climb generalization is unmeasured
  The “no knob beats baseline” verdict stands on the complete TRAIN matched set, but the holdout split was lost to a mid-run OOM crash, so the train-vs-holdout gap is not computable.
• The open-lane Tier B is unreplicated
  The pair board's Tier B is the open lane at n=1, and both boundaries touching it sit inside that lane's later-measured noise; a replicated open-lane run is required before those tiers can be claimed.
• Open-tier reliability at N>2
  Current-gen open models CAN reclaim the EARN trap, but two of the four do so on only 1/2 replicates; the consistency axis wants more replicates and more cells before it is called.