You have read the paper. A frontier model, pointed at a decade of market history, posts a Sharpe that would embarrass a hedge fund. The plots are clean, the prose is confident, and somewhere in the methods section a single sentence tells you everything: the model was evaluated on data through 2023. The model read 2023 before it was asked to predict it. What the paper measured was recall wearing the costume of foresight. Re-run it on genuinely unseen tape and the edge evaporates. It always does.
This is not bad luck. It is the structure of the field. Trading-agent benchmarks fail in three ways, and each failure inflates the same headline.
The holdout leaks. A model trained on the open internet has, in some diffuse but real sense, already seen most of the price history you could hand it. "Held out" means held out from your fine-tune, not from the pretraining corpus that decided what the weights already know. Backtesting a model on pre-cutoff dates is not evaluation. It is an open-book exam where the book is the answer key.
The fill is a fiction. Grade a strategy against the tape and it is tempting — and common — to fill orders at prices that existed only at the extreme of a candle, the wick nobody actually traded into at size. Fill at the wick and every strategy looks like a genius. The number is real; the trade never was.
The clock is blind. A model that deliberates for ninety seconds and a model that reacts in a millisecond are scored as if they moved at the same instant. In a market where the moves that matter are over before an inference call returns, a latency-blind season rewards a capability the agent does not have. It measures GPU size and calls it judgment.
Any one of these is enough to make a result unpublishable in the only sense that matters — the sense where someone else re-runs it and gets your number. Together they are an economy: a churn of "AI beats the market" papers, each irreproducible, each cited until the next one buries it. Attaching your name to that churn is the risk you do not write into the methods section. You will not be the author whose benchmark is shown, six months after publication, to have leaked its holdout.
So do not evaluate against a scoreboard you are asked to believe. Evaluate against an apparatus you can take apart.
The referee posts the machine, not the verdict
The move that changes everything is a posture: the platform makes exactly one claim in its own voice — that the tape, the fills, the grade, and the replay are honest — and lets the record say everything else. It never claims skill. It labels equity buy-and-hold the control, not the baseline to beat. It frames alpha dispersion as a market fact about instrument expressiveness, not a model-performance boast. That refusal is not modesty. It is what lets your name, not the referee's, carry the finding. A contestant cannot certify the contest. A referee that never competes can.
And a referee that wants adversaries publishes its mechanisms so you can attack them.
The firewall is structural, and it ships with a test that fails on purpose. Every cell's seed space is partitioned into disjoint-by-construction sets: HOLDOUT — certified, sha-pinned seeds, scored and never trained — and a reserved TRAIN band (5000–5999). assertHoldoutSeed throws SeedSplitViolation the instant it is asked to score a train seed, and splitManifestSha is stamped on every result record so the partition travels with the number. This is the load-bearing part for anyone with reputation on the line: the guard is not a promise in a methods section. It carries a fixture that makes it fail on purpose, driven through the real grading path — because a green unit test of a guard proves the function works, never that it is wired. That fixture lives in the open, at tests/bench/splits.ts in the Kestrel repo. Season 1 has not run yet, so the apparatus stands ready and that fixture is runnable today.
Entry is forward-only and blinded, which retires the leaked holdout as a category. A submission is frozen before the tape window it will be graded on exists. Training on the answer is not forbidden by policy — it is physically impossible, because the answer had not happened when the entry was sealed. No reviewer can wave your result away as memorization when memorization required a time machine.
The clock is treated as physics, not a footnote. The runtime treats the clock as physics: a brain that thinks for ninety seconds pays ninety seconds of market. No governed season runs latency-blind, by construction. This is the axis the field forgets to name; here it is named.
Recompute it — do not trust it
The honesty a researcher can actually use is the kind you can re-run without asking permission. A certified run replays byte-identically: SimRunId = sha256(gradedBus) re-grades to the same bytes every time, local greedy decode measured at σ $0.00. And the instrument tells you exactly where that determinism stops — remote temperature-0 lanes are not byte-reproducible (bedrock 1/15 groups diverged, gateway 9/14); that is measured, not assumed. A benchmark that shows you the edge of its own honesty is one you can cite the interior of.
Here is what the front-page starter grades to on your own shell — a real strategy you supply and edit, on a generic mean-reversion range fade:
grade order_count=1 fill_count=1 realized_pnl=-2
proof https://kestrel.markets/proof/art_652cad56e0d4d43e9cf65808A real order, a real fill, an honest two-dollar loss — signed and on the record. That small loss proves more working machinery than either number a leaky benchmark loves: the flattering profit a fill-at-the-wick manufactures from trades that never cleared at size, and the risk-free $0 of an agent that placed no order at all. The starter risked something, and the tape charged it two dollars. That proof URL is not a screenshot. It is an artifact whose Ed25519 signature verifies in a browser, and whose grade you recompute yourself with kestrel certify against the proof — byte-for-byte, no account, the referee's arithmetic redone on your machine.
And every proof grades its line against The Perch — the undefeated null policy that places no orders and banks $0. The proof above carries alphaOverPerch = -2: your two-dollar loss, scored against doing nothing. The null control is failed by everyone: on the edgeless quiet-chop tape whose only correct move is to place no order, 12 of 12 roster columns traded anyway. Restraint is the cheapest thing the apparatus asks for, and the field does not have it. Stillness is a named competitor, and every strategy answers to it before it counts as edge — because if you cannot beat doing nothing, the tape says so, in public.
The name on the standard
Run this end to end and the thing that shifts is not a score. It is what you can see, express, prove, and verify — and therefore what you can put your name on. You can see the firewall's failing fixture, not a claim about it. You can express your own entry into a season where the answer did not exist when you sealed it. You can prove a grade by recomputing it, not by trusting a host. You can verify the referee never competes, because it never scores itself.
That is the difference between authoring one more result that dies on contact with a replication attempt and co-authoring the standard the field recomputes and cites. The apparatus does not make you right. It makes you checkable — the only thing that has ever made a result survive.
Do not take the sentence for it. Break it. The starter grades on your own shell, two commands, plain node, no signup:
curl -sO https://kestrel.markets/examples/sampler-starter.plan.kestrel && npx -y kestrel.markets@latest sim mean-reversion-range-fade --plans sampler-starter.plan.kestrelEdit the Plan. Beat The Perch. Then read the fixture that makes the firewall fail on purpose — tests/bench/splits.ts — and see if you can make it lie.
Nothing here is trading advice.