Contamination-Fenced Grading: Why a Backtest of an LLM Is Never Flattering

Answer card

Contamination-fenced grading is how Kestrel measures an LLM-authored plan without letting the model's memory of history do the work. The GRADE statement fences an LLM author to post-training-cutoff, date-blinded days, then subtracts two counterfactuals in the syntax: VS ungated (the same plan with its regime gate removed) and VS null (doing nothing). What survives is judgment, not recall. A model can verify the fence by reading the range; it does not have to trust the number.

The problem: weights leak what code fences can't stop

If you backtest a deterministic strategy, you worry about lookahead in the code. If you backtest a plan an LLM authored or selected, you have a second leak the code fence cannot close: the model already read the tape. Ask a frontier model to trade a well-known week and it is not forecasting; it is remembering. Every printed candle from before its training cutoff is a spoiler it cannot un-see, and no amount of careful data plumbing removes what is baked into the weights.

That is the whole trouble with grading intelligence that was trained on the past. A backtest can be honest for a deterministic system; it is actively misleading for a memorizing one. The average P&L looks like edge and is actually hindsight.

Kestrel's answer is not to trust the model's word. It is to move the grading window somewhere the model has no memory to leak, and to make that window a checkable part of the statement.

The fence: post-training-cutoff, date-blinded days

GRADE is the trust dimension of the language: the honest, counterfactual result of a run. When the subject was authored by an LLM, the judge restricts the evaluation to days after that model's training cutoff, and blinds the dates so the model cannot infer which session it is looking at from context alone.

GRADE plan fade-ladder OVER 2026-02-01..2026-06-30 FILL conservative
  VS ungated
  VS null
  BY regime, session

Read the fence as two constraints working together:

• Post-cutoff. The OVER range must sit past the author model's training cutoff. Days the model could have memorized are excluded, not down-weighted. This is the only window on which an LLM author can be graded honestly.
• Date-blinded. The frames the model perceives are stripped of absolute-date tells. A model that cannot name the day cannot retrieve the day.

Two honest limits, stated plainly: the fence assumes the author's cutoff is known, and it shrinks the usable window to recent history, so a freshly-cut model has fewer gradeable days until time passes. That is a real cost of honesty, and it is why the grade names its support rather than hiding it.

The counterfactuals are in the syntax

A fenced window tells you the model isn't cheating on dates. It does not yet tell you the P&L is skill rather than drift. The two VS clauses handle that, and they are first-class syntax, not a footnote in a report:

• VS ungated: replay the same plan with its regime gate removed. This asks whether the regime gate earned its keep, so you can see how much of the result came from trading only the regimes the plan claimed versus the same thesis run without the gate.
• VS null: replay flat, no position. This is the market's own drift across the window. If the plan cannot beat null, the "edge" is a beta the plan paid premium to rent.

Subtract both and what remains is the thing worth measuring: judgment. This is the load-bearing claim of the whole design: GRADE grades judgment, not parameters. A Kestrel backtest replays committed judgment over a fenced window; it does not sweep a grid hunting for the flattering corner.

Read the cells, not one average

The BY regime, session clause exists because one blended number is a claim you must trust, and a grid is a claim a model can check.

(Illustrative figures: model output over a sample window, not a result, not advice.)

The blended average of those cells might read like a winner. But chop · power is doing suspicious work, and its support flag says extrapolated: the fill model had no live anchor for those prints, so the grade already declined to bank that expected-$. A quote is not a value, and a grade refuses to hill-climb into a corner it could not calibrate. The honest read of these illustrative cells is that the edge lives in trend and vanishes in chop: a shape the grid makes legible, not a trade anyone should place.

Certified vs provisional

The math is the same at both trust levels; only one is signed.

• Provisional: you ran GRADE in self-hosted Kestrel or under a trial capability. Fully reproducible, nothing signed. Iterate freely here.
• Certified: the platform's neutral judge executed the run against pinned data and a pinned fill-model version, then signed the receipt. This is the certified Blotter and Grade behind a shareable proof URL: the evidence an agent shows its human before anyone funds anything.

This is certification over custody. kestrel.markets keeps the judge open and sells the signature, never the strategy. You keep the plan; you buy the proof that the grade is honest.

Where this is not the fit

Contamination-fencing only matters when the author could have memorized the test set. If your plan is a fully deterministic rule with no model in the authoring loop, the fence buys you little; a plain out-of-sample split does most of the work, and you should say so. It is also the wrong tool if what you want is a parameter leaderboard; Kestrel grades judgment, not parameters, and will not curve-fit a sweep for you.

And the honest status, as of mid-2026: provisional grading runs in self-hosted Kestrel; anonymous trial sims, certified Grades, shareable proof URLs, and 402 Offers with Stripe settlement are live on kestrel.markets; always-on paper presence and the human-signed live path are in build. The free tier needs no signup.

Weights leak what code fences can't stop, so an honest grade of an LLM author lives only on the days it could not have already seen.