# Your Backtest Is a Story, Not Evidence (/blog/your-backtest-is-not-evidence)

2026-07-16 · kestrel.markets

You have a folder of green equity curves you do not believe.

You already think it: every one of them might be a lie, including the ones you drew. You have run enough parameter sweeps to know what a curve looks like when it has memorized the tape instead of learning the market. You have watched a strategy that Sharpe'd beautifully in Backtrader die the first week it touched money. You caught the vectorbt run filling at a price that never traded, the Pine backtest anchoring an exit to the wick of a candle no order could have hit. So you distrust the whole genre — and, being honest, you distrust yourself most, because you are the one holding the seed that made the curve green.

That distrust is not neurosis. It is the only correct response to the artifact you were handed. **A backtest you cannot independently re-run is a story, not evidence.** A PnL screenshot is a story with a picture. A curve you can regenerate but nobody else can is a story you told yourself twice. None of it is science, because the defining move of science — someone else recomputes your result and gets your number — was never available. The walled-garden backtester cannot even offer it: the fills are proprietary, the data is licensed, the run never leaves the building. Overfitting stays invisible precisely because reproduction is impossible; there is no independent recomputation to expose the memorization.

We are not writing this to sell you certainty. We are engineers who also stopped trusting green curves — our own included — and got tired of arguing about screenshots. So the platform was built to refuse to ask for your trust. It is a referee, not a vendor. It makes no claim about skill, no claim about alpha, no claim that any agent is good. It makes exactly one claim in its own voice: **that the tape, the fills, the grade, and the replay are honest, and that anyone can recompute them.** Everything else, the record says.

Here is what that looks like when it costs you nothing. This is the starter sim from the front page — a deliberately minimal Plan on a broad-market fixture, one share at the open, held to a risk-envelope TTL:

```text
$ npx -y kestrel.markets@latest sim mean-reversion-range-fade --plans sampler-starter.plan.kestrel
grade   order_count=1 fill_count=1 realized_pnl=-2
proof   https://kestrel.markets/proof/art_…
```

`realized_pnl=-2`. A real order, a real fill, an honest two-dollar loss — signed and on the record. Not a flattering zero, not a curated win: a small loss the tape actually gave, which proves more machinery than any hero screenshot, because a system willing to print your loss was not built to please you. Re-run the same recorded tape locally and the bytes are identical. The same tape for every agent. **$0 to reproduce** — a result that costs nothing to recompute cannot hide behind the cost of recomputing it.

And the loss has something to lose *to*. Every grade is scored against **The Perch — the undefeated null policy that places no orders and banks $0.** Doing nothing is a named competitor with a scoreboard line, and your strategy answers to it before it answers to you. Open the proof and it does not report profit; it reports `alphaOverPerch: -2`. Your machinery lost to stillness, out loud, in public. Most strategies do. A backtester that will not tell you when you lost to doing nothing was never measuring edge — it was decorating a story.

That proof URL is not a marketing asset. It is a receipt an auditor could subpoena: a `pinnedDataRef` with its sha256, a `pinnedFillModelVersion` of `honest-fill-v0`, a `pinnedAuthorProvenanceRef` attributing the strategy to you and never to the platform, `judge='kestrel-open-judge@0.1'`, an Ed25519 signature over the whole record. Every identity that could change the number is pinned and named, so the number cannot drift without one of them changing where you can see it.

Now the distinction. Verifying that Ed25519 signature only proves the platform signed the bytes. It does not prove they are the *right* bytes. So the platform hands you the harder verb: **`kestrel certify <proof-url>` re-projects the Bus on your own machine, byte for byte, and checks that your recomputation equals the published one.** Not "the platform vouches for this run" — *you re-ran it and got the same bytes.* That is the crossing from believing a curve to recomputing it, and it is offered to anyone holding the URL, including the coding agent you sent ahead to do the migration recon. The one honest boundary, stated once: today you falsify against three curated broad-market fixtures, not yet your own instrument and tape — bring-your-own import is the next rung, and it is filed.

**Overfit is a confession. Certify is a command.**

The asymmetry is the whole point. Overfitting is not a technical failure; it is a thing the curve is quietly admitting about how it was made — that it fit the past because it was allowed to see the answer. It is a confession the walled garden lets you keep private. Certification is the opposite grammar: an imperative you issue to anyone, yourself first, that recomputes the result and lets the bytes settle the argument. One is a secret you protect. The other is a challenge you publish. The determinism is what makes the second possible — an LLM's reasoning can read the catalyst text and drive the execution, but a Kestrel Session admits no wall time, no nondeterminism, no silent default into the run, because the brain stays outside the runtime it steers. Reasoning in, reproducibility preserved. That is the trade the whole genre told you was impossible: perception *and* byte-identical replay in one runtime.

Nothing changed about your skill. What changed is what you can do with a result. You can *see* the market as attributed text your model reasons over. You can *express* a whole strategy as one Plan small enough to stay in context. You can *prove* a run mints a signed, pinned, shareable proof instead of a screenshot. And you can *verify* — recompute someone else's proof, or your own, and let the recomputation decide. You end up not more confident but better armed: you publish results anyone can falsify, and you let the bytes, not your word, settle whether the edge is real. That is a strictly better position than sitting on curves you privately assume are lies.

You do not have to believe any of this. That is the point. Falsify it from your shell — two commands, plain node, no signup:

```bash
curl -sO https://kestrel.markets/examples/sampler-starter.plan.kestrel && npx -y kestrel.markets@latest sim mean-reversion-range-fade --plans sampler-starter.plan.kestrel
```

It runs the day loop, prints the honest grade against The Perch, and mints a proof URL you can hand to anyone — including the `kestrel certify` that recomputes it byte for byte. Edit the Plan. Beat the null policy if you can. Publish what survives.

Stop protecting the confession. Issue the command.

*Nothing here is trading advice.*