Backtests that lie

After this lesson you'll know the four ways a backtest deceives — including the one that fooled our own research desk, told with the real numbers.

A confession, first

This Academy's research side runs backtests professionally, with test harnesses, audits and version control. We're telling you the following story about ourselves anyway, because it's the best argument this course has.

Our desk once ran a strategy whose backtest showed spectacular results, on the order of plus-nine-thousand R across the test period. It was the best system we'd ever built, and one line of its code was wrong. When a trade's entry and its stop could both have been hit inside the same price bar, the simulator assumed the entry filled first and the profitable direction followed. It was resolving an ambiguity in the strategy's favor, using knowledge of how the bar ended that no live trader could have had. Fixing that single line moved the result from plus-nine-thousand R to minus four and a half thousand R. The strategy had never been profitable at any point, and every statistic downstream of the bug — win rate, expectancy, the equity curve we'd admired — had been measuring the bug.

The part worth carrying out of it: the error arrived looking like success rather than like a fault, so nothing about it prompted anyone to go looking. That's School VII, course 1's confirmation bias, now aimed at your own code.

The four lies

Lookahead — the story above: any decision that uses information nobody could have had at the moment the decision was made. It gets in through small cracks, like sizing a position on the day's final volume when the entry happened mid-morning, or computing an average through today's close and trading on it that same morning. The standing question our desk asks of every input is whether the number exists yet at the instant it's used. There's a useful symptom to know as well: if making a rule's stop tighter keeps making the backtest better, and it keeps doing that all the way down, something is inverted, because in the real world tighter stops eventually start bleeding.

Survivorship — testing today's stock list on the past. Run "S&P 500 members" over twenty years using the current members and you've quietly excluded everything that collapsed out of the index along the way, which hands your strategy a list it couldn't have had at the time. School IV, course 8 flagged the model-book version of the same problem. The fix is point-in-time universes, and failing that, a lot of suspicion about any long backtest without them.

Missing costs — School I, course 9, applied to a simulator. A strategy trading daily at a true 20 basis points per round trip pays roughly 50% of position value per year in costs, so a backtest charging zero can show a loser as a star. An honest backtest names its cost assumptions per trade somewhere in the write-up. When you can't find that sentence, assume the costs weren't charged.

Overfitting — the hardest of the four to see. Tune enough knobs against one stretch of history and the system ends up memorizing that stretch's accidents rather than any mechanism underneath them. The backtest looks perfect and the live results come in around chance. Things to look for: a lot of parameters, an unusually smooth equity curve, and results that fall apart when any single setting moves a notch. A robust edge sits on a plateau, meaning it survives its neighbors' settings as well as its own. Professionals go further and discount a result by how many variants were tried before it, since a great result found on attempt 200 is mostly an artifact of the search, and attempt 200 is what gets shown.

The stance

For your own tests, the Arena's tooling applies these checks mechanically. For everyone else's, treat a backtest as an argument that has to survive questioning, and use the four above to do the questioning. If the person selling it can't answer them, that's usually as far as you need to go.

Check yourself

1. Why did our bug produce a great backtest rather than a broken-looking one? (Lookahead resolves ambiguity in the strategy's favor, so the error arrives as profit, and nobody goes hunting for a bug inside a good result.)
2. A 15-year backtest of "current Nasdaq-100 stocks" shows steady gains. The flaw? (Survivorship. Today's membership is the list of what survived, and the casualties are missing from the test even though they were there in the past.)
3. What does parameter-neighborhood testing detect? (Overfitting. A memorized past falls apart as soon as the settings move a notch, while a real mechanism sits on a plateau and survives its neighbors.)

The idea this lesson installs

A backtest error announces itself as a wonderful system.

Next: Course 3 — "Paper trading and forward testing, done properly."