First-hand · verified against the real thing
Four ways a profitable backtest still loses money
In one line: Cost drag, measurement artifacts, universe-specificity and regime change — the four failure modes, each discovered the hard way inside one research project.
Every failed backtest fails for boring reasons wearing clever disguises. QuantLab's journal now holds enough failures to sort them into families, and the sorting is the useful part: each failure mode has a signature, a tell, and a defense. All four below were discovered inside this one project, on its own results, which is exactly why they are worth writing down.
One: costs eat the edge
The signature: superb results on high-frequency data that fade as fees are applied and collapse entirely with spread and slippage included. The lab's most complete demonstration: the five-minute crypto hunt, closed with seven independent lines of evidence that no cost-surviving edge existed at that frequency at all (R089–R095). The tell you are in danger: any backtest whose per-trade edge is smaller than a few times the round-trip cost. The defense: costs enter the simulation from the first prototype, never as a later “realism” pass; and a gate, not an afterthought — an edge must clear costs to even be discussed.
Two: the measurement lies (proxy artifacts)
The signature: results that are excellent, stable, reproducible — and wrong, because the simulator's bookkeeping flatters the strategy. The lab's retraction: an entire run family (R066–R072) was promoted on a backtest proxy that turned out to measure differently from tradeable reality; the exit-model audit withdrew the verdicts, and later found the same disease in the trend champion — a gate reading exit-bar features, quietly reading the future (the full story in the lookahead-bias piece). The tell: results that survive every statistical test because the tests share the same flawed simulator. The defense: re-derive key results with an independent script, and audit every feature's timestamp against the decision it feeds.
Three: the edge lives in one place
The signature: a strategy shines on the symbols and periods it was developed on, and evaporates on unseen ones. The journal's examples are by name now: R044, R049, R078 — each an edge that refused to travel. This is universe-specificity, overfitting's most convincing costume, because the in-sample evidence is real; it is evidence about a place, not a rule. The defense is leave-one-out validation across symbols and folds, plus per-year gates — the blind campaign's strictest rule was that a result must hold year by year, not on average, because averages bury losing regimes.
Four: the market changes under you
The signature: a config that passes everything, then loses when the regime turns. The lab's freeze-and-blind protocol existed precisely to catch this — and it worked: the frozen configuration that looked like the project's crown jewel failed its strict per-year blind re-test after costs, and the honest full-universe audit showed the surviving champion carried a far deeper drawdown than its small-sample report had suggested. Drawdowns, in particular, are regime-shaped: a backtest that never touched a losing regime is not evidence of a small drawdown, it is evidence of a short memory. The defense: sealed holdout years, regime analysis, and treating the worst simulated path — not the average — as the planning number.
None of these four are exotic. All four were caught by the same infrastructure: the validation gauntlet, running inside a lab built to make honesty cheap. The pattern-level summary of what overfitting feels like from the inside is in the overfitting detection guide.
Research note: QuantLab experiments are presented for educational and research purposes. Historical backtests and simulations do not guarantee future results and should not be interpreted as investment advice.
Sources
Next