First-hand · verified against the real thing
The validation gauntlet: what a backtest must survive before you believe it
In one line: Walk-forward, untouched holdouts, cost gates, bootstrap intervals, Monte Carlo, leave-one-out: the checklist QuantLab applies to every result — and the promotion vocabulary that turns tests into decisions.
A single profitable backtest is not evidence; it is a hypothesis wearing a suit. QuantLab's answer was to build a gauntlet — a fixed sequence of tests every candidate must survive, with a vocabulary that forces a decision at the end. This is that gauntlet, written as the checklist it operates as. It is the single most transferable artifact of the whole project: none of it requires a trading account, and all of it applies anywhere a model meets data it has never seen.
Before anything: the time discipline
Walk-forward optimization. Parameters are chosen using only data before the period being simulated, sliding forward through history — the simulator never trains on its own future. Untouched holdouts. Whole periods (calendar years, in the blind campaign) are sealed until the final verdict; a holdout you have peeked at is no longer a holdout. The lab's enforcement mechanism was structural: a separate branch, opened after a strategy freeze, whose only purpose was testing frozen conclusions on sealed years.
The cost gate: where most edges die
Every result is re-measured with realistic costs — exchange fees, spread, slippage; the lab's crypto runs used 0.05% per side, retail spreads plus swap on forex. The rule is absolute: an edge that dies at cost is not an edge. It is the single most lethal test in the gauntlet — the lab's five-minute-crypto hunt ended with seven independent demonstrations that nothing survives costs at that frequency.
The statistical battery
Bootstrap confidence intervals on the profit factor (5th, 50th, 95th percentiles) — is the result stable when the trade sequence is resampled, or a lucky ordering? Monte Carlo simulation — P(profit) and the drawdown distribution over randomized paths, because the average path is not the one you live. Leave-one-out validation across symbols and folds — does the edge survive when any single market is removed, or was it one symbol wearing a costume? Monthly stability — what fraction of months are profitable, and how long is the worst losing streak? Regime and parameter analysis — does it hold across trending and ranging years, and how sharply does performance fall off when parameters move off the chosen value? A result that lives and dies by one parameter setting is curve-fit, not found.
The vocabulary that ends every test
Each run ends in exactly one verdict: PROMOTE (passed the gauntlet), WATCHLIST (promising, sample too thin to trust), REJECT (failed), RETRACTED (a previously promoted result withdrawn — used here, in writing, more than once), OVERFIT (performed, then refused to generalise). The vocabulary is the discipline: it prevents the third outcome most projects use, which is “quietly stop mentioning it”.
For the failures each test was built to catch, see the four ways a profitable backtest still loses money; for the bug that prompted the mandatory causal audit at the top of this gauntlet, see the lookahead-bias story. And the lab these ran inside is documented in how the research lab was built.
Research note: QuantLab experiments are presented for educational and research purposes. Historical backtests and simulations do not guarantee future results and should not be interpreted as investment advice.
Sources
Next