BRYME Money · Education
Backtesting 101: the seven ways a backtest lies, and the discipline that catches every one of them — with real examples from a public research lab.
A backtest is a story a strategy tells about the past. Told carelessly, it is a story designed to flatter. The discipline of honest testing is mostly a list of ways to catch yourself flattering yourself. Every example below comes from QUANTLAB and the VWAP mean-reversion lab — public projects whose failed runs are documented alongside the good ones.
Test a strategy on the data you designed it on and it will look brilliant — it has, in effect, memorised the answers. The VWAP lab's final experiment made the gap explicit: about +$262 profit in-sample over its test month versus about +$18 out-of-sample, with a win rate of 54% in-sample falling to 44–47% outside it. Same strategy, same rules — the only thing that changed was whether the data had been seen before. Any backtest that does not show you this split is hiding its most important number.
Lookahead bias means letting the strategy read information that would not exist yet in live trading. It is rarely deliberate and almost always fatal. QUANTLAB's September 2026 audit is the textbook case: its best trend result was gated by a filter reading the exit bar of each trade — future information — and re-anchoring the filter to the true entry bar erased the edge (profit factor ≈1.03, roughly break-even). The fix is a standing rule: audit every feature for when it was actually knowable.
Fees, spread and slippage do not care about your equity curve. QUANTLAB gates every result at 0.05% per side on crypto and retail spread plus swap on forex — and its five-minute crypto research concluded no cost-surviving edge existed at all, proven seven independent ways. An edge that cannot pay its own transaction costs is not an edge; it is a donation schedule.
Results depend on which symbols you test — and testing only the ones that worked is survivorship bias. Two lessons from the log: edges that worked on the discovery universe failed on unseen symbols, and a drawdown measured on 30 favourable symbols read −9.4% while the full 50-symbol universe — including a hostile 2024 — read −28.2% on the same strategy. Same rules, different honesty.
A single profitable run proves almost nothing. Robustness comes from multiple independent checks: walk-forward optimisation (train on the past only), bootstrap confidence intervals on the profit factor, Monte Carlo simulation for the range of drawdowns, leave-one-out tests across symbols and time folds, monthly stability counts. If the result only survives one configuration of one test on one universe, it is not a result — it is a coincidence with a chart.
Paper drawdowns are abstract; lived drawdowns end accounts. The same research that logged a winning period also logged that risking 2% per trade instead of 1% turned simulated drawdowns from painful (≈−28%) to account-ending (−50% to −80%). Sizing, not strategy, decides whether a normal losing streak is survivable — which is what the position size calculator and the 1% rule are for.
'Works great', 'proven system', '9/10 traders' — vague praise is how overfit results travel. QUANTLAB forces every verdict through a fixed vocabulary: VALIDATED, WATCHLIST (promising, sample too thin), REJECT, RETRACTED, OVERFIT. Borrow the habit: give your own results names that cannot be negotiated with.
When you have an honest edge estimate, the expectancy calculator turns it into the only two numbers that matter per trade, and QUANTLAB, explained shows the whole discipline applied end to end.
General information, not financial advice. Everything on BRYME Money is educational. Trading forex, crypto and derivatives involves substantial risk of loss and is not suitable for everyone. Past performance — including any published research — does not guarantee future results. Never trade money you cannot afford to lose.