Why backtests overfit: the degrees-of-freedom problem, in plain terms

A backtest overfits for one underlying reason, and it has nothing to do with bad code: every parameter you tune is a question you put to history, and history will always produce an answer that sounds good. With enough questions, some answer will look excellent by chance alone - and that chance-looked-excellent answer is what you then mistake for a discovery.

Knobs are questions, and the past always answers

Suppose you test a rule and it loses, so you adjust a threshold and test again. Each adjustment uses the same historical data to pick the better-looking variant. After a dozen adjustments you have not discovered a rule that works; you have discovered the rule that, among the dozen you tried, happened to fit that particular past best. The fit is real - to the past. Its relevance to the future is what the tuning has quietly consumed.

The multiple-testing intuition

If you test one idea, a good historical result is meaningful. If you test a thousand, some will look good purely by luck - that is not skill, it is the arithmetic of many tries. The danger is that the researcher sees only the survivor: the one good line on the chart, not the nine hundred ninety-nine dead ones. Judging the survivor as if it were the only idea tested is the classic overfit.

The defences, in order of power

• Fewer knobs. A rule with two parameters has far less freedom to bend to noise than one with ten. Simplicity is not aesthetic; it is statistical.
• A holdout you never touch while tuning. Reserve a slice of data the tuning never sees, and judge the final rule only there. If the holdout collapses, the fit was noise. See the backtest validation checklist for the discipline.
• Punish complexity when comparing. Prefer the simpler rule that is nearly as good; the extra fit of the complex one is usually the noise you would otherwise carry forward.

The companion failure modes - look-ahead and survivorship - are separate leaks and are covered in why backtests fail and overfitting detection. None of this is investment advice; it is the hygiene of not fooling yourself with your own historical data.