In-Sample vs Out-of-Sample Testing

In-sample testing is when you develop and optimize a strategy on a period of data. Out-of-sample testing is when you take that finished strategy and run it on data it has never seen. The gap between these two performances tells you almost everything about the strategy's real-world potential.

The standard split is 70/30 or 80/20 — reserve 20-30% of your data as untouched holdout. Develop everything on the first 70-80%, then run the completed strategy on the holdout exactly once. If performance drops significantly, your strategy is overfit and shouldn't be traded.

The critical rule: you can only look at out-of-sample data once. The moment you tweak the strategy based on what you saw in the holdout, that data is now in-sample. Every subsequent tweak contaminates it further. This discipline is hard to maintain but non-negotiable.

A more sophisticated version is walk-forward analysis. Instead of one train/test split, you use a rolling window — optimize on years 1-3, test on year 4. Then slide forward: optimize on years 2-4, test on year 5. This gives you many out-of-sample results and mimics how the strategy would actually be traded.

Expect out-of-sample performance to be 30-50% worse than in-sample on average. A strategy showing a 2.0 Sharpe in-sample often shows 1.2-1.4 out-of-sample. If your out-of-sample performance is comparable to in-sample, you either have a genuinely robust strategy or you got lucky with the split — verify with additional walk-forward tests.