The Dangers of Backtesting - Best Practices to Avoid Disappointment

For the investors reading this article, you can skip ahead to "What is Overfitting?"

By this point you may have run a few backtests, on a simple script or using one of our educational scripts. You may have also achieved a very high Profit/Loss % and are wanting to run the strategy with real assets. Before this happens, I want to caution you and help set expectations.

What is Overfitting?

If an analysis corresponds too closely with a limited set of data we call this overfitting. Markets inherently have noise, that is to say small and frequent idiosyncrasies in the price data. When modelling a strategy, we want to avoid optimizing for a specific period because there is a chance this won't predict the future. 

It'd be like tuning a car specifically for one racetrack, while expecting it to perform well anywhere.

How do I avoid overfitting?

1. Test for random periods

Running a backtest during bull and bear markets is a good first step. However, the reality is you are probably not starting a strategy at the beginning of a new market cycle, but in the middle of one. Sideways markets can be especially frustrating as a strategy tries to decide bear and bull trends. How would you react if your strategy lost money for the first 3 months?

2. Be granular in your testing

Due to market volatility, it's possible that one good month (or one good trade) is enough to offset a large loss. Profitable trades and MDD offer a window to overall performance, but does not tell a detailed story. To help, we've made our own Power BI Template that organizes trade data for deeper analysis.

3. Run a Paper Trade (forward test)

This one's more of a reminder. 

I want to end in saying that these methods are not comprehensive. Each strategy is different, making it hard to offer blanket advice. 

Happy trading!