You’re Probably Backtesting Wrong

The article “You’re Probably Backtesting Wrong” was originally posted on Dave Mabe’s blog.

I’ve worked with a lot of traders over the years, many of whom are quite smart and able to backtest their ideas to generate successful trading strategies. But when I look at their backtesting routine, almost all of them completely ignore the most productive step for creating profitable models with a strong and durable trading edge.

Backtesting this way is like riding a horse compared to walking while not realizing that airplanes exist: you can travel a long way on horseback (even across the entire country, in theory) and you’ll leave mere walkers in the dust. However, if you’re unaware of air travel then you may as well be a sloth who has no idea that he can travel at least 100 times faster by plane.

Here’s a standard backtest workflow that I see traders doing over and over:

1. Trading strategy idea generation
2. Code the backtest
3. Run the backtest
4. Analyze aggregate metrics and equity curve
5. Decide to trade it or return to step #1

Let’s look at these steps one at a time in detail.

Trading Strategy Idea Generation

Generating ideas is the hardest part of trading for most since it starts off like a chicken and egg problem: ideas come from experience but to have good ideas you first need to have some experience. With enough practice, though, this becomes the easiest aspect of the whole process. I know some readers will find this hard to believe, but the more you’re able to exercise your “trading idea muscle,” the stronger it will be.

Because it takes regular practice and hard creative work, this step weeds out a lot of would-be traders.

Code the Backtest

For most traders, coding the backtest ranges from difficult to impossible. Even high-level traders who are excellent at coding have a hard time context-switching from trading (or other tasks) back to writing code. This is exacerbated by the fact that running the backtest takes so much time. You can’t stay in a productive coding zone when you get interrupted by a backtest that takes an hour or more to run.

Run the Backtest

Depending on the markets and instruments you’re running a backtest on, this part will likely be the most time intensive. If it does NOT take a long time to run, you’re probably not scanning the entire market, thereby missing out on profitable trading opportunities.

If your backtesting platform doesn’t have the ability to scan the entire market, then – back to my analogy – you’re on horseback. Better than walking? Yes, and certainly faster than a sloth, but you’re not really in the same game as successful systems traders.

Analyze Aggregate Metrics and Equity Curve

A lot of traders get tangled up here in deciding which aggregate metrics to focus on. There are plenty to choose from and it’s tempting to try to find the “fanciest” one that supersedes all others. Rather than trying to find the perfect single metric, I look at a handful of metrics in this phase. (While important, how I choose my metrics is beyond the scope of this post. I’ll outline my exact process in a forthcoming message to my newsletter.)

Decide to Trade It Or Return To Step 1

Using whatever criteria you’ve settled on, you either move forward and trade the strategy as is, or you start the process over at step 1.

Here’s this workflow in visual form – note that the pink boxes represent particularly time-consuming parts of the process:

The Step You’re Missing

This workflow is… fine. You’re on horseback competing against walkers. But there’s a very important phase of my routine that is completely missing from this workflow. How important is it?

I call this the Filter Phase and I literally spend 95% of my overall backtesting time in this phase. It allows me to reduce the time I spend creating a strategy from days and weeks to minutes and hours. It’s the reason I’ve been able to create and trade 15+ (and counting) profitable strategies.

The Filter Phase

In order to add this step to your workflow, you’ll need to be able to make a couple of small adjustments to your overall routine. Your backtesting platform must have the following capabilities, otherwise, you’re stuck on horseback:

• Export a trade list from the completed backtest to a CSV file
• Add a custom column to each trade in the backtest

These features are not available in most backtesting platforms.

Let’s say you have a strategy based on an 8 and 20-period moving average crossover. Or maybe you’ve created a basic strategy using ChatGPT as I did. It works ok, but you want to improve it. You have a theory that, say, volume relative to average volume (a.k.a. Relative Volume) has an effect on the profitability of the strategy.

Most traders at this point will “guess and check” – that is, pick a number that seems reasonable to them for relative volume and then they’ll code it into the strategy such that the trade set you end up with in the backtest is a subset of the original. This is a subtle mistake that costs traders a ton of time and resources.

Here’s what you NEED to be able to do in your backtesting platform. Instead of modifying the original strategy, you simply add a column to the trades that end up in the resulting trade list. When you do this, the number of trades in your backtest DOES NOT change – you’re only adding additional columns that capture the numerical value of different filters that you MIGHT decide to filter on later on in your process.

For example, let’s say your original backtest produced a trade set like this:

After you add your column, your new backtest looks the same as the original, but with a new column added that corresponds to the filter that you want to examine (see the column called Relative Volume).

The important thing to note here is that you’re still including all the original trades even when you’re confident you’ll be filtering some of them out when you eventually go live with the strategy. It’s not at all important to remove them from the trade list in the coding phase. When you have a Filter Phase in your backtest workflow it’s a piece of cake to remove them in this step.

OK, So What’s the Big Deal?

So you’ve got some extra columns in your backtest. How does that help?

Because you have the entire range of values for Relative Volume in your trade set, you can determine the optimal filter value without having to run a bunch of time-consuming backtests to find your answer. You’ve gone from a brute force “guess and check” with at least hours between the guesses to learning and iterating on the knowledge you’ve gained in seconds.

The effect on your workflow is MASSIVE. This allows you to reduce the number of times you have to code and run a very time-consuming backtest from oftentimes several dozen to just… one time. What once took days and weeks in backtesting, can now be accomplished in minutes and hours.

As you can imagine, this opens up entire worlds of trading strategies to test (and learn from!) really, really quickly.

Rapid, Continuous Learning

When you add the Filter Phase to your routine, you’ll get better and better at efficiently learning what makes a trading strategy tick. It allows you to go from answering a very simple question:

Should I trade this strategy? Yes or no?

to a much more powerful and profitable question:

Under what circumstances should I trade this strategy with full size?

To answer that question, you must thoroughly understand your trading system. If you’re not using some form of the Filter Phase in your backtesting workflow, you’re riding horseback in a world of airplanes.

***

To learn more about how you can use this concept to create your own trading strategies, sign up for my free newsletter where I regularly share my techniques for doing so.

Go Source