You know how it feels. Time has come and gone, and still no progress. Then hours later, you’ve found the mistake, and you’re absolutely kicking yourself. Coding is very much a case of trial and error. You just need enough stubbornness not to give up to get your code to run properly. Most of the coding I’ve done over the past few years, has been geared towards developing systematic trading strategies. I spend most of my time trying to think up trading ideas. This step basically has nothing to do with coding. It’s more a case of observing the market, reading books, scouring academic papers and gathering ideas from general market chitchat etc.
The next step is the coding phase, when I try to verify my hypothesis. This involves accessing market data and coding up various trading rules. Perhaps unsurprisingly the coding stage can be fairly time consuming, in particular when debugging the code. Whatever trading rule we choose to test, we want to make sure that our backtest is representative of a real market. We also want to ensure that the trading rule we are testing actually matches with the one in our head as well. If the backtest is not realistic and our trading rule is implemented sloppily, the whole exercise becomes a bit of a waste of time.
It is obviously very frustrating, when after hours of going through code, you find there’s some trivial mistake in your code which has caused all the problems in a backtest. After many years of doing it, I have to admit, I am still not immune to this! However, what I can say, is that over time, you do get a feeling for the types of backtesting mistakes which can occur most often. Whilst my focus is on trading, some of these mistakes can be applicable for many data science problems. I’ve listed some of the mistakes you can make when backtesting a trading strategy. I think I’ve made most of these mistakes over the many years I’ve been doing this… Obviously, these are not the only mistakes that can happen when you backtest a trading strategy. The focus here is on the mistakes which render a backtest totally unrealistic (rather than the general approach of how to form a trading strategy and the hypothesis etc). I hope it provides food for thought. Of course, there are many other pitfalls in backtesting.. which Cuemacro can help you avoid 🙂
Problem 1: you’ve messed up the time zone in your market data
Somewhat annoyingly for anyone trading, the world does not have a single timezone! As a result, you need to be careful to make sure that whenever you’re dealing with market data (especially intraday data). It is easiest to store all your market data in a single time zone (typically UTC). However, you need to be careful that all your timestamps are still correct. When reading in data, vendors might not always give you UTC data timestamps, and they could be in their own local timezone. It might take a bit of detective work if the data you are given is not labelled. If it’s exchange traded, you know the opening and closing times at least of the trading day, which can help. For FX, which trades 24/5, you don’t have that. You can however, infer the timezones by looking at volatility patterns, eg. 8.30am NY, when US data is released, if you have no other information at all.
Problem 2: you are trying to trade an asset outside market hours
Let’s say you are backtesting a strategy which trades an Asian asset. You might be able to get quotes for that asset at New York close. However, in practice, that asset might not really trade at that time. Hence, it’s likely the quote is stale by a few hours, when the asset was actually trading. If you end up backtesting that strategy but end up using other data at New York to generate the signal, guess what…? You’ll get look ahead bias! You are trying to trade an asset at a price, which you cannot trade. The tell tale signs are usually an unexpectedly good Sharpe ratio!
Problem 3: forgetting about total returns
If we are long an asset, the P&L we accrue is not simply made up of the spot returns. We usually also have some carry component. Take for example a long AUD/JPY position. If we are holding it for a long period of time, the carry we accrue can be considerable. If we ignore total returns in a backtest, we can end up making our simulated P&L totally unrepresentative. We can either construct our own total return indices or very often, we might be able to find pre-made ones.
Problem 4: using incorrect transaction costs
Whenever you do a trade you have some element of market impact and we might experience some slippage. If you are taking liquidity you will cross the bid/offer spread. A simple way to take this into account when backtesting is by adding transaction costs, to cover both paying the spread and also the slippage. We can also create market impact models, to make transaction costs a function, rather than a constant. Many factors will impact what our transaction costs will be, the volume traded of an asset, how much we want to trade, the time of day etc. If we make our assumed transaction costs too small, we might find that strategies might appear profitable in our backtest, but less so when actually trading. If we are trading large sizes or we are trading at very high frequencies, transaction costs can make a substantial impact on our returns. Hence, we should not think that transaction costs are “not important”.
Problem 5: wasting time making a backtest too low level
If you are doing very high frequency trading, it makes sense to make your backtest as accurate as possible. You might want to model the order book and model the market at a tick level. However, this is likely going to be overkill for understanding whether a daily trading strategy is profitable in a backtest. Yes, we want to make a backtest realistic. However, we don’t waste time modelling things which are irrelevant for the time frequency you are looking at.
Problem 6: not documenting a trading strategy
It’s always so much quicker not to bother documenting a trading strategy or commenting the backtesting code…. it also makes it difficult, when you come back to that code a year later and have no idea what is going on. Time saved earlier, will be inevitably be spent later. Document your code properly (and the underlying strategy) and this won’t happen.
Problem 7: making every backtest a one off
If you don’t have a generic way of backtesting, it can be incredibly time consuming to update your analysis. Every time you come up with a new trading idea, you end up having to start a backtest from scratch. In practice, backtests should all be pretty similar – the main differences are in how the signal is calculated (which is where you should be spending your time!). Do yourself a favour and spend time making your backtesting process as generic as possible. It’s worth looking at my own backtesting framework finmarketpy, which is pre-made library which does exactly this!