Backtesting is Never Perfect

Backtesting a trading idea is a good way to gain an understanding on how good your trading idea is.  Jumping in as soon as a new trading concept is thought up is generally a poor idea and the signs of a beginning trader.

When you backtest an idea, this requires you to take historical data and identify how your system would've behaved in the past. It gives you confidence that if it worked in the past it will work in the future.

Backtesting Method

Once you have developed a trading system you are generally quite pumped about this. You expect that the new system is going to be vastly superior to your old system and are expecting large quantities of cash to start flowing towards you. This is an emotional high that I typically get every time I developed a new system.

Once you start backtesting systems that you develop you'll see that an awful lot of them really aren't that profitable. Many of them don't trade often enough to make it worth your effort, many of them have more wrong signals than profitable ones and in many cases there's no clear defined exit point that can reduce the risk of the trade.

The stock below might not have been a buy using your backtesting software due to the low price. However you may have actually bought it back in 1996 as the price was much higher. The low price is a result of stock splits.

On this page, I'm going to introduce a number of ways in which your back testing may be skewed. That is to say, you think you are doing a good back test but in fact, for a number of reasons, the end result will not be what you expect in the future. In some cases if you use better data than I am using some of these issues should disappear however I strongly believe that the majority of them will remain.

So after using your back testing software to obtain numbers such as the win ratio, maximum drawdown and overall profitability you may be thinking that these numbers are 100% true. When you have this belief, I think the sooner you give it up the better you will be because from my opinion that is a false belief. The only thing that you can expect your back testing software to do is to give an approximation of what to expect going forward.

Backtesting Errors

Multiple reasons why your backtesting software will only give you an approximation of what to expect going forward are listed below.

1. Splits - let's take a look at a stock like DELL (picture above) back in 1996. My data base has a breakout price for DELL at $1.2 back in April 1996.  From there it went on and increased over 4000% by 1999. Unfortunately, a lot of traders won't trade a stock which trades less than $20.  Interestingly, in 1996 DELL was well over $20 so you actually would have been alerted to it's potential.  The issue is DELL has split so many times that the value of it back in 1996 is now listed as $1.20 so when you use a high closing price in your code you will miss stocks like this.

2. Sorting the entry - When you develop a trading system you need to code your backtesting software so that it can understand your idea.  At some point you need to tell the software how to sort the stocks that are identified. This can be done in a number of ways such as alphabetically, the highest close, the lowest close, or the largest volume just to give you some ideas on how you could potentially choose which stocks to purchase. You will find that the way you sort the stocks will have a large effect on your end results.

3. Takeovers and bankruptcies - Over the years great stocks get taken over and other stocks go bankrupt. Unless the data set that you have contains all stocks that were ever traded there's a high probability that when you run your backtesting software it will not pick up these types of stocks as they simply will not exist in your dataset. Enron, the bankrupt utility company, would be a good example of this. Your software would not have caught the large increase in this stock nor would it have played the downside. These types of stocks can have huge effects on your return.

4. Bad data - I use a free data source and there are evidently errors in this data. For instance, occasionally, on statutory holidays, when the market is closed, the data base indicates that the stock was traded.  In these situations, the volume is zero however the higher the low tends to be incorrect. In some cases these trades make vast sums of money which are totally artificial. Another common occurrence is missed splits which is seen above.

5. Coding errors - When you first start backtesing, there is a good probability that you will have some coding errors.  If you do not catch these errors, you may believe you have a great system when in reality you will lose money.

6. Missed entries - Backtesting software will always buy as much stock as you said it should provided the price is hit. In most cases this works great but there will definitely be times when only 100 shares are traded at the low of the day which just happens to align with the price that you want to buy 5,000 shares. So in reality you may only pick up 100 shares if you're lucky while the computer and the back test will indicate that you own all 5,000.  You may be able to avoid this error when you have all trades going back 20 years.

7. Your availability - a computer can work 24/7, 365 days a year.  You cannot.  Ultimately you will need down time or you will likely end up sick and end up with downtime anyway. For instance, you might decide to take some downtime at the lake and miss an important trade.  Since vacations are generally a good thing and you require them to maintain a balanced life you will need to accept the fact that every once in a while you will be unavailable to trade.

8. You were not there - Not surprising the backtest that you are doing did not have your influence. So if you purchase at the open for example and were not in a specific trade historically then your presence in the trade may affect the opening price and the open may have been different than it is recorded as being.

9. Not following your system - Backtesting software just follows the code that you wrote. When you indicate that you only want to own a maximum of 8 stocks that's what it will do. It will not buy 7 and it will not buy 9 it will buy 8. You, on the other hand, may buy more or less than 8 and also may allocate your capital differently than you've coded.  Thus your behaviour, unless extremely mechanical, will be different than what the computer generates.

10. Dividends - In most cases the stock price continually drops due to the payment of dividends. So from a historical perspective the price of the stock may have been $10 but due to dividends being paid it may now only be $2 or 3 in the databank. This isn't a concern provided you don't have a minimum price which you will purchase a stock.

11. The database - When your backtesting software is also from your data provider this may not be a concern. However, when you are using back testing software that is independent of your data provider then it will be up to you to transfer a list of symbols into your computer program. You also have to identify how to add new companies as they come onto the stock exchanges. When you leave off a stock, then when you carry out your back test, you obviously will not find that opportunity when it is presented.

Summary

The above examples where just obvious ones which I have picked up while backtesting stocks using Amibroker fed with data from Yahoo to back test a number of trading ideas I have developed. There are likely more concerns which you must be aware of however, I think if you just understand that the results that you get after a back test are only an approximation of what your system does and not exactly what your system will do in the future then you've got a fighting chance to succeed.

Terms of Use and Website Agreement