Key Questions To Ask About Your ETF’s Back-Test

Investors joke that they never see a bad back-test, but I wonder about the truth of this statement.  Do investors ever see a good one? By that I mean; is it solid and robust?

Given that back-tests are a part of every marketing pitch, I think it is necessary to apply a few common sense tests to filter out the fact from the fiction.

Proof Can Only Be In The Pudding

The proof of a robust back-test is that its results should be replicated by the live performance. This is almost never the case. The best researchers admit to back-test shortcomings and have told me I should “take their alpha and cut it in half”.

Many back-tests are based upon the assumption that the future will equate to the past. It is also tempting to feel regret when you see positive - but simulated – returns over a given timeframe.

“If only I had known, I could be on the beach now, sipping cocktails and not giving a care about the world!”

This regret could be known as “look ahead bias”. It results in strategies from product providers that cope fantastically well with the crises of the last 10 years, but never do quite so well when they go live.

Strategies with “look ahead bias” often take relatively simple concepts, but build in complex filters, risk controls or loss mitigation strategies. These strategies are often optimised to fit in with the historic data, which is fine, provided the future path of financial markets is the same as the past. Again, it never is. The next shock will not be like the last and the risk controls will not be as effective.

Factoring In All Costs

A surprisingly large number of back-tests assume no transaction costs. This is not just the cost of brokerage and stamp duty; there are also bid/offer spreads, market impact costs and rebalancing costs.

For instance, bonds mature and have to be replaced – the rebalancing costs can be large, particularly for credit strategies. Overseas strategies may also be subject to dividend withholding tax which will eat into returns.

Covering Up A Multitude Of Sins

I often find back-tests are selective with the start date they choose to display. The year 2000 is a favourite starting point, as that is the year the Dot.com bubble burst and the year that the bull market in risk factors such as value, minimum variance and quality started. It was also a low point for many commodities. I recommend you ask for data starting in 1996 or earlier and I guarantee the excess returns will be less attractive. Data can be expensive, but this should not stop you asking for more information.

Economic data in particular, but also corporate data, tends to be adjusted and revised in the following weeks, months and even years. Common examples are GDP growth and inflation. There are many databases on which to run a back-test, but only one or two databases that are “as reported”. Thus if the trading strategy rebalances quarterly, for a given year, it should only use the accounting and economic data that was available at that particular point and not made available or altered later.

One example is the Office for National Statistics reporting that the UK had a double dip recession in 2010/11, which caused a political storm at the time.  That figure has now been revised successively to become a period of growth.

If you are presented with back-test data and live data for a strategy it is probably worth asking if there have been any changes since the strategy went live. If so, how has the data presented been spliced together?  This does not happen often, but some of the charts that have been presented to me have been highly misleading.

Key Questions On Your Back-Test

What do I look for from a good back-test? I like to see intuitive strategies that are based upon published statistical research that has been subject to robust peer review.

I like to see back-tests that cover as many financial cycles as possible, using “as reported” data.

I also like to use logarithmic charts so I can see if the returns of the strategy were relatively evenly distributed or came in a short period. A logarithmic chart will also help you to spot any big drawdowns that occurred early in the life of a strategy.

A performance attribution is also often useful to ensure that the returns of a strategy are broad-based or reliant on one of two securities.

We are in a risk-taking business and that may mean we have to lean heavily on back-tests when assessing a new product. The back-tests that are presented to investors can be very persuasive, but a dose of healthy scepticism and a few common sense questions can help weed out the few innovative products from the batch of financial charlatans.

Peter Sleep is senior portfolio manager at Seven Investment Management