Why Many Smart Beta Backtests Fail

The September 2015 announcement that John Hancock and Dimensional Fund Advisors were teaming up to launch ETFs set the ETF world on fire. John Hancock seemed a natural fit for the ETF market, with a massive distribution capability and a strong track record in the mutual fund business. But Dimensional was a shock: It is famous for distributing its funds only to advisors who work with Dimensional, who have undergone extensive training and education with Dimensional. What would opening up those strategies to the masses through ETFs do?

Lukas Smart, Dimensional’s senior portfolio manager, will tackle that question and more as a headline speaker at the forthcoming Inside Smart Beta conference, taking place June 8-9 in New York City. In the run-up to that conference, Inside ETFs CEO Matt Hougan sat down with Smart to talk about why Dimensional entered the ETF space and how investors should think about its funds.

Matt Hougan: The John Hancock/Dimensional deal rocked the ETF world. Why did it make sense for Dimensional?

Lukas Smart: The basic conceptual hurdle that Dimensional had to overcome prior to entering the ETF space was whether our way of managing portfolios would work in an ETF. We have a long history at Dimensional of using an index-like approach that’s active and systematic, and we wanted to ensure that would work in an ETF framework. After some work, we found that we could in fact implement our approach that focuses on the dimensions of expected returns—or what other people may call “factors”—in ETFs.

Hougan: What was your concern?

Smart: Our history involves adding value through very careful implementation. So the question was, does the ETF framework—and particularly, a fully replicated ETF framework—support that level of care? We answered that question through a very careful construction of the index.

Hougan: Why would the ETF bring more concern than a mutual fund?
Smart: The key difference is that the ETF framework has a greater sensitivity to tracking error. In our traditional mutual funds, we don’t have a tracking error constraint, so we can be more flexible in implementation. That means a portfolio manager in our mutual funds has some discretion, and can determine if a particular trade has a benefit that outweighs the cost of implementation.

In the ETF space, you don’t have that discretion at the portfolio level, so you have to put rules into the index to accommodate that. Our index committee spent a lot of time putting in rules around things like Index Memory^TM and enhanced redistributions.

Hougan: I’ve never heard of “Index Memory” or “enhanced distribution.” What do you mean by that?

Smart: You could summarize Index Memory by saying you don’t spend a buck to make 50 cents. It takes into consideration the position of the entire index and considers if a particular change makes sense. So, for instance, we put in place concepts like “hold ranges.” Our large-cap index cuts off at 750 names, but if we have a company that bounces around from name 750 to name 800, we don’t want to constantly trade it in and out of the portfolio; that has a real cost.

We have similar rules around things like momentum, small changes in weight, etc. If a change won’t have a meaningful impact on the portfolio, it’s critical that we avoid the cost of doing it, and we had to design rules to accommodate that.

Hougan: And enhanced redistribution?

Smart: Enhanced redistribution can be compared to turning lemons into lemonade. For example, periodically, the market may have some sort of corporate action that will effectively take a position away from you; for instance, one company buys another for cash, and the company that gets bought had a 50 basis point position (0.50%) in your portfolio. All of a sudden, you have 50 basis points of cash.

There are three ways you could handle this. One way would be for the index to pretend it never happened. That’s fine for the index, but the portfolio manager then has to put that cash to work, which means buying all the other individual names in the index, usually in very small amounts.

That can have a real cost that, relative to small trades, presents high return hurdles. Another way would be to replace the lost position with a similar security, and many indexes do this. That’s fine if you’re looking for a market-like return, but we’re looking to outperform the market through the structure of our index, so that wouldn’t work for us either.

What we actually do is use it as an opportunity to re-establish the return objective of the index and thus of the ETF. When we have a corporate action like the one in this example, it’s effectively a chance for us to do a rebalance, so we look at the most recent prices to find securities that the index most wants to hold, and we’ll invest there. That maintains the investment objective of the portfolio while limiting the costs of small trades.

Designing in all those rules was hard, but it was critical.

Hougan: How do the John Hancock ETFs differ from others on the market?

Smart: Dimensional has a long history of using the information in prices to find the differences in expected returns. One of the things we pay a lot of attention to is distinguishing what factors are worth pursuing. There are a lot of factors out there, but not all of them are worth building a portfolio around. We have several criteria to evaluate what we call the “dimensions of expected returns.”

The first is robustness: You want to see that something works not just in a particular study period, but in the past, over different periods, in different parts of the world, and is robust to alternative measures. It should not fall apart if you poke it.

The second is sensibility: Why should a dimension be there? If you can’t answer that, for us, it’s difficult to believe it will continue into the future. You can go back in history and see a lot of different patterns in the data; that’s called “data mining.” Whether that pattern makes any sense helps determine if you can build a portfolio around it.

After that, we’re careful in implementation, asking how these strategies will perform in the real world. We have a long and rich history of taking into consideration the real-life cost of investing. Smart-beta strategies that look good in backtests may fail in real life because the implementation costs are larger than the factor benefits. You have to design for real life.

Hougan: Part of the magic of Dimensional is the education it provides to advisors, which helps these advisors steer clients to hold through thick and thin. How does this carry over into ETFs, and are you worried about the behavioral use of these ETFs?

Smart: Part of the reason Dimensional focuses on investor education is because we understand the impact that cash flows can have on a traditional mutual fund. When cash comes into or out of a fund, the portfolio management team has to react. That trading activity has a cost, and subscriptions and redemptions can generate a lot of turnover, which diminishes the experience of everyone else.

That’s less of an issue in the ETF space primarily because of how creations and redemptions happen. In the ETF space, creations and redemptions happen with baskets of securities. When you’re getting a perfect basket of securities, that has less of an impact on the ETF.

Advisors can help their clients maintain discipline in their asset allocation, which can benefit the advisor, their clients and the portfolios. If the education that both we and John Hancock provide helps advisors, the better off everybody is.

Hougan: There are lots of multifactor ETFs out there. How should investors be evaluating these products?

Smart: There are a few aspects that are important to evaluating any investment.

The first is developing an opinion on the returns hypothesis. Is the returns hypothesis robust and tradable? Does it have more benefit than cost? Is the strategy sensible?

Then, equal attention should be paid to whether the investment can be implemented effectively. Something may look nice on paper, but is it built to take the movement of the market into consideration and to focus on things that are cost-efficient?

And of course, you need to think about where the strategy fits in the overall portfolio and if it meets your returns objective.

Hougan: We look forward to hearing more at Inside Smart Beta.