The Key Statistic When Evaluating ETFs

Real-world tracking difference is incredibly important. So why does nobody look at it?

[This article was co-written by Dave Nadig.]

Our goal here at ETF.com is to help ETF investors make better choices.

Core to that mission is helping investors identify which ETFs do the best job of tracking their indexes. Historically, academics have used “tracking error” to measure this.

Tracking error is a measure ripped from academic studies that, used in isolation, can mislead investors and cause poor decision-making. Unfortunately, it’s also the statistic you’re most likely to find on your Bloomberg terminal or fund fact sheet.

At ETF.com, we present an alternative set of statistics we developed years ago to actually measure how well ETFs that track indexes do their jobs: “tracking difference.” It’s more accurate, more intuitive and, ultimately, more useful than traditional measures of tracking error.

What Is Tracking Error?

Most people assume that tracking error measures how well a fund performs compared with its index. If I hold the fund for a period of time, how close will I be to that index’s return when I sell?

But that’s not actually what tracking error measures. Instead, tracking error measures—and here’s the textbook definition—the standard deviation of the daily differences in return between the net asset value (NAV) of a fund and its index. In English, that means that what drives tracking error is the consistency of a fund’s return, not the quality of that return.

Here’s what we mean. Imagine two funds: One trails its index by 5 basis points every day, while the other bounces back and forth, trailing by 5 basis points one day and beating it by 5 basis points the next. Using the standard measure of tracking error, the former fund is perfect: Because it misses the index return by a fixed amount each day, it has precisely zero tracking error.

Back in the world of reality, that “perfect tracking” means that, by the end of the year, the fund is trailing its index by roughly 12 percentage points. Ouch.

By comparison, our 5-basis-points-up, 5-basis-points-down fund has enormous tracking error … but it ends the year with exactly the same return as the index.

There are other perverse results of the way tracking error is calculated:

• Outperformance is treated with the same disdain as underperformance
• Small, one-day data errors—or even data-rounding—create massive impacts
• Accounting discrepancies between the fund and the index can produce shocking “false” results

What’s bad about all this is that these statistics—available widely on Bloomberg and other financial sites—lead investors to make wrong decisions time and time again.

How About A Real-World Approach?

ETF.com takes a more real-life approach to the problem. Instead of using academic statistics, we try to answer the question investors really want to know: What is the average experience of an investor in any given fund compared with its index?

To do this, we measure something we call “tracking difference.” Tracking difference compares the ETF’s NAV return against the index return over rolling one-year periods, looking back two full years. It asks, in a typical one-year period, how much did this ETF trail its index? Your baseline expectation would be that your fund would trail its index by its expense ratio. Anything else is slippage.

The use of longer time periods smooths out the frequent and self-correcting data blips that can distort tracking error statistics, and also makes the number more digestible. If I tell you that a fund trailed its benchmark by 0.50 percent over the past one-year period, and that the worst one-year miss was 0.65 percent, you know what you’re probably going to get.

Let’s see the real-world implications of this in making a few ETF choices. (Note: All of these tables are available for free in our ETF segment reports. Just type “etf.com/[ticker]” into your browser.)

Example 1: Biotech

Consider this data-tracking snapshot of ETFs the U.S. biotech sector.

ETF.com’s data shows that the SPDR S&P Biotech ETF (XBI | A-45) has done fantastically well. A naive assumption would be that XBI would trail its benchmark by its expense ratio of 0.35 percent. However, on the median one-year period, it has actually beaten its index by 0.58 percent. In fact, it’s beaten its index in every single one-year time period we measured. It’s “worst” experience was to “only” beat its index by 0.19 percent.

Compare that with its competitor, the iShares Nasdaq Biotechnology ETF (IBB | A-27), which came much closer to standard expectations: The fund charges an expense ratio of 0.48 percent and trailed its index by 0.42 percent in the average one-year period.

That’s powerful information: While only 0.13 percent separates the funds on an expense-ratio basis, the gap widens to a full 1 percentage point on a tracking difference statistic.

What would academic analysis (or competing data providers) show if you look at the two funds? IBB would show a tracking error of 0.031, and XBI would show a tracking error of 0.091.

Aside from the fact that those are unintuitive numbers (what does 0.031 even mean for my expected returns?!). More to the point, they’re misleading, suggesting that IBB is the hands-down winner at tracking its index.

While statistically correct—remember that consistency trumps outperformance in tracking error analysis—any investment decision based on this analysis would fail to recognize that XBI is actually a better managed fund. Such a decision would also not be mindful that investors in XBI enjoyed better tracking performance over every 12-month period in the past two years.

Example 2: High Yield

You can see this same analysis play out in other asset classes. Consider high-yield U.S. debt.

Our analysis clearly shows that, despite having a substantially higher expense ratio, the iShares iBoxx $ High Yield Corporate Bond ETF (HYG | B-74) has delivered a better user experience. During the median one-year period, it trailed its index by just 0.12 percent, with tight bands between all potential outcomes.

By comparison, the largest competing fund, the SPDR Barclays High Yield Bond ETF (JNK | B-77), trailed its benchmark by 0.72 percent on average … and missed in the one one-year period by 0.99 percent.

Traditional tracking error for these funds is 0.536 for HYG, and 0.188 for JNK, making JNK look like a winner. Once again, the math works—JNK does in fact have lower academic tracking error than HYG, but every long-term investor would make their choice HYG if their only consideration was how well the fund hewed to its index.

Example 3: Emerging Markets

Finally, consider emerging markets, where the two largest funds come from Vanguard and iShares, and the lowest-cost—and lowest median tracking difference—fund coming from Schwab. But let’s focus on the classic question: Which of the giants do you pick? The iShares MSCI Emerging Markets ETF (EEM | B-98) or the Vanguard FTSE Emerging Markets ETF (VWO | C-90)?

ETF.com tracking clearly shows that Vanguard’s VWO is the winner; its lower expense ratio echoed by its significantly lower median tracking difference.

The leading data providers we look at reverse this order, suggesting that EEM was by far the choice in the segment, while VWO was wildly inefficient in managing its index. In fact, no matter how you run the numbers, VWO has a statistical tracking error of 2.704, versus EEM’s 0.208.

The VWO statistics are simply incorrect from a real-world perspective. The problem, in this case, is peculiar.

There are a handful of ETF providers—Vanguard among them—that adjust their end-of-day NAVs using “fair value” standards. Their intention is good: If a fund holds, say, a Japanese stock, that stock stopped trading at the close of day in Tokyo.

By the time the New York Stock Exchange closes at 4 p.m. ET, the Japanese stock hasn’t traded in 13 hours. While most fund companies still use the closing price in Tokyo to calculate their NAV, Vanguard and others guess at where that stock might trade if the Japanese market were open.

Unfortunately, indexes make no such effort, meaning the NAV for VWO and other Vanguard funds is offset with the index. Given how particular tracking error calculations are, it creates wild results.

Not only does tracking difference minimize this problem, but here at ETF.com, we work with issuers like Vanguard to get their nonfair-valued NAVs to allow a like-for-like comparison. We are, we believe, unique in taking the time to do this.

The end result isn’t trivial. On average, EEM missed its index by 0.56 percent a year last year; VWO trailed by just 0.20 percent. Between them, they have $97 billion in assets. Multiply those numbers out and you get a big result.

Conclusion

We’re big math nerds around here. I don’t know any place in the ETF universe with more folks studying for or coaching for the chartered financial analysts tests twice a year. But sometimes, the traditional ways of using math to measure indexing just don’t make sense.

Our goal here at ETF.com isn’t to sit in an ivory tower and spout opinions. It’s to help actually investors make better actual decisions. Understanding real-world tracking difference, we hope, is a great first step.

But admittedly, it is just a first step. Obviously, there are many, many other considerations in picking an ETF. In the above examples, each ETF mentioned tracks a different index. VWO doesn’t include South Korea, EEM does. JNK and HYG have very different liquidity screens for what junk bonds to include, and XBI and IBB track very different parts of the biotech world. You can never get around the knowing-what-you-own step in ETF selection.

But once you’ve made it, it’s nice to know whether or not the ETF is actually delivering on its promise.

At the time of this writing, Matt Hougan had a position in VWO. Dave Nadig had no positions in the securities mentioned. You can reach Matt at [email protected] and Dave at [email protected], or on Twitter @dnadig.