Selection Bias

August 23, 2011

Redefining Credit Risk

An important contributor to performance for any portfolio is the selection of companies for inclusion, no less so for indexes than for active portfolios. Rather than exploring this topic broadly, as many have done, we focus on the differences between two well-known lists of companies: the Fortune 500 and the Standard & Poor’s (S&P) 500.

The Fortune 500 and the S&P 500, launched in 1955 and 1957, respectively, offer similar cap-weighted returns, with a scant 21 bps difference over a 53½-year span. While this seems trivial, the two indexes share most of their largest names, leading to an almost 92 percent overlap, by some measures. If 21 bps of return difference is attributable to as little as 8 percent of our portfolio, then the nonoverlap holdings must have materially different returns. Indeed, they do.

In fact, the closer we look at the nonoverlap portfolios, the more significant the differences seem. Some companies are in the Fortune 500 but not the S&P 500, and other companies are in the S&P 500 but not the Fortune 500. When these two nonoverlap lists are cap weighted, the former list outperforms the latter by 198 bps per year for over 50 years. To be sure, there is substantial volatility around this mean, but this performance gap does achieve statistical significance and invites questions on the basic principle of selecting stocks based on market capitalization.

It is neither our intent to promote the Fortune 500—which does not exist as an index—nor to criticize the S&P 500. The S&P 500 is arguably the most widely used benchmark in the world, for many legitimate reasons. Our intent is simply to show that selection rules make a very significant difference … even for indexes.

A Question Of Selection Bias
Selection bias is an interesting and often-explored topic in the investment industry, and the differences between the Fortune 500 and S&P 500 exemplify its impact. Both lists were intended to be “representative” of the broad market, albeit in different ways. The Fortune 500, which is objectively selected to include the 500 largest U.S.-domiciled companies as determined by the previous year’s revenues, might be viewed as an effort to capture the broad economy, focusing on business success. In contrast, the S&P 500 was launched to capture the performance of the U.S. stock market. The S&P 500 is subjectively selected by a committee with a preference for large-cap, actively traded companies that are of greatest interest to the investing community. It has rules for profitability and domicile (which are infrequently waived for particularly important or popular companies), but it is studiously not formulaic in its stock selection, in order to prevent advance gaming of changes in its component list.

The two lists of companies offer similar cap-weighted returns, with a 21 bps difference over a 53½-year span. Given that both indexes are cap weighted, that modest difference must be attributable to selection bias, or rather, the different selection criteria used to construct the Fortune 500 and the S&P 500.

In fact, the closer we look at the nonoverlap holdings, the more significant the difference seems. The companies that are in the Fortune 500 but not the S&P 500, weighted by Fortune’s revenue metric,1 outperform the companies that are in the S&P 500 but not the Fortune 500, weighted by S&P’s market capitalization metric, by 303 bps per annum over a span of more than 50 years.

If the S&P 500 and the Fortune 500 are equally weighted, which magnifies the importance of the small nonoverlapping names, the 21 bps performance gap widens to 57 bps per annum. To be sure, there is substantial volatility around this mean, but it does achieve statistical significance. And it highlights the importance of selection rules—even in an index.

 

 

Find your next ETF

Reset All