Printed and electronic copies are for personal use. Any unauthorized distribution by fax, email or any other means is prohibited and is in violation of copyright. If you are interested in redistribution, reprints or a subscription, please contact us at or +1 (415) 659-9899
Nate Silver: Confidence Kills Predictions

Nate Silver: Confidence Kills Predictions



(Editor's note: IndexUniverse's Inside Indexing conference in Boston was postponed in May due to the Boston marathon bombings. The rescheduled event is this month.)

Perhaps best known for his highly accurate election predictions, statistician Nate Silver is the creator of the blog (now part of the New York Times website) and the author of “The Signal and the Noise: Why So Many Predictions Fail—But Some Don’t.” Journal of Indexes Managing Editor Heather Bell recently spoke with Silver, a keynote speaker at Inside Indexing to be held in Boston June 17-18.


IU: You’re basically the world’s only celebrity statistician. How did you get into that line of work?

Silver: I kind of fell into it. I had a consulting job after college, and I was really bored there. So I left that to play poker and write about baseball, both of which involve a lot of math. I kind of stumbled into it. Then, the election stuff—where people are so starved for substantive or quantitative coverage of elections, as compared to what they get most of the time—took off in 2008, and then again last year, of course.

Part of the job is figuring out the balance between being very rigorous with your work, but also finding ways to have fun with it, and do TV appearances and talks to promote your idea. The book was obviously a labor of love. Books are really, really hard things to write, but it helps to make the case I’m trying to make.

IU: What I’ve taken away from the book is that the vast majority of experts are stunningly bad at making predictions.

Silver: That’s the whole irony, I guess. There are specific studies that find that the more often people go on TV, the worse advice they tend to give. When I talk to groups, I try to preach a certain amount of humility before these big, difficult problems that we face and to not tell people that if they do this and that, then magic will occur.

IU: What do you see as the common theme among bad predictions? What most often leads people astray?
A lot of it is overconfidence. People tend to underestimate what the uncertainty that is intrinsic to a problem actually is. If you have someone estimate what they think a confidence interval is that’s supposed to cover 90 percent of all outcomes, it usually only covers 50 percent. You have upside outcomes and downside outcomes in the market certainly more often than people realize.

There are a variety of reasons for this. Part of it is that we can sometimes get stuck in the recent past and examples that are most familiar to us, kind of what Daniel Kahneman called “the availability heuristic,” where we assume that the current trend will always perpetuate itself, when actually it can be an anomaly or a fluke, or where we always think that the period we’re living through is the “signal,” so to speak. That’s often not true—sometimes you’re living in the outlier period, like when you have a housing bubble period that you haven’t historically had before.

Overconfidence is the core linkage between most of the failures of predictions that we’ve looked at. Obviously, you can look at that in a more technical sense and see where sometimes people are fitting models where they don’t have as much data as they think, but the root of it comes down to a failure to understand that it’s tough to be objective and that we often come at a problem with different biases and perverse incentives—and if we don’t check those, we tend to get ourselves into trouble.



IU: What standards or conditions must be met, in your opinion, for something to be considered “predictable”?

Silver: I tend not to think in terms of black and white absolutes. There are two ways to define “predictable,” I’d say. One is by asking, How well we are able to model the system? The other is more of a cosmic predictability: How intrinsically random is something over the long run?

I look at baseball as an example. Even the best teams only win about two-thirds of their games. Even the best hitters only get on base about 40 percent of the time. In that sense, baseball is highly unpredictable. In another sense though, baseball is very easy to measure relative to a lot of other things. It’s easy to set up models for it, and the statistics are of very high quality. A lot of smart people have worked on the problem. As a result, we are able to measure and quantify the uncertainty pretty accurately. We still can’t predict who’s going to win every game, but we are doing a pretty good job with that. Things are predictable in theory, but our capabilities are not nearly as strong.

Predictability is a tricky question, but I always say we almost always have some notion of what’s going to happen next, but it’s just never a perfect notion. The question is more, Where do you sit along that spectrum?

IU: How would you explain the concept of “margin of error”?

Silver: That has a very specific meaning within the context of political polling. What “margin of error” refers to is the error that occurs because of random sampling. If you sample only 1,000 people in California and 10 million are going to vote, that introduces error. And that type of error—sampling error—is easy to quantify. The “margin of error” represents a 95 percent confidence interval generally. There are different interpretations to that, but in lay terms, it means there’s a 95 percent chance that the true outcome is within that margin of error.

It gets a little trickier because polls are supposed to take a random sample of people, but they’re also making predictions about who’s going to vote and who won’t. That adds some additional error that might not be measured by the margin of error. People can change their minds if you go further out in time. Also, you’re not really getting a random sample anymore. People often have better things to do than answer political polls: Only about 10 percent of people regularly respond to polls, so you’re getting a lot of political junkies, potentially—people who are almost abnormal, in a way. In a sense, it’s a miracle that polling has done as well as it has. You really are taking a small and not particularly random subset of the population and trying to extrapolate it out to what will happen when millions and millions of people vote, some of whom you can’t even find because they don’t answer phone calls.



IU: How do you invest your own money?

Silver: I’m actually a rather dull and conservative investor for the most part—I mostly just invest in index funds or mutual funds with low fees. One thing I know about the market is that it’s a case where a little bit of knowledge can be a dangerous thing and funds do very well when you’re basically piggybacking all the decisions that every other investor is making. For the most part, I’m playing it quite safe; but frankly, I’m looking for things to invest in right now. I think the markets are more likely to be a little hot right now, a little overvalued rather than undervalued in terms of U.S. equities.

One piece of advice that I get from some people is that if you’re going to invest for fun, take 10 percent of your investments at most to play around with. I’ll invest in a few individual companies in a kind of contrarian way. If I hear friends of mine who work at banks trashing a company and I think it’s for the wrong reasons, then I might be a little more likely to put a little money in it. But 90 percent of my money is in very dull and safe “index fund”-types of vehicles.

IU: Do you think that’s the best way for the average investor to go?

Silver: The evidence is pretty compelling that, for the most part, highly managed funds that get a lot in fees aren’t outperforming simpler index funds. I’m not just saying that—it’s a very rich and robust finding supported by literature. There are questions about whether very advanced funds like hedge funds get above-normal returns. I think the scholarly view is that certainly some probably do, and not just as a matter of luck, but whether you have access to them is a tricky case.

Wall Street is pretty sophisticated. That doesn’t mean it can’t have huge blind spots from time to time, but if you’re betting against the market when you have this option to get the average return, you’re being pretty presumptuous. You should have pretty good reasons to be doing that, and if you are going to do that, then I say invest a little bit of your capital like that and look at it as “recreation.” If people are planning for the long term, it’s a wonderful thing that the stock market does tend to go up over the long term, and there’s no reason to get really fancy with it, in my opinion.

IU: Are you looking at any particular areas in the market as potential investments?

Silver: I’m not really looking actively at too many places to invest. I’ve thought about whether some commodities—like gold—might be in a bubble, but that’s really part of the problem: I see more things I’d want to short, frankly, than things I would want to make a long bet on. It’s not as easy to take a short position—obviously, you can, but that requires more effort than just clicking a couple buttons on eTrade, and there are more risks involved as well.



IU: Do you think the ratings agencies have changed in the wake of the financial crisis? Do you think they’re more or less relevant now?

Silver: I can’t speak to whether they’ve changed or not, but I do know their ratings of sovereign debt still seem kind of strange for the most part. One of the odd things, if you look at the upgrades and downgrades of different countries’ credit, is that they’re highly correlated. If a country is downgraded, it’s much more likely to be downgraded again the next time rather than upgraded, which means that they’re accounting for new information very slowly and there’s predictability in their changes.

I tend to think that looking at what markets say in terms of interest rates or credit default instruments is more useful potentially. I’ve found that those tend to lead instead of lag what the ratings agencies do, and the rating agencies’ function is very odd for the most part, and almost political in a sense, where they try to provide cover for certain groups to make investments. But I don’t believe the ratings are market-leading products, and smart investors are looking elsewhere. They are doing their own analysis and not looking at the recommendations of these companies, for the most part.

IU: How big of a role do you believe our hard-wired behaviors play in how we invest?

Silver: That’s part of it. The typical retail investor frankly does things exactly wrong—they tend to buy at the top and sell at the bottom. Theoretically, you make this long-run average return, but a lot of people are buying at the market peaks. For many years, the Gallup Poll has periodically been asking investors whether it’s a good time to invest or not. There’s a strong historical negative correlation between when people think it’s a good time to invest and the five- or 10-year returns on the S&P 500.

I still remember in the late ’90s, when I’d visit friends in New York City, you’d sometimes see CNBC on the televisions in the sports bars. I would think, “This is becoming a little too fashionable.” I’d think that it had become too much of a “cool” thing and that people were investing for the wrong reasons, like to keep up with the Joneses—that can be a little dangerous.

From an evolutionary point of view, it’s very good to be aggressive at detecting patterns. It makes us aware in dangerous “caveman-type” of environments of threats that might be out there. But I don’t think our brains are really engineered to work with as much data as we get nowadays, even as much as we see on a stock market ticker. And that can really lead to some poor instincts. It’s really more about what Kahneman would call “thinking slow”—learning to ignore your instincts. For example, if a stock just lost 20 percent of its value, if there’s not a big underlying change in the structure of the company, that’s potentially a good time to buy more of that stock, but nine out of 10 retail investors would be thinking they want to dump that stock and cut their losses instead.



IU: Has quantitative easing been good for the economy or not?

Silver: I’m not a macroeconomist, so I can’t really offer an educated point of view on that. I do think that in the context of where the economies are around the globe, the U.S. recovery looks OK by comparison. That point is sometimes overlooked. People sometimes overlook the fact that it’s hard for policymakers to get credit for preventing a catastrophe. Things could have been worse, but how do you measure the likelihood of it being worse?

I sometimes make comparisons between that and the fact that there hasn’t been another Sept. 11 attack. That doesn’t mean you support every measure that was taken in the so-called war on terror, but it does mean that the Bush and Obama administrations have achieved an objective in preventing another attack—or maybe they’ve been lucky. Who knows?

Similarly, the fact remains that we had a very severe recession, we still have an output gap and things aren’t all that great by any means right now, but there could have been even more calamitous outcomes. And certainly other parts of the world still face the potential for more calamitous outcomes—Europe being one case

I don’t know exactly what overall grade I’d give to fiscal policymakers, but it’s a middling grade as opposed to a failing one, just because operating under difficult conditions can be tough.

IU: Bias and wishful thinking seem to play a lot into poor predictions in your book. When does this cross over from bad judgment into bad faith?

Silver: That’s a tricky thing, especially when you look at political predictions. Those two things overlap there a lot. The main reason is you have political partisans or political operatives whose objective is basically to manipulate public opinion in some sense. And how well can you judge public opinion if you’re in the act of trying to change the reality? It’s fairly difficult, and I tend to be cynical—with reason—about how smart the Washington beltway consensus is. I think it’s not very good for a lot of reasons.

Acting in good faith is important. It’s sometimes hard to say, “I’m going to cheat a little bit, but I’ll be honest about these other things.” It’s hard to shift back and forth successfully between those two modes. You have to have a cleaner set of objectives for what you want.

Also, in competitive environments, it’s hard to beat the market. If your incentives are perverse at all, then it’s very unlikely you’ll be batting the market average. In poker, which I played for a long time, you lose a lot more money when you’re “on tilt” than you may when you’re playing your regular “A game”—it’s really easy to do stupid things then. The 20 percent of really dumb decisions that we make in life tend to hurt us a lot more than the smartest 20 percent of our decisions tend to help us. Trying to have your cake and eat it too by being a political partisan trying to do neutral analysis is tricky a lot of the time.

IU: Even though the baseball season really just started, who do you like for the World Series?

Silver: I was saying the Angels and the Nationals before the year began. I’m going to stick with that, even though I know the Angels haven’t played particularly well. I don’t get to follow baseball as much as I once did, but that’s what I’ve been telling people. I don’t want to cheat by having 10 different predictions in 10 different places and then pulling the one that was right out after the fact.