When experts make predictions about economics, science or politics

It's bad enough that we often have to listen to blowhards while we're out and about--I'm referring to people whose are rendering long strings of opinions even though they have no credentials, expertise or curiosity about the facts. Now what do you think when you hear those many experts pontificating about the future?   I'm talking about those many experts the media provides to us, people talking with great confidence about upcoming catastrophes, including the prices of houses or stocks, or the consequences of social unrest (and, what the hell, let's add sports "experts" to the mix). Dan Gardner wondered about this, and he wrote a book titled: "Future Babble:  Why Expert Predictions Are Next to Worthless and You Can Do Better.   I learned about Gardner in a well-written article by Ronald Bailey in Reason, "It's Hard to Make Predictions, Especially About the Future":

In Future Babble, Gardner acknowledges his debt to political scientist Phililp Tetlock, who set up a 20-year experiment in which he enrolled nearly 300 experts in politics. Tetlock then solicited thousands of predictions about the fates of scores of countries and later checked how well they did. Not so well. Tetlock concluded that most of his experts would have been beaten by “a dart-throwing chimpanzee.” Tetlock found that the experts wearing rose-tinted glasses “assigned probabilities of 65 percent to rosy scenarios that materialized only 15 percent of the time.” Doomsters did even worse: “They assigned probabilities of 70 percent to bleak scenarios that materialized only 12 percent of the time.”

The problem with experts was also discussed in a March 2011 issue of Scientific American, "Financial Flimflam:  "Why Economic Experts' Predictions Fail," which offers this finesse to Tetlock's findings:

There was one significant factor in greater prediction success, however, and that was cognitive style: “foxes” who know a little about many things do better than “hedgehogs” who know a lot about one area of expertise. Low scorers, Tetlock wrote, were “thinkers who ‘know one big thing,’ aggressively extend the explanatory reach of that one big thing into new domains, display bristly impatience with those who ‘do not get it,’ and express considerable confidence that they are already pretty proficient forecasters.” High scorers in the study were “thinkers who know many small things (tricks of their trade), are skeptical of grand schemes, see explanation and prediction not as deductive exercises but rather as exercises in flexible ‘ad hocery’ that require stitching together diverse sources of information, and are rather diffident about their own forecasting prowess.”

I suppose the bottom line advice is that you need a psychological profile of an expert before determining whether to believe him or her.   But maybe a nice long impressive track record would be a reasonable substitute.

Continue ReadingWhen experts make predictions about economics, science or politics

How undependable are the experts?

We are in the middle of a huge economic crisis. Should we listen to the experts? Of course we should, because the economy and the financial sector are horrifically complicated. What happens when the experts disagree, however? To which experts should we listen? I took a stab at that question recently, but I remain unconvinced that any of the economics experts can be trusted. Yes, there are people like George Soros who have made a phenomenal amount of money during the crisis, but this makes me wonder whether he (and all of the other recent success stories) are smart or whether they are lucky. Today, Nicholas Kristof (in the NYT) reminds us that many experts (at least political experts) have a terrible track record. His opening sentence: "Ever wonder how financial experts could lead the world over the economic cliff?" He warns us of the “Dr. Fox effect,” named for a "pioneering series of psychology experiments in which an actor was paid to give a meaningless presentation to professional educators." Despite the fact that the lectures consisted of gibberish, they were well received. He mentions a study showing that "clinical psychologists did no better than their secretaries in their diagnoses." He also mentions a study by Philip Tetlock which determined that "The [82,000] predictions of [284] experts were, on average, only a tiny bit better than random guesses — the equivalent of a chimpanzee throwing darts at a board." Those experts who were the most impressive to most people "provided strong, coherent points of view, who saw things in blacks and whites." I'm reminded of Alan Sokal's intentionally nonsensical article that he submitted to the postmodern journal, Social Text. See here for more of the details. BTW, if you want to generate your own postmodern bullshit, use this postmodernist bullshit generator (every time you hit the link, more impressive-sounding bullshit will be assembled automatically into an article). How far astray are we led by "experts"? Consider investment "experts." There are none worse. Entire industries are built on the thoroughly disproved notion that a stock-picker can consistently beat the market. Dan Smolin has made a career of proving that stock-picker experts are thoroughly and demonstrably terrible at what they claim to be. But many of us still run to these financial "experts" to help us pick the "right" stocks. Just think of the hundreds of political military experts who were similarly awful at their recommendations and predictions regarding the invasion of Iraq. They appeared hundreds of time on network TV during the few weeks prior to the invasion, all of them confident in their assessments and advice. Consider, also that fewer than 1% of them took anti-war stances. Consider, also, that many of these "experts" were secretly in positions to financially benefit from an invasion of Iraq. Consider the thousands of religious experts, from coast to coast, who loudly and confidently tell their religious followers that there is a heaven and that they will go there, without the tiniest big of evidence in support. The followers of fundamentalist preachers continue to listen to these guys even when they attack evolutionary biologists, even though these religious leaders have no training in science and no basic understanding of the principles of evolutionary biology. Everyone loves weather forecasters, right? These guys are wrong so incredibly often that no station dares to post their track records for those five-day forecasts they confidently present night after night. The list goes on and on. We insist on listening to the experts, medical experts, beauty experts, psychologists, their track records be damned. That's because they are the best that we've got, no matter how wrong they are how often. The bottom line is that we crave experts because we crave certainty, even where there isn't any. The confirmation bias causes us to rely heavily on experts hawking our own opinions, even when there is no evidence in support, as long as the expert dishes out those opinions with a loud confident voice. And a fancy business suit doesn't hurt either.

Continue ReadingHow undependable are the experts?