How undependable are the experts?

We are in the middle of a huge economic crisis. Should we listen to the experts? Of course we should, because the economy and the financial sector are horrifically complicated. What happens when the experts disagree, however? To which experts should we listen? I took a stab at that question recently, but I remain unconvinced that any of the economics experts can be trusted. Yes, there are people like George Soros who have made a phenomenal amount of money during the crisis, but this makes me wonder whether he (and all of the other recent success stories) are smart or whether they are lucky. Today, Nicholas Kristof (in the NYT) reminds us that many experts (at least political experts) have a terrible track record. His opening sentence: "Ever wonder how financial experts could lead the world over the economic cliff?" He warns us of the “Dr. Fox effect,” named for a "pioneering series of psychology experiments in which an actor was paid to give a meaningless presentation to professional educators." Despite the fact that the lectures consisted of gibberish, they were well received. He mentions a study showing that "clinical psychologists did no better than their secretaries in their diagnoses." He also mentions a study by Philip Tetlock which determined that "The [82,000] predictions of [284] experts were, on average, only a tiny bit better than random guesses — the equivalent of a chimpanzee throwing darts at a board." Those experts who were the most impressive to most people "provided strong, coherent points of view, who saw things in blacks and whites." I'm reminded of Alan Sokal's intentionally nonsensical article that he submitted to the postmodern journal, Social Text. See here for more of the details. BTW, if you want to generate your own postmodern bullshit, use this postmodernist bullshit generator (every time you hit the link, more impressive-sounding bullshit will be assembled automatically into an article). How far astray are we led by "experts"? Consider investment "experts." There are none worse. Entire industries are built on the thoroughly disproved notion that a stock-picker can consistently beat the market. Dan Smolin has made a career of proving that stock-picker experts are thoroughly and demonstrably terrible at what they claim to be. But many of us still run to these financial "experts" to help us pick the "right" stocks. Just think of the hundreds of political military experts who were similarly awful at their recommendations and predictions regarding the invasion of Iraq. They appeared hundreds of time on network TV during the few weeks prior to the invasion, all of them confident in their assessments and advice. Consider, also that fewer than 1% of them took anti-war stances. Consider, also, that many of these "experts" were secretly in positions to financially benefit from an invasion of Iraq. Consider the thousands of religious experts, from coast to coast, who loudly and confidently tell their religious followers that there is a heaven and that they will go there, without the tiniest big of evidence in support. The followers of fundamentalist preachers continue to listen to these guys even when they attack evolutionary biologists, even though these religious leaders have no training in science and no basic understanding of the principles of evolutionary biology. Everyone loves weather forecasters, right? These guys are wrong so incredibly often that no station dares to post their track records for those five-day forecasts they confidently present night after night. The list goes on and on. We insist on listening to the experts, medical experts, beauty experts, psychologists, their track records be damned. That's because they are the best that we've got, no matter how wrong they are how often. The bottom line is that we crave experts because we crave certainty, even where there isn't any. The confirmation bias causes us to rely heavily on experts hawking our own opinions, even when there is no evidence in support, as long as the expert dishes out those opinions with a loud confident voice. And a fancy business suit doesn't hurt either.

Continue ReadingHow undependable are the experts?

How to weed out junk science when discussing climate change.

George Will's recent journalistic malpractice has inspired much discussion by many people concerned about climate change. It's a critically important issue given that 41% of Americans currently think that the threat of global warming is being exaggerated by the media. The intellectual energy runs even deeper than criticism of George Will, though, leading us to the fundamental issue of how journalists and readers can distinguish legitimate science from sham (or politicized) science. The Washington Post recently agreed to publish a precisely-worded response to Will by Christopher Mooney. Here's Mooney's opener:

A recent controversy over claims about climate science by Post op-ed columnist George F. Will raises a critical question: Can we ever know, on any contentious or politicized topic, how to recognize the real conclusions of science and how to distinguish them from scientific-sounding spin or misinformation?

Mooney methodically takes Will to task on point after point. For instance, weather is not the same thing as the climate. The state of the art in 1970s climate science has been superseded by 2007 climate science. You can't determine long-term trends in Arctic ice by comparing ice thickness only on two strategically picked days. The bottom line is not surprising. If you want to do science well you have to do it with precision, measuring repeatedly, crunching the numbers every which way and then drawing your conclusions self-critically. What is not allowed is cherry picking.

Readers and commentators must learn to share some practices with scientists -- following up on sources, taking scientific knowledge seriously rather than cherry-picking misleading bits of information, and applying critical thinking to the weighing of evidence. That, in the end, is all that good science really is. It's also what good journalism and commentary alike must strive to be -- now more than ever.

Mooney has given considerable thought to these topics. His byline indicates that he is the author of "The Republican War on Science" and co-author of the forthcoming "Unscientific America: How Scientific Illiteracy Threatens Our Future." I would supplement Mooney's well-written points, borrowing from our federal courts. They have long been faced with the struggle to determine what is real science and what is junk science, and they have settled on what is now called the "Daubert" test, (named after the case first applying the test, Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993)). The Daubert analysis is applied many times every day in all federal courts (and many state courts) all across America. The problem facing judges is that the parties to law suits often produce experts who express scientific theories and explanations that are never heard outside of courtrooms. This justifiably makes judges suspicious. Is the witness doing "real" science or his he/she doing sham science to further the interests of the party paying his/her bills? The Daubert test asks the judge to serve as gatekeeper, to make sure that only legitimate science sees the light of day in courtrooms. Here are the relevant factors:
  • Does the method involve empirical testing (is the theory or technique falsifiable, refutable, and testable)?
  • Has the method been subjected to peer review and publication?
  • Do we know the error rate of the method and the existence and maintenance of standards concerning its operation?
  • Is the theory and technique generally accepted by a relevant scientific community?
Positive answers to each of these factors suggests that the witness is doing real science. Astrology would fail this test miserably. Applied to climate science, the Daubert test would require that we listen carefully to what the scientists talk about with each other, in person and in their peer-reviewed journals. Daubert would require that we know enough about the techniques of climate science to know how it makes its measurements and conclusions. Daubert would certainly require that we know the difference between the weather and the climate. Applying Daubert is not simply a matter of listening to the scientists. Quite often, the scientists are bought and paid for (e.g., scientists working for tobacco companies and corrupt pharmaceutical companies). Applying Daubert requires taking the time to understand how the science works to solve real-world questions and problems and then taking the time to see that its methodology is being used with rigor in this application. There are no shortcuts, expecially for outsider non-scientists. No shortcuts. No cherry-picking.

Continue ReadingHow to weed out junk science when discussing climate change.

“Extenuating circumstances” for faking drug testing data?

I don't get it. The Wall Street Journal recently reported that

A prominent Massachusetts anesthesiologist allegedly fabricated 21 medical studies that claimed to show benefits from painkillers like Vioxx and Celebrex, according to the hospital where he worked.

This fabrication is not surprising in light of the fact that Vioxx has now been shown to be of highly questionable effectiveness and based on real world use that has arguably caused tens of thousands of deaths--people who had heart attacks because they used Vioxx when they could have, instead, continued to use the extremely safe over-the-counter drug Naproxen. But then comes the good part, a claim by Dr. Rueben's attorney:

"Dr. Reuben deeply regrets that this happened," said the doctor's attorney, Ingrid Martin. "Dr. Reuben cooperated fully with the peer review committee. There were extenuating circumstances that the committee fairly and justly considered." She declined to explain the extenuating circumstances.

There you have it. There were "extenuating circumstances" for faking data in 21 medical studies. I wonder what those "extenuating circumstances" were? The desire to get rich by conniving with a dirty drug company (see the article for the evidence)? Our did those "extenuating circumstances" include the lack of any sense of professional responsibility? Or did those "extenuating circumstances" include sadistic impulses to endanger the lives of thousands of people?

Continue Reading“Extenuating circumstances” for faking drug testing data?

Freedom of Speech as Religion

I think it should go without saying (but of course, nothing does) that anytime someone wants to protect something from "denigration" or other forms of criticism, we should all grab hold of our rights and hang onto them with a death grip. To put this case most eloquently, I offer the following.

Continue ReadingFreedom of Speech as Religion

A Master’s in Creationism

LOL*cough* Excuse me. I nearly choked on a handful of almonds. My created/intelligently designed oesophagus is just a little too close to my trachaea, making it very easy to block the latter with particles of chewed food whilst laughing my arse right off its hinges when I read stuff like this FOX article:

State Rep. Leo Berman (R-Tyler) proposed House Bill 2800 when he learned that The Institute for Creation Research (ICR), a private institution that specializes in the education and research of biblical creationism, was not able to receive a certificate of authority from Texas' Higher Education Coordinating Board to grant Master of Science degrees.
So. "Education" and "research" (grin) of biblical creationism (chortle). OK, well, although I'm no theologian or creation scientist (snort) , allow me to sum up the entire proposed course for you: Education: God did it. Research: it says in the Bible that God did it. Thesis: the Bible says God did it; the Bible is always right because God wrote it; God is always right; the evidence for this is in the Bible which God wrote; my conclusion is that God did it (and is always right, just like the book He wrote). Well done, here's your diploma & whimsical "WWJD" t-shirt that all the kids are wearing. That'll be several thousand dollars. Hmm?

Continue ReadingA Master’s in Creationism