In an article entitled “The Crowd Within,” The Economist has commented on some recent work that has expanded on the earlier and well-publicized counter-intuitive findings of James Surowiecki, author of “The Wisdom of Crowds” (2005). Surowiecki found that the aggregated guesses of non-experts were often startlingly accurate. The averaged guesses of non-expert crowds were often more accurate than detailed predictions by individual experts. Here’s a more detailed description of Surowiecki’s surprising findings.
That’s where this new research comes in:
That problem solving becomes easier when more minds are put to the task is no more than common sense. But the phenomenon goes further than that. Ask two people to answer a question like “how many windows are there on a London double-decker bus” and average their answers. Their combined guesses will usually be more accurate than if just one person had been asked. Ask a crowd, rather than a pair, and the average is often very close to the truth. The phenomenon was called “the wisdom of crowds” by James Surowiecki, a columnist for the New Yorker who wrote a book about it. Now a pair of psychologists have found an intriguing corollary. They have discovered that two guesses made by the same person at different times are also better than one.
It appears that having a single person wait for awhile and then make a second guess tends to create a situation where that person’s mind is wiped relatively clean of the first guess, so that the second guess can be somewhat independent of the first, such that averaging the two guesses together allows a phenomenon similar to that of having two people make independent guesses.
The above summary of the finding that there can be “intelligent crowds” residing in a single person’s mind reminded me of a thought that that has repeated occurred to me. I have long suspected that religion is driven by social needs, not dogmatic and certainly not intellectual. Why is it, then, that so many scientists don’t feel compelled to follow a religion or to adopt religious beliefs? My speculation is that scientists follow the crowds in their own heads, so that they are immune to the charm of real life external crowds. They thus don’t feel the need to be joiners or to espouse beliefs for the sake of pleasing crowds. Why? Scientists excel at being self-critical. They need to be self-critical or else they will be horribly embarrassed (or even have their careers destroyed) when some other scientist comes along and disproves their favorite theory.
Good scientists have the courage and skills necessary to test their own theories by attacking them inside of their own heads. They brutally challenge their own ideas to an extent that would horrify many people. The best way to get to the best idea is to cultivate lots of voices and perspectives (again, scientists excel at doing this in their own heads). See, for example, my concept of my personal Greek Chorus. The skeptics’ strategy is thus that one shouldn’t muzzle the voices in one’s own head. It’s not wise to adopt “group think” in one’s own mind, to impatiently homogenize one’s own mental narrative. Instead, good thinkers allow the voices in their heads to have free reign, at least until objective findings (especially those based on experiments) silence some or all of the alternative viewpoints. This has been my own version of “The Crowd Within.” When done well, it is an exhausting endeavor, but potentially rewarding. This way of thinking is not for the faint of mind.
On what topics is this analysis relevant? I wouldn't want to ride in a ship or plane whose course was plotted by popular opinion. I wouldn't want to have medical diagnoses based on popular opinion.
In brief, I regularly bet my life that anything that has been worked out by disciplined observers probably fails the mass opinion test.
It seems like a validation of the classic Emperor's Nose fallacy (one version):
Dan: There are limits to the wisdom of crowds, but within those limitations, the crowd is a lot smarter than you would ever anticipate. For more on Surowiecki's surprising findings, check out this Wiki article:
http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds
This is also the basic method used in artificial intelligence for handwriting recognition by neural network simulation. In the handwriting recognition algorithim, each letter is isolated for an image of a signature. The image is then analyzed by 9 subprograms (called nodes) that look for specific similarities to letters, one node may look for descenders, another may assess roundness and so on.
When the 9 have finished, they vote and the result of the voting is used to eliminate the results that have nothing incommon with the majority. The process then repeats with each node slightly biased in favor of a subset of the previous set of letters. until a certain percentage of the nodes are indicating the same letter. This percentage is called the confidence level and it is usually defaulted to 70 percent.
For this to work in artificial intelligence and in groupthink, the initial choice must be made independently by each individual, based on unprejudiced and un biased information. The political process, is one that is determined to introduce an artificial bias and prejudice into mass decisions. This is regardless of what political group are backing the prejudice.
Much in the realm of social psychology is counter-intuitive. Which is exactly what makes it so interesting
I see my misinterpretation: From the context I assumed that "of crowds" meant "by crowds" rather than "about crowds".
We recently had this football tournament (www.uefa.de)and at work we placed bets (not for money, just for fun and some small prizes) on the outcomes of the games. Everybody played alone, except for this group of students, who placed their bets for most of the games right at the beginning as far as it was possible, unlike most of the rest of us who waited for outcomes of the different rounds and then placed their bets for the next round. The students were leading pretty quickly and eventually won.
Coincidence?
Niklaus's post was pretty interesting.
I read the Surowiecki book some years ago and thought the title was quite misleading – crowds have very little wisdom – lots of independent opinions (preferably expert, eg, farmers guessing the weight of the cow) combined can do much better. It was hardly a new or surprising insight – "two heads are better than one" is a saying that has been around for a while. The Economist article is, however, interesting. I sometimes worry when asked for an opinion on something that I may have expressed a different view on it previously, since inconsistency is something to be avoided (flip-flops!), but maybe not. It need not be new evidence that causes us to come to a different view but just a new starting point and maybe we should welcome that diversity of outcomes – certainly sounds like a better approach than I have made up my mind on this so I don't need to think about it again.