Nature and Britannica: Round 2

Nature has responded to Encyclopedia Britannica’s accusation of "sloppiness, indifference to basic scholarly standards, and flagrant errors" with a response like this:

In our issue of 15 December 2005 we published a news article that compared the
Internet offerings of Encyclopaedia Britannica and Wikipedia on scientific topics
(“Internet encyclopaedias go head to head”, Nature 438 (7070) p900-901; Encyclopaedia Britannica has now posted
a lengthy response to this article on its website, accusing Nature of misrepresentation,
sloppiness and indifference to scholarly standards, and calling on us to retract our
article. We reject those accusations, and are confident our comparison was fair.

Our original article made clear the basis of our comparison. Conducted by our news
staff, it consisted of asking independent scholars to review 50 pairs of articles from
the Wikipedia and Encyclopaedia Britannica websites. The reviewers were not
informed which of their pair of articles came from which source; the subjects of the
articles were chosen in advance to represent a wide range of scientific disciplines. Our
staff compiled lists of factual errors, omissions and misleading statements that the
reviewers pointed to (we had 42 usable responses) and tallied up the total number for
each encyclopaedia: 123 for Britannica, 162 for Wikipedia. Turning the reviewers’
comments into numerical scores did require a modicum of judgement, which was
applied diligently and fairly.

Nature won’t reveal the full reviewers reports nor the method by which they made this judgement. They go on:

Britannica’s general objections to this article were first made to us in private some
months ago, at which point we willingly sent them every comment by a reviewer that
served as the basis for our assessing something as an inaccuracy. While we were quite
willing to discuss the issues, the company failed to provide specific details of its
complaints when we asked for them in order to be able to assess its allegations. We
did not receive any further correspondence until the publication of its open letter on
22 March 2006. It is regrettable that Britannica chose to make its objections public
without first informing us of them and giving us a chance to respond.

Meanwhile at The Register, in a new article called Unnatural acts at Nature , Andrew Orlowski actually asks Britannica about Nature’s response:

Britannica says Nature prevaricated and still won’t release the referees’ reports in full.

"We asked for the data so that anyone could replicate the results," Britannica spokesman Tom Panelas told us today.

"At first Nature said they couldn’t release the data because they had promised their referees anonymity. We said that’s fine. Then they said it’s too much trouble."

Nine days after Nature’s news story and rallying editorial, the magazine published a file of "supplementary information" which merely listed the errors referees had found. Britannica had to figure out what the sources were. That information still hasn’t been released.

Orlowski did note that Nature’s editor was a big fan of Wikipedia’s anarchic approach to scholarship and encouraged scientists to get involved in this new bold experiment. However those bold scientists might have to do battle with undesirables:

Editing pages is not always straightforward, as some users may disagree with changes. In politically sensitive areas such as climate change, researchers have had to do battle with sceptics pushing an editorial line that is out of kilter with mainstream scientific thinking. But this usually requires no more than a little patience. Wikipedia’s users are generally interested in the reasoning behind proposed changes to articles. Backing up a claim with a peer-reviewed reference, for example, makes a world of difference.

Yes. Quite.

technorati tags: , , ,


  1. Jack Lacton
    Posted Mar 25, 2006 at 5:42 PM | Permalink

    At the very least Nature could release the methodology it used to conduct the comparison. I’d be especially interested to see where they sourced their referees. Surely there’s a general view in the scientific community that Britannica upholds the highest standards and has the appropriate processes to deal with any inaccuracies in order to maintain their reputation? Equally, nobody would agree that a consensus approach is going to guarantee a better outcome. In Wiki’s case, the fact that anybody can become a reviewer makes it more likely that opinion and religious/scientific/political position will become the driver of input and change than pure facts. In this case I would suspect that at least some of the referees are also Wiki reviewers, which would invalidate the process completely. I look forward to Nature’s guarantee that none of the referees were Wiki contributors.

  2. ET SidViscous
    Posted Mar 25, 2006 at 7:05 PM | Permalink

    Hey John. Wheres that thing you said you were going to do on/with/about William Connelley on wikidpedia.

    You said we’d all know when you did it.

  3. jae
    Posted Mar 25, 2006 at 7:47 PM | Permalink

    IMO, the Wikipeida “experiment” is laughable. it amounts to a concensus and political determination of truth. It is Dark Ages stuff. It is progressive liberalism at its bestworst. How can anyone subscribe to this BS?

  4. ET SidViscous
    Posted Mar 25, 2006 at 7:57 PM | Permalink

    TO be fair Wikidpedia has it’s uses. For some things (particuarly unquestionable things, like chmistry and so forth) it is very usefull. I’ve gone to it for a variety of obscure reasons, and recieved the information I needed.

    But for any politically charged issue, while you can get decent information, you have to ignore the thinly veiled editorial comment. In the end it’s written by the most tenacious contributors.

    It’s only real advantage over a written encyclopedia (besides being free) is that it is quickly updated and easily accesible.

    It’s intersting to note that my home town has one of the most extensive entry for my state, while the town itself is fairly small and innocuous, and there are much larger and interesting towns in the state. In fact our entry is still larger than the state capital.

  5. Max
    Posted Mar 26, 2006 at 2:50 AM | Permalink

    This is truly ridicioulous. If Nature’s Editor prefers Wikipedia, than he should have written so, but do up a fake test. This is especially sad, because Nature loses major credibility with such nonesense. Every Professor at my University has told us over years, that WIKIPEDIA IS NO SOURCE FOR FACTUAL SCIENTIFIC MATTERS. It might be the biggest ressource for trivial matters, but it is not a source used in scientific works, because there are more errors on one page than in any expert literature. It is fine to look up simpsons episodes in wikipedia, or to inform yourself about some history events or a specific famous date, but it is by no means a work to be used in scientific debate.
    Especially in the Science department I have seen some very strange articles full of factual errors (not only the math, but also the citations).
    So, why would a reputable magazine ridicule itself by posting such a test that is at best as “scientific” as many wikipedia entries.

  6. jae
    Posted Mar 26, 2006 at 6:42 AM | Permalink

    It is really surprising and intriguing to see a scientific journal get involved in something like this. What is their purpose?

  7. John A
    Posted Mar 26, 2006 at 7:44 AM | Permalink

    Re #2

    All in good time, young padwan, all in good time.

    I have a freelance researcher gathering the information as we speak.

  8. TCO
    Posted Mar 26, 2006 at 8:23 AM | Permalink

    I think Wiki has it’s uses and Britanica has its flaws, but it’s pretty evident that a poor test was done by Nature and that they won’t come clean on it.

  9. John A
    Posted Mar 26, 2006 at 8:53 AM | Permalink

    Re #8

    You’re not wrong there. TCO. Wikipedia does have some excellent articles, but on even slightly controversial topics, it gets rapidly poorer.

    *This comment is a stub. You can help Climate Audit by adding to it*

  10. kim
    Posted Mar 26, 2006 at 9:21 AM | Permalink

    On another blog I used to joke about the Nursing Home Sewing Circle, a dozen octogeneriennes parallel processing a millenium of experience.

  11. Posted Mar 26, 2006 at 9:46 PM | Permalink

    Wikipedia is extraordinarily and surprisingly effective. Encyclopedia Britannica is a representative of the old school that has had many advantages – but if you can make thousands of people to work on an encyclopedia for free and if you find the right balanced rules, Encyclopedia Britannica just can’t compete with you. I find it obvious that the large collaboration projects based on state-of-the-art technology, such as Wikipedia, will be increasingly taking a lead. Even if Britannica’s statement that they’re more solid and balanced than Wikipedia were true today, it is all but guaranteed that it won’t be the case in a couple of months or years.

  12. Steve McIntyre
    Posted Mar 26, 2006 at 9:56 PM | Permalink

    There’s a certain irony in Nature’s position here. Enc Brit has formal peer review, just like Nature. If Nature is right, then peer review as practiced by Enc Brit contributes little to the merit of the article relative to a Wiki procedure. But what if the same were true of peer review as practiced by Nature?

  13. Chris Chittleborough
    Posted Mar 27, 2006 at 11:50 AM | Permalink

    I’m a little less optimistic than Prof Motl; see my my comment (and the following one) on the earlier thread.

  14. John A
    Posted Mar 27, 2006 at 12:24 PM | Permalink

    I’m very surprised that Luboà…⟠is taken in by Wikipedia since Wikipedia represents the very opposite of scholarship and the scientific method. Instead of history being written by people known and written about, we have what amounts to a wholesale assault on the past by whoever is the most persistent crank or political propagandist.

    I think it goes without saying that wikipedia actually presents us with a model of knowledge rooted in anarchism and mob rule. Truth on wikipedia is decided by the mob, science by fashion and popularity and history rewritten by revisionists without a scintilla of demonstrated scholarship nor any oversight.

    What Wikipedia represents to the sum of human knowledge is what the Czech regime after the “Prague Spring” was to political freedom.

  15. Jim Erlandson
    Posted Mar 27, 2006 at 1:09 PM | Permalink

    Wikipedia is like hitchhiking — its free, relies on strangers and usually you get where you’re going without a problem. Other times … well … you just never know.

    I wonder if Nature will permit Wikipedia citations.

  16. frank borger
    Posted Mar 27, 2006 at 1:27 PM | Permalink

    As a test, I looked up something I’d previously been very involved with and could be considered an expert on, the Therac25 radiation therapy accelerator accidents that smoked a couple of patients.

    The article in wikipedia was so filled with butchered physics, wrong references, etc that an annotated listing of the problems would have been three times as long as the article itself. I’ve never seen a “scientific” article with so many errors in 40 odd years.

    Until wiki gets some editors, it’s a room full of monkeys and typewriters.

  17. TCO
    Posted Mar 27, 2006 at 1:46 PM | Permalink


  18. J. Sperry
    Posted Mar 29, 2006 at 2:59 PM | Permalink

    I’m one of those Wiki editors, because if I see something that’s wrong, I try to fix it (if I can) rather than complain about it. What you will find about Wikipedia is that the serious editors stick around while the vandals and “monkeys” are fleeting, so the “roses” tends to outweigh the “garbage.” (I’m speaking here about the content of the meaningful pages. Wikipedia is full of useless pages that you would only find if you specifically look for them.)

    The serious editors have “watchlists” so that they can easily see what happens on the pages with which they are most familiar or interested. Reverting harmful changes is easy. There’s a permanent history of *everything* that goes on so you can find exactly where an article gets flawed.

    Regarding the perceived bias in Wikipedia, while W. Connolley does get opinionated on the discussion pages (which is the intent, of course), I find him to be mostly reasonable when editing the main articles (in Wiki terms, using neutral point of view), with my main gripe being the use of blogs as references. Just today, he restored a comment somewhat critical of the Kyoto Protocol.

  19. Jim Erlandson
    Posted Apr 28, 2006 at 8:34 AM | Permalink

    Late post. Some interesting thoughts from Nicholas Carr roughly in agreement with what J. Sperry says above.

    The quality of any entry in Wikipedia, for instance, is ultimately determined not by how many people work on it but by how many talented people work on it. An entry written by a single expert will be better than an entry written by a hundred fools. When you look deeply into Wikipedia, beyond the shiny surface of “community,” you see that the encyclopedia is actually as much, or more, a product of conflict than of collaboration: It’s an endless struggle by a few talented contributors to clean up the mess left by the numbskull horde.

    He includes some numbers to support his claims.

%d bloggers like this: