Nature recently carried out an experiment on its own initiative supposedly comparing the accuracy of Wikipedia and Encyclopedia Britannica, reported here in the Register. The study concluded that the Encyclopedia Britannica had quite a few errors, nearly as many as Wikipedia. Here’s what’s reported:
Nature magazine has some tough questions to answer after it let its Wikipedia fetish get the better of its responsibilities to reporting science. The Encyclopedia Britannica has published a devastating response to Nature‘s December comparison of Wikipedia and Britannica, and accuses the journal of misrepresenting its own evidence.
Where the evidence didn’t fit, says Britannica, Nature‘s news team just made it up. Britannica has called on the journal to repudiate the report, which was put together by its news team.
Independent experts were sent 50 unattributed articles from both Wikipedia and Britannica, and the journal claimed that Britannica turned up 123 "errors" to Wikipedia’s 162.
But Nature sent only misleading fragments of some Britannica articles to the reviewers, sent extracts of the children’s version and Britannica’s "book of the year" to others, and in one case, simply stitched together bits from different articles and inserted its own material, passing it off as a single Britannica entry.
Nice "Mash-Up" – but bad science.
"Almost everything about the journal’s investigation, from the criteria for identifying inaccuracies to the discrepancy between the article text and its headline, was wrong and misleading," says Britannica.
"Dozens of inaccuracies attributed to the Britannica were not inaccuracies at all, and a number of the articles Nature examined were not even in the Encyclopedia Britannica. The study was so poorly carried out and its findings so error-laden that it was completely without merit."
In one case, for example. Nature‘s peer reviewer was sent only the 350 word introduction to a 6,000 word Britannica article on lipids – which was criticized for containing omissions.
A pattern also emerges which raises questions about the choice of the domain experts picked by Nature‘s journalists.
Several got their facts wrong, and in many other cases, simply offered differences of opinion.
"Dozens of the so-called inaccuracies they attributed to us were nothing of the kind; they were the result of reviewers expressing opinions that differed from ours about what should be included in an encyclopedia article. In these cases Britannica’s coverage was actually sound."
The Encyclopedia Britannica stated: [my emphasis]:
We discovered in Nature’s work a pattern of sloppiness, indifference to basic scholarly standards, and flagrant errors so numerous they completely invalidated the results. We contacted Nature, asking for the original data, calling their attention to several of their errors, and offering to meet with them to review our findings in full, but they declined
Update (SM): Nature has responded, refusing to back down from its original article.
technorati tags: Nature, wikipedia, misconduct, replication
Climate Audit 
32 Comments
Is this an early April Fool’s joke?
=================
No. If John Maddox was dead, he’d be spinning in his grave.
I wonder if the Nature study was peer-reviewed.
Hummm, Nature do a comparision between Wiki and Britannia. They find they’re about the same. You, John, then (because, as we all know, you don’t like Nature) find the response of Britannia and, amazingly(?) we find Britannia think they’re far better than Wiki.
And you just accept Britannia’s view? Is this a serious thread?
BTW, I’ve no doubt Britannia is a very fine encyclopedia.
Rule Britannia, consult Britannica.
=====================
Re: #4
Once again we find evidence that Hearnden never bothers to read underlying articles, as it might threaten his preconceived ideas on insulting me.
What, so what you’re really doing is complimenting Nature rather than trying to trash it…
Re #7
What I’m trying to do is clearly beyond your capacity to judge.
John, do you really have to feed the troll?
I’ve edited out a number of John A’s editorial comments – sorry about that, John. I think that the quotations speak loudly enough without the need of the extra adjectives.
You also edited out the link to the Britannica reply, so I’l have to put that back in.Edit: No I won’t. You’ve edited the article in such a way as it makes no sense.
Nature has posted a response.
Click to access Britannica_response.pdf
It is very interesting to see how Nature handled critical comments in this issue. Instead of either not responding, or just “moving on”, they came back with a cogent, well argued piece. I am not in a position to judge who is right on this issue, but at least we are witness to an engaged discussion on both sides.
Would that it were that way in climate science.
Has Peter Hearnden waved the white flag over my arguments in Re Benchmark in A&W ?
Re: #13
Actually, no I don’t think they (Nature) did. Britannica accused them of:
1. Refusing to send the full replies of the reviewers, or providing them with any extra information
2. In more than one case providing cut-and-paste articles from more than one source article on Britannica
3. Counting questionable omissions in Britannica’s articles as “errors”
I note also that Britannica cite specific errors given by Nature as not errors at all. Nature does not (so far) cite the actual verified errors that they found, nor retract claimed errors which are not errors of fact at all.
The greatest difference is one of source authority: Britannica can name the person and check the validity of its sources, Wikipedia cannot.
I would have thought that it should have mattered that Wikipedia’s citations are often missing or wrong, but Nature did not appear to think that accurate citations are important in encyclopedia articles.
I doubt very much that if the method was flawed, those flaws did not favor either Britannica or Wikipedia. Certainly Nature did not produce any evidence of that assertion.
It also appears to me that Britannica is most concerned with scholarship and accuracy and goes to great lengths to maintain those things. Nature, on the other hand, thinks that evident flaws balance themselves out and appears remarkably unconcerned about them.
Since Nature has chosen this sort of stance, I can only assume that it thinks what it wrote about Britannica’s scholarship to be a fair reflection. I wonder if its readers will feel the same way.
The global warming topics in wiki are heavily censored by William Connolley
Re 13, Bruce, thanks for posting. I thought that Nature’s response was neither “cogent” nor “well-argued”.
In fact, they did not touch on most of the Britannica’s points, preferring to take the low road of not revealing their information. Is this refusal to reveal scientific information an epidemic? Is it catching?
They also cleverly managed to “misunderstand” the Britannica’s points. The Britannica pointed out that Nature drew its quotes, not from the Encyclopedia Britannica, but from the Yearbooks and the Britannica Student Edition, and in one case merely made the quote up..
In response, Nature says “In one instance Britannica alleges that we provided a reviewer with material that was not from the Britannica website. We have checked and are confident that this was not the case.” Which is lovely, but totally meaningless. The Britannica made no claims about whether the material was on their web site, they said that much of the material was not from the Encyclopedia. Nature is playing with words here in a shameful way.
Also, Nature responded to Britannica’s statement that Nature did not check the reviewers claims by saying
However, in direct contradiction to this, Nature also said
In other words, despite their claim, Nature did check the responses of their reviewers, and now they’re lying about it.
Finally, Nature did not respond in even one case to the claims of inaccuracy. Britannica made a very clear case on why each of their cited “errors” were not errors at all. Nature said … nothing.
So no, the Nature response is neither “cogent”, nor is it “well-argued”. It is a specious, pathetic attempt to cover up some very bad research. Unfortunately, these days this seems to be the common stance of both Science and Nature, backing bad research because it fits with the editors’ political beliefs. Disgusting, and sad that two once-great publications have fallen so far.
w.
Re #15 & 17: Concede!
Neither side comes out too well from this, but Nature comes off worse.
Britannica sure ain’t perfect in their articles, and no doubt they have been caught out by several reviewers’ criticisms. Their handling of the issue has also been defensive and apparently, in some cases, disingenuous.
But Nature were asking for it. They reprove Britannica for not contacting them before publishing a riposte, but Nature do not appear to have even notified Britannica of the existence of their investigation before they published its results, which directly undermine Britannica’s core marketing claim of usefulness and reliability.
And what sort of an approach is this from a supposedly scientific journal?:
1. “In a small number of cases, to ensure comparable lengths, we provided reviewers with chosen excerpts, not full articles; this was done with entries from both Encyclopaedia Britannica and Wikipedia.”
Excerpts cannot provide valid comparisons, especially if you are going to accept as allowable criticism that an excerpt is incomplete. The length of an article is an important part of editorial judgement, and a material factor in considering the reliability of various sources of information.
2. “Britannica objects that Nature did not check the assertions of its reviewers. This is true; nor did we claim to.”
Expletive deleted! Doesn’t Nature even care if the “errors” they report are actually wrong?
Bottom line: I’d be sceptical of any encyclopaedia articles, especially those that don’t seem to have been revised for a while, but I wouldn’t trust anything in the Nature “News” section.
Re #18
“Aw come on, that was never five minutes” 😉
Re #20: I know when I am beat!!! LOL
Today’s Wall Street Journal Online has an article about the dustup between Nature and Encyclopaedia Britannica.
Wikipedia-TrekCentralStation
Follow up heavy on the editorial
http://www.theregister.co.uk/2006/03/25/britannica_wikipedia_nature/
I wonder why Nature seems to be determined to destroy its reputation. Their conclusion that Britannica was in error regarding Bethe is petty. The discussion given in the editorial of #24 regarding Wkipedia’s article on Seigenthaler is pretty bizarre as well.
Roger Bell
#19 quoting Nature: “This is true; nor did we claim to.”
How can we trust the commitment to scholarship of a group that blithely splits an infinitive writing in defense of its intellectual standard?
To what is the world coming? This is something up with which we should not put! (thank-you Winston) 🙂
I’ve got a copy of Britannica-the-program somewhere, but I no longer bother to install it when I (re-)install Windows, because Wikipedia is more useful. Of course, "useful" is not the same as reliable. Before taking a Wikipedia article seriously, you should always glance at its discussion page to see if there is any controversy or "edit warring" going on; if so, do not rely on the article (but do see if it provides links to reliable sources).
Note: a few months ago I would have said that I could tell in advance which articles would be controversial. Boy, was I wrong! There are lots more disputes and cranks out there than I had ever imagined. Hence the "always glance at its discussion page" advice. (Aside: if you want to see why William Connolley was made an administrator, skim the discussion page for Cold Fusion. Warning: very long, extremely tedious.)
There are lots of good articles on Wikipedia, but also quite a lot of rubbish. Full disclosure: I’ve been editing Wikipedia for a few months now. In fact, I’ve edited the articles on Steve McIntyre (new draft under preparation), Ross McKitrick and Climate Audit.
One of the main things about this Britannica/Wikipedia dispute is the different organizational models: a top-down approach in which the authors are chosen by a centralized body, vs a bottum-up approach which tries to attract volunteers then guide and teach them. (Related issues: commercial vs open-source software, mainstream media vs bloggers, prominent climate scientists vs people who understand about correlation coefficients 😉 … see Glenn Reynold‘s new book, An Army of Davids.) Wikipedia is a faith-based project, the faith being in the power of ordinary people to work productively together to produce a reliable encyclopedia. In my studies of Wikipedia, I have seen instances of this working “¢’¬? but there are other cases (probably more numerous) where it doesn’t work.
(I wish this blog had a preview button.)
John replies: It does. Look below the post button and you’ll see a live preview of your post
Re #22: that article ends with Jimbo Wales (the leader of Wikipedia) saying that “he was glad Nature chose to compare science-related themes ‘because on history and the social sciences, we’re much weaker.’ In other areas — including computer science and the history of ‘Star Trek,’ he says, Wikipedia is ‘way better.'”. He’s exactly right.
Also, Andrew Orlowski is (in)famously skeptical about open-source software and other “bottom-up” ventures, including Wikipedia, so his article should be treated with even more skepticism than the rest of The Register.
(BTW, I messed up the link to Glenn Reynolds in my previous comment.)
RE #27 – John’s Edit:
John, I’m using Firefox 1.5 and don’t have a preview. It worked before the last crash, and hasn’t come back. I’ve got the same problem on a different machine, too. (Both XP Pro).
Same problem with me. And that’s true using both Netscape and IE.
Looking at the source there’s nothing between the div id=”commentPreview” and close div. So whatever script the pages are being produced from isn’t producing the code for displaying a comment preview.
No, that’s right John A, no preview, (I’m running Firefox 1.5.0.1) and there used to be a ‘live’ preview before the crash a few days ago. It did however get extremely slow when typing a long comment. Towards the end it was taking up to several seconds for each letter to be echoed to the preview box and typing became extremely painful.
re #31,
You shouldn’t be typing in long messages. There are too many things that can go wrong and it’s too difficult to make corrections. Produce your message on a word processer and then cut and paste. Short messages, like the one I’m doing now, who cares? If you lose it you can redo it. But a long message lost will haunt you!