Lewandowsky has stated that an online survey by an opponent was “useless” “without the authors demonstrating the integrity of their data” and that their study “should not have been published without the authors demonstrating the integrity of their data—I doubt that they could”. Words that apply even more forcibly to his own study. I guess it all depends on whose ox is being gored.
On Sep 27, 2010, about a month after the Lewandowsky survey was posted at several anti-skeptic blogs, John Cook posted the following at the private SkS forum entitled “Excerpt from Steve [Lewandowsky]’s email”. At the time, Hans von Storch and Dennis Bray had just carried out an online survey. Lewandowsky complained that there was “no way to check or verify the integrity of the data” and therefore the data was probably “useless”. Lewandowsky said that the study should not have been published “without the authors demonstrating the integrity of their data”:
I’m sure Steve won’t mind if I excerpt from the email he sent me about von Storch’s paper:
Got the paper some time ago and I have corresponded with Bray. He seems like a nice guy but the survey data are apparently compromised by the login and password information having been circulated on a denialist mailing list. I have a copy of that email (via Deltoid).Bray also violated all internet survey methodological standards by not recording dates, times, and IP numbers of respondents (I know this from him personally). He thus has no way to check or verify the integrity of his data. In other words, the data are possibly (probably?) useless, although the published paper seems to include more data than the previous unpublished report which was entirely compromised as just stated.
Now, all that said, the results are not particularly distressing from our perspective, and he correctly identifies that there is a large segment of the scientific community who think that the IPCC understated the problem.
Overall, though, this study should not have been published without the authors demonstrating the integrity of their data—I doubt that they could.
Lewandowsky himself then commented at the SkS forum on Sep 27 entitled “Background info on survey methodology“, favorably citing a critique at Deltoid of methodology in a similar earlier survey, results of which Lewandowsky believed to have been included:
Hi all, see here for background on the survey problems: http://scienceblogs.com/deltoid/2005/05/bray.php
I believe that at least a subset of the data reported in the published paper relies on the data in the unpublished survey critiqued by Deltoid.By the way, a Swedish journalist, Jens Ergon, has done a better controlled survey of Swedish cllimate scientists and their publications and found the usual 97% agreement on AGW. This was reported on Swedish TV and I have corresponded with him.
Graham Wayne then commented that the “problems of data integrity” put him off writing about the topic:
Well, the problems with the data integrity do throw something of a spanner in the works. If I were to write this up, I don’t see how I could be less than candid about the problems, which would in turn sufficiently dilute the value of the research. Climate contrarians will jump on the flaws and hammer them to the exclusion of all else (surprised WUWT didn’t do this already), so I’m rather put off writing about it.
Anyone else got a view on the practicality of focusing on this?
The (2005) Deltoid post that Lewandowsky had cited is here, which discussed the following survey result – a result that appears to contradict the “97%” of all climate scientists that is commonly cited:
Lambert commented as follows:
Is global warming skepticism amongst climate scientists as widespread as this survey indicates? To answer this we need to look at how the sampling was conducted:
The 2003 survey was conducted as an on-line survey. The existence of the survey was posted in the Bulletin of the American Meteorological Society, the Climlist server, and was sent to institutional lists in Germany Denmark and the U.K. As an effort to prevent general access to the survey, the survey was password protected. The password was contained in the informative message distributed according to the above.
However, the information about the survey was reposted (mail list membership required to read link) to the climatesceptics mail list by Timo Hameranta on Sep 20 2003: (my emphasis)
the survey (below) is directed to those involved in the natural sciences related to climate change and not, for example, those involved in policy analysis or economic issues.
I suggest that you participate by completing the questionnaire (instructions to participate below).
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
L�hett�j�: CLIMLIST Climatology Distribution List
[mailto:CLIMLIST@…] Puolesta CLIMLIST
L�hetetty: 19. syyskuuta 2003 18:12
Vastaanottaja: CLIMLIST@…
Aihe: Survey of Climate Scientists
CLIMLIST Mailing Number 03-09-24
Origin: “Dr. Dennis Bray”Due to the nature of the distribution of electronic surveys some recipients of this message might have received the same previously. If that is the case, my apologies. Please do not submit the survey twice, although, once would be much appreciated. The survey is directed to those involved in the natural sciences related to climate change and not, for example, those involved in policy analysis or economic issues. Your discretion in choosing to participate in the survey on this basis would be greatly appreciated, as would your cooperation in making the survey a success. If you know of colleagues not contained on climlist but involved in the climate sciences, it would be appreciated if you could bring this survey to their attention with the suggestion that they too might like to participate by completing the questionnaire. Simply forwarding this message is likely the most convenient method.
… If you do choose to participate, the survey can be reached by opening your web browser and going to the following link:
http://w3g.gkss.de/G/Mitarbeiter/bray.html/
When the page opens click the link to “survey of climate scientists” Here you will be asked for a username and password.
For username enter “respondent” (without quotation marks)
For password enter “ccsurvey” (again without the quotation marks).
The survey is password protected as an effort to limit the respondents to those involved in the climate sciences. There is also the option to print the survey from a PDF file and submit though regular postal services. Electronic submissions do not transmit your email address and consequently anonymity is ensured. Thank you in advance for your cooperation.
Since the survey was anonymous, there is no way to ensure that only climate scientists participated and no way to prevent people from submitting the survey multiple times. Furthermore, the survey was distributed on the climatesceptics list which has over 200 members, almost all of them strongly skeptical about global warming. Since the total number of participants was just 557, this could serious skew the results. I don’t believe that the results of this survey are representative of the views of climate scientists.
Lambert then cited a response from Bray at realclimate in which Bray stated:
It would be possible however, should one wish, to submit a duplicate return in the 2003 survey. 2003 surveys were checked for identical response patterns and none existed. Of course this does not mean that a single person could not fill out two surveys using different responses.
Lambert criticized this as follows:
The only thing Bray did to check for multiple responses by the same person was to see if there were any responses that were exact duplicates
As I interpret Lewandowsky’s methodology, Lewandowsky’s check was whether consecutive responses from the same IP address were identical (but this is only an interpretation of poorly described methodology.)
Lambert summarized his criticism as follows:
While no study is perfect, this study is so imperfect as to be useless. Since it was posted on the climatesceptics list, the sample is not representative. As a social scientist Bray should know this.
It is obvious that all the criticisms leveled by Lewandowsky and Lambert against the von Storch and Bray survey apply even more strongly to his own survey. Applying Lewandowsky’s own standards, his own survey is “useless” unless he can demonstrate “data integrity”, which he can’t.
Instead of demonstrating data integrity, in Lewandowsky’s response today (his first non-juvenile response), Lewandowsky argues, in effect, that data integrity doesn’t matter – a familiar enough reaction in past paleoclimate disputes.
Lewandowsky, although he was quick to fabricate a jibe at me about a supposed pastrami sandwich, did not respond to or cite my post on this matter, though he did cite Tom Curtis’ more limited critique. Nor did Lewandowsky examine the crosscut of data that I had used. Instead, Lewandowsky carried out a different crosscut of the data, arguing that he got similar results with this different crosscut. I placed source code for my analysis online so any critics could readily interpret my conclusions and understand points at issue. Unfortunately Lewandowsky has not dne so, making the process of replication and analysis less efficient than it ought to be. I’ll try to take a look at Lewandowsky’s most recent post over the next few days.
however, one point is obvious: Lewandowsky’s defence is framed in terms of “outliers” as opposed to “data integrity”. It seems evident that, using Lewandowsky’s own words (about the Bray-con Storch study):
this study should not have been published without the authors demonstrating the integrity of their data—I doubt that they could
Climate Audit 
82 Comments
All sorts of ideas for similar research, but done the other way round spring to mind.
Follow the same protocol to show whatever you want.
Then get it published in the same journal, or if they refuse for peer review etc, then its shown as a sham.
Could one of Lewandowsky’s ancestors have been the inspiration for the Wile E. Coyote cartoon character?
“…survey of Swedish cllimate scientists and their publications and found the usual 97% agreement on AGW…”
The ‘usual’ 97%?
Really?
Any credibility his argument might have had is kind of flushed down the pan with that comment alone. We expected 97% and that’s exactly what we got, just like (post)normal.
Lewandowsky co author (on “The Debunker Handbook” project) and blog sponsor-manager-support John Cook has shown how he handles “data integrity” matters wrt numerous blog comments…. Just delete, revise, and re write history long after the fact:
http://nigguraths.wordpress.com/2011/10/10/skepticalscience-rewriting-history/
Since Lewandowsky’s publicly funded university sponsored STW blog exists as an appendage of SkS (and has been deleting and snipping many critical comments lately) it is appropriate to ask Lewandowsky to explain how his data integrity research practices and blog moderation practices compare to those of his collaborator John Cook. I do not find Cook’s explanations remotely credible.
So, I guess we can rule out incompetence as an excuse on the Lew paper.
No, I think there is ample evidence of incompetence shown. What I’m curious about is if there is research misconduct as well as incompetence.
There are many things hard to believe in this post, even for a hardened observer of the climate scene, but the idea that Lewandowsky posted something non-juvenile in response to criticism, after so long, goes too far. Relieved to hear then about a fabricated jibe about a supposed pastrami sandwich, though I remain unclear exactly how to parse that. Is ‘supposed pastrami’ a substitute made from soya?
The exalted level of academic discourse required to smear me has seldom been more intimidating for this outsider.
Steve – there may be a problem with the last chart still, “Conspiracy Adherence” – see my recent comment on the code at “Anatomy of the Lewandowsky Scam”. Mentioning it here because the comment is in moderation (for some reason), and so could disappear.
Even with 100% data integrity, only 4 responses said moon landing was fake, so how can he base his title on that? This most extreme “conspiracy” really paints sceptics as nutters, since everyone knows the moon landing took place, so of course that is why he chose it. The journal should have at least noticed this aspect.
Once again, pretending one’s critics don’t exist. Attention lew: you’ve been audited!
Please don’t assume for a minute that Lewandowsky’s sensitivity test is a test of “data integrity”. It tests against very grotesque scamming, but not against a scammer who was guarding against an elementary screen against too many conspiracies.
In the restricted subset, I calculate 5 CYMoon believers, 4 of whom were warmists. In his signature 9/11 conspiracy, the proportion of warmist conspiracy theorists is noticeably higher.
Steve, I think your totals are wrong, unless I’m missing something.
He is right
Should read
# total 1000 168 100 732
That looks like raw response count minus the 60 something he tossed at the beginning of his analysis, right?
That table destroys his pet theory. There was of course no real reason for anyone to believe that AGW convinced are any less prone to conspiracy theories than the unconvinced. IMO (speaking as someone more left than right) it was more likely to be the other way round and so it proves.
So it seems the raw data seems not to be that controversial but Lew’s conclusions are apparently contradictory to what the data tells him.
Given that the very title of Lewandowsky et al (2012) is based upon alleged significance of of the fantasy conspiracy about faked Moon landings for “skeptics” about climate science, that should discredit the paper right there. Not only is Lewandowsky overtly seeking to equate dissent from climate “consensus” with demented conspiracy-mongering, he is picking the most easily falsifiable conspiracy claim when it (faked moon landings) is the least supported in his own data. It’s not clear it has ANY support in his own data, but in any case the “therefore” of the title is blatantly unsupportable: there is no possible causal connection between a belief of one (probably fake) respondent to an online survey and Lewandowsky’s generalized “therefore (climate) science is a hoax” attributed to skeptics generally.
The title of Lewandowsky’s paper is clearly “inflammatory” (and unjustifiable) — therefore, by the moderation rules of Lewandowsky’s own STW blog, this title should be snipped as “inflammatory” whenever it appears in a post at STW (I am quoting it here not to be inflammatory, but simply to quote what I am discussing):
Utterly unacceptable as the title of a scientific paper when the paper’s own data provide no basis for attributing [A} to skeptics generally and no possible causal connection between [A] and {B} to justify the “therefore”.
Craig tell that to the journal editor, before it gets published! Rather than here. Feedback link to the journal, is in comments here, in the previous blog post.
The title is the problem. It does not fit with the data. Just a new soundbite for all the activists. Based on 3 ‘sceptic’ responses, that were probably gamed by SKS,or Deltoid, or Tamino readers
From Lewandowsky:
While his overall conclusions may remain the same, the title of the paper certainly cannot remain the same if he discards the loony responses.
Hoist by his own petard seems to aptly fit Lewandowsky’s situation.
Lewandowsky in Sep 2010:
No variation across nation then. It’s 97%, not one more or less. The usual, as they say in climate statistics across the globe. Some would wish to quibble about how agreement on AGW was defined but I say the man’s a comic genius. It’s all in the deadpan delivery.
That Swedish poll was just as flawed as the original 97% poll. Q’s almost the same, the TV web site closed the comments fast but not fast enough to to hide the storm of comments about the flawed poll. It’s just in the same bullshit Lew/Oreske division of science.
But do they all have to come to 97%? Surely, with all the best possible flawed questions and massaging of the data, for some country somewhere it came to 96%? Or 98%? That’s why, without looking into the original Swedish, I found Lewandowsky’s comment so laughable. It’s like you bring another country to the attention to the transnational climate statistics lab, they look at you and say “the usual?”, you nod and sure enough, within days, 97% of climate scientists agree with the consensus there too.
As Sherlock Holmes said it was the dog that didn’t bark that was the vital clue. Such lack of noise in this crucial stat across the globe (if we are to believe those that continually parrot it in the MSM) is truly a wonder of the modern world. Are we close to discovery of another fundamental constant, rivalling the Planck constant or c in e = m * c ^ 2 ?
Reblogged this on Climate Ponderings.
On Sep 27, 2010, about a month after the Lewandowsky survey was posted at several anti-skeptic blogs
~~~~~~~~~~~~~~~~~~~~~~~
Shouldn’t that be 2012?
No. The survey was done in 2010. It then took them two years hard work to create that piece of trash they call a paper. Can you say Peter Principle?
By his own definition of data integrity, quoted above, Dr. Lewandowsky’s data is not compromised. Or am I missing something?
Don’t get me wrong, his collection methodology and analysis are flawed. There are experts well distributed across the spectrum of AGW opinion which have made that point. Rarely, if ever, have we seen such consensus.
But I suspect he’ll dance around this accusation.
Did I miss it, but was the continuation of Lewandowsky’s critique that:
posting a climate scientist opinion (consensus) survey to climate skeptics sites skewed the results
hence
posting a climate skeptics opinion (conspiracy supporter) survey to climate warmist sites skewed the results
discussed?
No, you see it is ok to post a survey about sceptics on true believers (TB) sites because TB are pure as driven snow and would never spoof a survey, whereas sceptics are oil-conspiracy funded loonies who have no integrity so the earlier survey by Von Storch was compromised (as if everyone who got Timo’s email did the survey, I sure didn’t and I was on his email list at the time).
I incline to the view that things have got a bit milder since the middle 19th century. A strong argument against my view is the endless stream of falsehoods produced by the Global Warmmongers. If it really has got milder, why do they choose to lie, and lie, and lie?
Steve, you could do a post showing the percentages of respondents who ‘believe’ crazy stuff and agree with Professor Lewandowsky’s view on climate change.
64% of the survey respondents who believe in the Roswell conspiracy theory also believe Professor Lewandowsky’s theory of catastrophic climate change…
How effective would it be to data integrity to require an email address and an active return confirmation to validate each response?
What about the persistent belief in the “Big Oil-funded-anti-climate-conspiracy,” or the belief that skeptical bloggers are in the pay of the Koch brothers?
If you want a lot of angry, conspiracy-laden (and foul-mouthed)* rants, just mention the Koch’s.
*Yes, I know “foul-mouthed” doesn’t properly apply to the written word, but I stand by my words.
Im also confused about the totals..
If one wanted to skew a survey of this type, all one has to do is log into several different computers.
I don’t know how many computers you guys can log into, but where I work, there are something like 35,000 that I could log into.. if I wanted to.
and @ David Hagen.
My isp account allows 10 different email accounts, and I have at least 3 email addresses at work. So the issue still exists.
Basically, internet surveys are a MEANINGLESS JOKE because they can be easily skewed by a specific group working together (‘GetUp’ in Australia do it on a regular basis), or even by a single person if they choose to.
“Basically, internet surveys are a MEANINGLESS JOKE because they can be easily skewed by a specific group working together (‘GetUp’ in Australia do it on a regular basis), or even by a single person if they choose to.”
It’s even worse than that. It doesn’t take much skill to write a bot that could post as many responses as desired from spoofed IP addresses and wipe out any human input whatsoever.
Ignore SkyDragons vs Skeptics.
Instead, simply eliminate anyone who believes in the New World Order conspiracy and you will find that skeptics beliefs regarding other conspiracies are almost identical to warmists beliefs regarding the other conspiracy theories. Thanks for the script to make that check easy.
This makes complete sense because the New World Order conspiracy theory and AGW conspiracy are inextricably linked (ie conspiracy is that AGW is an invention to promote a NWO).
All this paper does is show that NWO proponents reject AGW completely (duh) and that they also tend to believe in other conspiracies (duh). Eliminate them and there is nothing left to distinguish skeptics from warmists.
Re: Charlie Z (Sep 12 16:23), What is fascinating is the source of all the rumors… Feel free to do your own search, wrapping the phrase in quotes if you prefer.
http://www.un.org/ga/president/62/statements/carnegiecouncil101207.shtml
Reflecting on what may be the reason that prevents us from establishing a new world order – whether political or economic in nature — I believe it is because people in today’s world—in the world of ICTs, of information communication technologies, of interdependence of globalization — are frightened when they hear the word “order.” They may be tired at attempts to impose an order. I believe people would like to have something else: normal transparent relations amongst themselves in a world that is very interdependent and made transparent by the ICTs. In today’s world you can communicate with every person you would like to; you just need an email address and a computer. Nobody can prevent you from doing that. No order can prevent that.
It is a phrase much favored in their documents. I took the test at WUWT and was most startled when I saw the phrase as I have read many UN documents where the phrase occurred. I do not believe in anything but “like minded people”.
PS. I think it is highly inadvisable for skeptic site to ever host links to these sorts of surveys. It would be so easy for Lewy to talk to Anna Rose to talk to GetUp, who do an email out to ‘trusted’ members saying what answers they want and you immediately have a few thousand scattered (or scatterbrained)people answering exactly as the author wants them to answer.
And if they are pointed to a link is on a skeptic web site, Lewy can then show that link.
Be very wary of the Trojan Survey. !!!
After reading this, I went to Jo Nova’s site to get her latest on the Lew and get a screen that says “Account Suspended”?
Duke, Yep, unfortunately Jo’s site gets hit and taken down regularly.
It will come back up once people down here get out of bed and fix it !!
Its about 5am in Perth !! be patient 😉
Her site gets hacked a lot. It should be up in a few hours.
… and probably by ‘GetUp’ operators.
Andy … as Steve, Thomas Fuller, Tom Curtis at SKS and others have shown, it is certainly possible to identify fraudulent or “gamed” responses by simply reviewing the data.
And on the one hand Lewandowsky is somewhat correct – it a large sample and handful of responses ARE nothing but noise – ie: if you have 1,000 samples, it would take a pretty large number of fraudulent responses to move the data much if at all.
Anytime you have a large enough group of fraudulent submissions like that – with responses enough different to sway the data, they would be fairly easily recognized.
On the other hand – Lewinsky wants you to focus on that relatively factual statement – that a few fake returns do not skew the overall conclusion, which he demostartes by removing the suspect data and showing the big picture result is unchanged.
He does that so you don’t look closely at the real issue – that those same few fraudulent responses can and do dramatically change the conclusion on a per question basis. In this example, removing those same couple fraudulent responses regarding the “Moon Landing” question eliminate the basis for his already specious conclusion altogether.
Carefully watch the pea and you’ll learn the hucksters trick – and the magicians secret … misdirection … “HEY! – look over THERE!”
Steve, Lewandowsky threw out all but the first response from any given IP adress. He did not check to see if the results duplicated each other. That makes his check far more rigorous than that by Von Stork. If anything it is too rigorous, being more likely to generate false negatives than false positives.
Further, your tailored screen for scamming is nothing of the sort. On the contrary, it merely asserts on the basis of your intuition what proportion of a certain kind of response you expect, and assert as probably suspect a proportion of responses of that type in excess of your quota. No peer reviewed journal in social sciences should (I suspect, would) treat that as acceptable practice. Such a method is far too easily gamed by the researcher.
An appropriate analogy to your method in the physical sciences would be CRU disgregarding a random proportion of all thermometer records showing cooling in excess of that predicted by models.
Do not get me wrong. Your analysis is interesting, and is relevant to the interpretation of the results, but it is not a viable screen for scamming.
Even the two most suspect responses should not be simply disregarded. It is my opinion that Lewandowsky should have mentioned them, and why they are dubious – decided to keep them in or disregard them with reasons explained, and then shown the results with them both included and excluded. Alternatively if excluding them does not change correlations by more than error, simply stating that would be sufficient. Further, I believe this not just because they are outliers, but because they show a very improbable strong universal acceptance of conspiracy theories, and come from a group with low representation in the sample. Had the come from “acceptors” of AGW (mean CC response greater than 3.33) with its very large number of respondent; or the “undecided” (mean CC response between 1.67 and 3.33) which has a much broader distribution of responses than either of the extremes, it is not clear that they would have been statistically aberrant enough to require special treatment.
Steve: I’m not suggesting that there are simple ways of “fixing” this data set or that discarding certain outliers is a magic bullet. Only that such analyses clearly indicate the presence of fake responses. Lewandowsky’s problem arises from his initial ridiculous idea that a survey of die-hard anti-skeptic blogs was an rational way of carrying out a survey. His observation about the need for authors to be able to guarantee “data integrity” and, if they cannot do so, the article should not be published – seems irrefutable to me. DO you agree?
ANd why do you attribute me as proposing a “tailored screen for scamming”? I didnt suggest any such thing.
Mr. Curtis, I believe best practice is to eliminate all data from responses coming from the same IP address. Keeping the first one is similar to finding 2 $20 bills and hoping that one of them is real.
That is to say, finding 2 $20 bills with the same serial number and hoping that the first one is real.
More like finding 2 $40 bills and hoping that the first one is real.
Steve, no social science survey can “guarantee ‘data integrity'”. That is for the simple reason that respondents are not automata. They have diverse, and distinct motives from those desired by the researcher. Consequently their responses are liable to be biased from “genuine and straightforward” responses in a variety of ways, and for a variety of reasons, ranging from maintaining a self image, wanting to present a particular image of a particular demographic (including gilding the lily on your own responses to make your own group look good, which is likely to be far more frequent than straight out scamming the survey), trying to conform to expectations of what the respondent thinks the researcher wants to find; and no doubt a host of others as well. It is not possible by statistical means to work back and determine which responses are affected by which bias except with very carefully prepared surveys devoted to studying just that bias. Nor is it normally possible to distinguish between gamed or otherwise biased responses and simple statistical aberrations.
What you can expect from social science surveys is that they take reasonable measures to ensure data integrity; and for the most part Lewandowsky has.
Your analysis is, as I said, interesting, but it does not make a significant case for large scale rorting of the responses. The analysis you made is easily explained by:
1) A higher proportion of skydragon slayers (or near equivalents) among “skeptics” on pro-science blogs than on “skeptic” blogs;
2) Significant differences in interpretation and weighting of responses between those with firm opinions and those without firm opinions on AGW;
3) A combination of the two.
And of course, it is also clear that some other reason may explain it that neither you nor I have thought of.
Tom: in my opinion, the proportion (21%) of skeptics and skydragons in a survey drawn mainly from Deltoid and Tamino readers is implausibly high. I realize that reasonable people can disagree on this, but I really doubt it. (I assume that it is mainly from these two blogs, as there is no evidence that Skeptical Science posted a link to the survey.)
For what it’s worth, the proportion of warmists in the first tranche (1554) of WUWT survey response was negligible and, of these, there were a number of opposite fakes: people pretending to be wacko-conspiratorial warmists.
Re: Tom Curtis (Sep 12 17:40),
Tom …
A few questions …
1. What is your opinion regarding the fact that for a survey which they clearly indicated was intended to review skeptic responses …
… that they used only data from non-sceptic/pro-AGW sites, and for the most part strongly pro-AGW sites where skeptics are often ridiculed? How can the data be taken remotely seriously?
And;
2. As Steve identified in his prior post the paper claims they deleted “duplicate” responses from any IP address:
… while Lewandowsky’s researcher, Hanich, in a response to Pielke on this subject stated – offering a cite as support:
… that they chose “not to block multiple replies” as recommended by Whitead 2007.
Steve noted the significant difference between blocking “multiple” vs. “duplicate” replies.
3. Then there is the “hide the pea” antics of Lewandowsky today … where he asserts that even if you eliminate the two suspected fraudulent responses it doesn’t change the overall conclusion, that they are essentially noise with a larger sample … but avoids the glaring issue that when you look at the data on a question level, the removal of these two suspected fraudulent responses eliminates all of the tiny shred of support he had for his ridiculous, headline seeking, construct ‘Skeptics are conspiracy theorists about Moon Landings’ …
Lewandowsky is not a scientist and is not doing science here. He is a committed and virulent crusader for, and promoter of, his “cause” … which he tries to further by the illusion of “science”
A. Scott, my opinion is that various “skeptic” blog owners have shot themselves in the foot by failing to advertise the survey. Had they done so, any attempt to rort the survey, and any unrepresentative bias of “skeptics” frequenting pro-science blogs would have been overwhelmed by the large number of genuine responses from frequenters of “skeptic” sites.
What is more, I consider the failure to post, and the more recent attempts by Lucia to teach people how to scam scientific surveys indicative of an anti-science attitude. It indicates that the individuals involved do not want credible research on the differences in attitudes between consensus supporting and opposing people to be analyzed and understood. It reflects more poorly on the individuals in question then does any matter of interpretation or analysis by Lewandowsky, excepting only the title of the paper (which is outrageous).
Steve: If Lewandowsky wanted to analyse readers of skeptic blogs, he should have ensured that he had participation from a major skeptic blog. The survey at anti-skeptic blogs was an invitation to scamming – one which the readers of Skeptical Science appear to have participated in. You and your readers should be ashamed of yourselves.
Lewandowsky’s decision to use data that includes fake responses is his own responsiblity. He appears determined to compound the problem by recklessly carrying on even after the existence of the fake data has been widely publicized.
Lewandowsky seems to think that the way for critics to respond to his use of fake data is to submit responses in academic literature. I disagree. Academic codes of conduct prohibit reckless use of fake data. It’s Lewandowsky’s responsiblity to comply with academic codes of conduct. Of course, as shown in his defence of Gleick, Lewandowsky’s attitude to scientific codes of conduct is about as anti-science as it gets.
I don’t understand why you are dissipating your own reputation defending this guy.
Well, with the same situation in hand I come to the opposite conclusion. Most skeptic blog owners either didn’t notice the e-mail in their overfull boxes or thought little of that mail. Maybe some smelled a rat?
Lucia anti science? What’s next? Lewandowsky’s research says much more about warmists practices than about those of climate skeptics. I can barely differentiate between Lewandowsky and a political lobbyist reading his language, style and tactics.
I rather suspect that you now dim your own criticism of Lewandowsky under peer (or higher) pressure. John Cook looks like The Man here: <a href="http://wattsupwiththat.com/2012/09/12/the-cook-lewandowsky-social-internet-link/"The Cook-Lewandowsky Social-Internet Link
And this means what? That it is skeptic blog owners fault that the findings of this survey is questionable?
If there is a flaw or a weakness in a method isn’t the process of science supposed to expose it? Flush it out so that that researchers can go back to the drawing board?
In fact it looks more that your critisism of Lewandowsky article title was a false flag operation meant to confuse/ distract scrutiny of SkS dubious involvement in this unreliable survey. It failed. You have not shot yourself in the foot but somewhere else, more fatal.
AntonyIndia: that’s quite the conspiracy theory: Tom Curtis pretending to criticize the Lewandowsky study at Skeptical Science merely to distract scrutiny away from Skeptical Science. At Skeptical Science we play a lot of three-dimensional chess and we always think six moves ahead. Pro tip: never play knife/paper/scissors with John Cook for money unless you want to finance his blog.
Seriously, and for what it’s worth, as a Skeptical Science author myself, I also have some concerns about the Lewandowsky paper: the title is silly; there are some conspiracy theories that may appeal to people who lean Green that could have been included; and I think that there are some real data quality issues, as raised by Tom and by Steve McIntyre, that may, once corrected, could change the results a little.
Initially, I thought that it was also perhaps unfair to use such a broad brush to tar lukewarmers as being anti-science and prone to conspiracy-theorizing. I have to say that, on the other hand, some of the more, er, speculative comments made on this blog and WUWT do seem to corroborate some of Lewandowsky’s predictions, perhaps better than his study did.
I think you need to bold your entire response there.
Re: Tom Curtis (Sep 12 21:58),
Tom Curtis … as I stated at “The Lew” 😉 …
It is entirely reasonable to expect, that even if a concerted effort had been made to include a comparable number of skeptic sites in the survey, they might have been wary of the offer, considering the authors history.
I suspect we might agree that perhaps is why an associate made the 5 contacts, by all appearances without mention of the author.
It should have been apparent to the authors that a diligent effort would be needed to secure the support of the skeptic blogs, but no such outreach appears to have been attempted.
Yet despite the known lack of skeptic participation they went ahead. They collected data intended to be reviewed to find connections between skepticism and conspiratorial ideation – to explore motivations of skeptics toward rejection of science – yet knew all the data was being obtained thru predominantly anti-skeptic sites.
We know there was nothing remotely resembling a concerted effort made to elicit the participation of skeptic sites. There was not truly any significant effort to obtain a response at all – to make sure they even received the email and understood what it was about – let alone a concerted effort.
They also made no effort to contact the one site we all know would have been the single best place to obtain skeptic participation … WUWT. It is highly likely IMO Anthony would have posted it.
And we’ve shown WUWT’s readers will participate. They energetically responded to the Re-Created Lewandowsky “Attitudes Towards Science” Survey. A single guest post, by a relatively unknown 😉 WUWT reader, led to an amazing large, prompt, global response.
I did not see Lucia “teaching” anyone anything. She simply showed how simple ordinary efforts, that we all pretty much know anyway, can easily circumvent duplicate IP blocking.
Addressing duplicates and suspect/fraudulent responses, as you yourself have well demonstrated, is handled best thru analysis of the responses.
I agree that Lewandowsky’s actions are silly and counterproductive. And you’ll not see me defending him.
I’m one of those guys that when someone sells, ‘if you don’t like his work then do it yourself’ … takes that challenge seriously. And does it 🙂
Tom Curtis (Sep 12, 2012 at 9:58 PM): “my opinion is that various “skeptic” blog owners have shot themselves in the foot by failing to advertise the survey.”
Well, of the the five invitations, three likely discarded it immediately as possible spam. Can’t say there’s any shooting going on there. Dr. Pielke Jr. — as you hint with the quotation marks, hardly a skeptic — had a thoughtful discussion with Hanich, concluded that the survey had serious flaws, and declined to post. In what way does that constitute “shooting himself in the foot”?
“What is more, I consider the failure to post…indicative of an anti-science attitude.” To me, discarding a post from an unknown sender is indicative only of a well-placed concern for phishing and the like. And the Pielke correspondence showed a positive attitude towards the reliability of results obtained. Perhaps if the study design evinced a more thoughtful approach towards the reliability of results, Pielke would have posted the link.
And I disagree that Lucia’s post was an attempt “to teach people how to scam scientific surveys”. It showed one fairly simple way in which it would be possible to game the survey. There was discussion at the non-“skeptic” blogs about gaming. Whether particular method described by Lucia was used or not, I couldn’t say. But neither can the authors.
I have to say, your prior comments have been of a much higher quality.
But isn’t also bothersome that in order for Lewandowsky have any justification to undertake this study, he must cite other authors who say skeptic climate scientists are corrupt, and each of those authors in turn relies on a single unsupportable source to make that claim?
Demo #2: http://wattsupwiththat.com/2012/09/11/the-other-problem-with-the-lewandowsky-paper-and-similar-skeptic-motivation-analysis-core-premise-off-the-rails-about-fossil-fuel-industry-corruption-accusation/#comment-1076717
Demo#3: http://wattsupwiththat.com/2012/09/11/the-other-problem-with-the-lewandowsky-paper-and-similar-skeptic-motivation-analysis-core-premise-off-the-rails-about-fossil-fuel-industry-corruption-accusation/#comment-1076808
And the main one: http://wattsupwiththat.com/2012/09/11/the-other-problem-with-the-lewandowsky-paper-and-similar-skeptic-motivation-analysis-core-premise-off-the-rails-about-fossil-fuel-industry-corruption-accusation/
Steve: Russell, I read your interesting exegesis. It’s quite a daisy chain of repetition of assertions with no empirical research.
Steve, exactly the point I’ve been trying to make since early 2010. Back in Oct 2009, I asked SEJ board member Robert McClure questions about the treatment of skeptic scientists, and one of his assertions at his blog here http://www.invw.org/2009/10/sej-didnt-single-out-journo-who-questioned-al-gore/ (his 10/20/2009 – 14:36 comment) claimed “Gelbspan was only the first of many to document payments by industry to a small group of scientists…”. When I asked who the other ‘documenters’ were, McClure never gave an answer, so I’ve been trying to prove to myself that there were others….. and I’ve failed. Everything in my huge pile of notes indicate Gelbspan was the only source for the ‘big coal & oil funding accusation’ AFTER late 1995, and prior to that we have a more murky time span from mid’91 – to ’95 where these efforts look like they were clumsily handled by an indeterminate group of people that includes Alan Miller/Curtis Moore, Andy Rowell, some unidentified person(s) at the Sierra Club, and one or more Al Gore Senate staffers. All of which entirely surrounds the “reposition global warming as theory rather than fact” coal association memo phrase that sure doesn’t look like the sinister top-down industry directive it has long been portrayed as.
The kicker is, I can’t find that anybody EVER checked the veracity of it, they just repeated it ad infinitum. Not a speck of empirical research on their part as far as the eye can see….
thomaswfuller2, most repeat survey responses from internet surveys are from people trying to find how their “score” would have differed had they answered slightly differently. Hence the advise from the article cited by Lewandowsky to only delete subsequent responses from the same IP address if they occur very soon after the first response, or if they are too similar to previous results from the same IP address. Lewandowsky was in fact more vigourous in excluding potentially faked responses, not less.
Tom Curtis, nonetheless, excluding honest responses from multiple IP addresses does not prejudice the survey findings. Including responses from someone who intends to game the survey does. They entered with the intent to game. I don’t want their answers to my questions. My apologies to those left out. I don’t think people frequently do it to see their scores–submissions from multiple IPs don’t vary between surveys with results displayed and those without.
In commercial surveys it is often an issue of earning additional incentives by repeat submission. With non-paying surveys it is an attempt to load the dice.
Sorry again–I’m getting dyslexic at the end of the day. Multiple submissions from the same IP don’t vary between surveys with results displayed and those without.
thomaswfuller2, I quote Steve quoting from Gosling et al, 2004:
What is more, I have observed exactly that sort of behaviour from one of my daughters. To complicate things, I will often take non-scientific surveys with scores displayed multiple times from multiple different perspectives as a means to determine the assumptions of the scoring model, and to see whether I consider those assumptions reasonable. My first attempt will always be entirely sincere and spontaneous.
On top of that, you have examples of multiple people using the same IP because they are:
In the same workplace;
In the same household;
Using library computers;
(Possibly) using per fee internet access from an internet cafe, airport or shopping center.
Any filtering mechanism is a compromise between excluding gamed responses and “ballot” stuffing, and not excluding genuine responses. Excluding genuine responses will bias the survey just as much as not excluding gamed responses or “ballot” stuffing. Therefore a compromise is needed. Of necessity, any such compromise will be, to a certain degree, arbitrary. Lewandowsky’s option represents a reasonable balance so long as due care is taken not to over interpret results.
Tom Curtis, I don’t think excluding valid respondents biases a survey at all. You’ve got lots of valid respondents. There are many more with valid opinions that you’ll never hear from because they don’t take the survey.
It costs more to get valid respondents. That’s the reason for inventing excuses to include marginal or invalid responses in commercial surveys. There’s no excuse in non-commercial research.
As a counter example, there are also respondents who want to change their answers after submitting the survey. Their second attempt is a more valid expression of their honest opinions. You would throw out their second attempt and keep their first.
When in doubt, toss ’em out.
Tom (F), you raise an excellent point for analysis of the Lewandowsky fake data. It is essential to know which retained responses were from respondents who submitted multiple responses and whether the results are sensitive to the exclusion of ALL these responses, as you suggest.
Tom Curtis, it would be a good idea if you encouraged Lewandowsky to provide a data set including ALL the data, with an additional column for IP address. The IP addresses could be somewhat anonymized but the form should show which responses came from multiple respondents.
I’ve also received information from the University of Western Australia that the questionnaire was distributed on campus. It’s hard to understand what legitimate purpose was served by this. I’ll post separately on this point.
This is particularly true when one is talking about very small number of conspiracy responses – which are, as we see, sensitive to fake data.
BTW in the reduced data set for which Lewandowsky says the changes “dont matter”, there is precisely one non-warmist respondent who purported to believe in his signature Moon Landing conspiracy. And that response is almost certainly a fake, merely one that wasn’t caught up in his filter of “outliers”.
Nor are Lewandowsky’s assertions about the MLK conspiracy or the 9/11 conspiracy true either. Both conspiracies are held more than proportionally by warmists. As is the Moon Landing “conspiracy”. Not only does Lewandowsky use fake data, but some of his claims are unsupported by the data. Lewandowsky’s assertion:
as it relates to the MLK conspiracy, the 9/11 conspiracy and the Moon Landing is untrue.
I’ve done an attempt to replicate his factor analysis. Many of his results are not replicable.
It’s late here and I’ll post more on this tomorrow.
“When in doubt, toss ‘em out.”
Unless it is data from trees, ice, sediments, etc. right?
Steve Mc:
Wow. I refuse to speculate on any illegitimate purpose, because that’s not what we do on climate blogs, is it folks. But thanks for that, UWA.
A.Scott.. “it is certainly possible to identify fraudulent or “gamed” responses by simply reviewing the data. ”
I’m sure that if they had any competence at all, they could rig a survey so it was pretty darn difficult to pick the “gamed” response from a scattered population sample, such as GetUp.
This whole business is juvenile. What line of reasoning/evidence does Lew have that could possibly demonstrate any rational tie between conspiracy theory belief and any lack of proper reasoning/understanding on Climate Change? There is no rational reason why someone could not be completely wrong on everything else in life, but be correct when it comes to the nature of climate change. Human diversity allows for infinite flavors of ignorance and genius.
The whole point of Lews study is to discredit people, and this man wants to call himself a scientist? He’s turned research into a vile popularity contest and he did it willfully. He should be condemned on that basis alone. The very notion of attempting to defend the methods on a research project whose abstract would have been accurate to simply state “We have performed an Ad Hominem Attack on those we disagree with” is nauseating.
Has anyone, in real life, ever met any person who believes the moon landing was faked?
You do the survey on a warmist site and only a tiny fraction of the answers come from supposed skeptics and 6 out of 10 of those skeptic responses indicate belief that the moon landing was faked.
Had you done this survey on a skeptic site you would have gotten a huge number of answers from skeptics — and does anybody truly believe that 6 out of 10 of those responders would indicate that they believed the moon landing was faked?
But this guy Lew implies just that. He implies that 6 out of 10 skeptics believe that the moon landing was faked.
There are a huge number of people who are skeptics and 6 out of 10 of them believe the moon landing was faked? Has anybody in real life ever met any person who belives the moon landing was faked? I have not but apparently according to this guy Lew they are crawling all over the place. BUT I HAVE NEVER MET ONE!
Or am I missing something?
How could any journal not realize that the extrapalations he makes from his data are bat shhit crazy!!!!! Has any editor of the journal that accepted this paper ever run into anyone — ANYONE — who believes the moon landing was faked? Even the smallest application of common sense would have told them to reject this paper out of hand.
God! The mentally numb have taken over the world.
Eugene WR Gallun
I’m starting to think Tom Curtis is a conspiracy theorist:
You have to be some sort of nutjob to think Lucia pointing out how proxy servers could have been used to bypass a “security measure” in a survey means she is attempting to “teach people how to scam scientific surveys.”
Are we absolutely sure all the 1100 responses came fom the ‘8’ blogs and does not include the UWA circulation.. Just for clarification.
Some of those 8 blogs have very low traffic. And Skeptical science cliams it was hosted and then deleted. from the ‘8’ blogs, and do not include the UWA circulation.
Additionally how many responses came from Skeptical Science, as it was claimed that it was ran here, then deletd.
As fars as I can see the paper does not explicitly say the results came from only, lim
Nks to the 8 blogs identities that were released. No conspiracy, just aclarification requird for the avoidance of doubt
Some of you may be interested in the poignant saga of Alene Composta. I’ll begin with the end of the story where Lewandowsky becomes involved.
On the March 21, 2011 Alene posted on her blog the email exchange she had with Lewandowsky. Even though she was unknown to him, he felt her ideology and experiences were similar to his. After hearing her sad tale he immediately commiserated with her and offered some advice concerning the abusive comments she had been receiving from those who don’t accept climate change as a serious problem. Evidently he felt their kinship was so strong he even wrote back later to pass on more advice from John Cook.
Here is their email exchange. http://verdanthopes.blogspot.com/2011/03/tomorrow-belongs-to-us.html
Sad to say, her sons stopped by two days later and found this poor woman deceased. It appears they are plucky lads as they took the time to inform the public of the circumstances before signing off her blog forever. Quite thoughtful lads. Here is their description for those who can handle such details. http://verdanthopes.blogspot.com/
Hoping to avoid having too many links send this comment to spam, the beginning timeline which includes several notable Aussies is below.
Tim Blair has done an excellent job composing the beginning portion of the timeline, so I’ll leave that to him.
http://blogs.news.com.au/dailytelegraph/timblair/index.php/dailytelegraph/comments/composed_an_alene_composta_timeline/
Some of Alene’s other posts may bring a chuckle or a tear depending on your mood.
So, the Lewandowsky paper is a hoax?
They’re taking the Mickey out of Sceptics?
If so, it has been a delight to read the well argued, the courteous and constructive comments of so many of the Sceptics who have contributed to the “discussion”.
Bob K
That was hilarious reading, and I particularly liked how Lewandowsky professed that he abusive blogcomments when making professional presentations (to reinforce the strength of his position and show how deplorable those who disagree must be, I presume):
It’s quite common to try to use the gutter scrapings from the opposite side to imply some moral highgrounds, even attempting to estrapolate this to one’s own position. But this is not only a sloppy thinking and a logical fallacy. It exposes how incapable some (quite a few) people are to applying their won ‘arguments’ symmetrically.
If one reads the comments at say Deltoid, RealClimate and Tamino, all who ‘moderate’ heavily to ‘improve discussion quality’, and read what is left, particularly the amount and ‘quality’ at the lower end, and remember that this actively has passed moderaters approval ..
.. if you do this and apply Lewandowsky’s own argument (above)to it, it would be devastating.
But I guess these people rarly reflect over what they are actually saying and arguing, what their arguments actualy say and imply (if taken seriously, and generally applicable). No wonder som many contrived and nonsensical claims come from that corner ..
I see some of Tim’s links no longer work sometimes due to age and a few due to embarrassment. Kind of takes some of the flavor away. It was a well played satire of a leftist blog.
It shows that when being blinded by ideology causes a failure to verify an unknown source you can leave yourself looking foolish. Sort of like Crikey, The Drum Unleashed and Lewandowsky.
Is it just me, but I can not help but wonder about some things.
1. Could there be a link between asking UWA staff to do the survey, and going of the way to allow multiple responses from a single IP address (as previously reported on ClimateAudit)?
I mean wasn\’t the explanation given of why the methodology chosen allows multiple responses from a single IP address, that in circumstances like multiple staff responding from \”our lab\” (or similar words), this IP address flexibility would be neede.
2. ClimateAudit chooses the title of the \”Unreported Results\”. Are we sure they are unreported? Is it possible that some of the UWA staff responses may be included in the survey results?
Is there any statement from Lewandowksy, or other evidence, to exclude this possibility?
the proportion (21%) of skeptics and skydragons in a survey drawn mainly from Deltoid and Tamino readers is implausibly high.
This observation seems to me much more important than the rather marginal question of duplicated IPs.
And it has an advantage as it can be quantified.
I have made a brief overview at Tamino’s and Deltoids over the last few months.
In every post I identified what :
a) Is a skeptical opinion
b) Comes from a regular reader (as opposed to a 1 time comment)
If one admits that regular posting is correlated to regular attendance and that the correlation is not different for a skeptic and for a warmist what is a reasonable hypothesis, then the conclusion is that the number of skeptics who are regular readers of Tamino and Deltoid is far below 10%.
My conservative estimate would be 5%.
As for 1 time visitors the probability that a skeptic would happen to do his first and last visit of this blog precisely during the time the survey was up is negligibly low.
All the above can be quantified and uncertainties computed even if I did not do it because I won’t waste too much of my time on Lawandowski.
And based on the estimators above, up to 16% of respondents are necessarily fake skeptics under which is a subcategory “double fakers” of fake skeptics who gave fake answers on conspiracy.
Personnaly I would be interested to have a table which gives the number of respondents per source.
Clearly if 60% come from Deltoid and 20% from the university labs, then we have a problem with the Moon Landing Houston 😉
The publication “Psychological Science” should read and consider this article from “The Scientist” :”Bring On the Transparency Index: Grading journals on how well they share information with readers will help deliver accountability to an industry that often lacks it.” http://the-scientist.com/2012/08/01/bring-on-the-transparency-index/
Lewandowsky is the perfect example of non-transparency.
5 Trackbacks
[…] LewWorld. This time, Steve McIntyre finds a statement by Lewandowsky that effectively hoists him by his own petard. Apparently, data integrity only applied to others, not the inhabitants of […]
[…] Lewandowsky: study “Useless” unless authors demonstrate “data integrity” […]
[…] Lewandowsky: study “Useless” unless authors demonstrate “data integrity” […]
[…] Lewandowsky: study “Useless” unless authors demonstrate “data integrity” […]
[…] https://climateaudit.org/2012/09/12/lewandowsky-study-useless-unless-authors-demonstrate-data-integri… […]