John Cook, whose crush on Lewandowsky continues unabated, asked various blogs, including Climat Audit, to direct readers to another online survey. Lucia has discussed the survey here.
The links to the survey from SKS here is http://survey.gci.uq.edu.au/survey.php?c=1R9YT8YMZTWF and from Rabett hereis http://survey.gci.uq.edu.au/survey.php?c=II7WP4R4VRU7. More IDs are available at Lucia’s.
It is easy enough to access both blogs using hidemyass.com and then click on their link to the survey. In the survey, readers are asked to rate various abstracts according to their support for AGW. I urge readers to take as much care with the survey as the respondents to Lewandowsky’s Hoax :), where Lewandowsky argued that fake responses should not be excluded.
Climate Audit 
79 Comments
Tactical decision to post Steve?
Note that I suggested that readers spend equivalent time to those who responded to Lew’s survey. If, for example, you don’t care about the quality of your answer or you are answering the question the same way – as some Lew respondents did -, it takes scarcely any time to fill out the survey. Indeed, if one were so inclined, one could submit multiple responses very quickly. If one were so inclined, HideMyAss.com enables IP address changes in the blink of an eye as well.
I took it. Many of the 10 abstracts I received were on biological or ecological micro-issues which cannot tell anyone whether or not “AGW” WILL occur overall, globally. The structure was typically more like “IF there is AGW as some/many predict, what MIGHT happen to this particular region or ecosystem.”
Most of the studies seemed highly speculative and only related to a context of beliefs or assumptions about the implications of “AGW” — not at all about establishing the scientific basis for any global “AGW” hypothesis.
Thus, I rated most of them “4-neutral” since they could not provide evidence for or against “AGW” in any way useful to the IPCC or science or humanity.
One of the 10 was blatantly some kind of press release, hyping some purported IPCC results but without any structure of analysis and evidence.
I may well go back to play around with different sets of 10 abstracts, different approaches to ratings etc., but I wanted to take it once in a semi-serious way to see what I think about it.
Strikes me that the gap between “abstract says paper assumes AGW” and the Cook survey which is supposed to be about whether “abstract says paper implies or demonstrates AGW” (with or without quantified results) is a large gap with vague understandings about what exactly participants are supposed to rate.
Should (serious) survey takers try to judge whether the paper will show or prove anything about AGW, or allow every paper that “assumes” AGW to count as evidence for AGW??
I think this kind of issue may vitiate all the broad paper-counting exercises ala Oreskes. If the abstracts I just viewed are any indication, there are many scientific papers which do not argue or prove anything about whether AGW will occur — they pay homage to assumptions and beliefs about AGW, then go on to study some potential implication. Such papers do NOT add anything to evidence for whether “AGW” will in fact occur…..
Cook’s language of “endorsement” seems to embrace abstracts in which “AGW” is merely assumed or asserted. Yet, it can only be scientifically sound to include the papers which actually provide EVIDENCE for “AGW” — all of the others are merely taking the assumption as a starting point for some other kind of study (of possible future effects in an ecosystem etc.). Cook’s words:
Perhaps a new category for ratings is called for – “Tenuous linkage to global warming owing to sparsity of grants in other areas” or similar.
It seems that for those studying respiration in Sepia officinalis, sinistral/dextral coiling in gastropods, or productivity of Zea mays, the money’s only coming if the answer is framed in terms of the possible effect of global warming on said subjects – or is that the cynic in me talking?
Skiphil: “Cook’s language of “endorsement” seems to embrace abstracts in which “AGW” is merely assumed or asserted.”
I tried the survey and I agree. 70% of the papers merely posit x degrees of global warming and use that as an input to project y amount of devastation of [good thing] or exacerbation of [bad thing].
Does a paper that posits global warming implicitly “endorse” the idea that global warming is real? I think you could argue that one either way.
BUT unless the cause of the warming (anthropogenic or not) is actually relevant to the conclusion of the paper, then the mention of anthropogenic warming is not an endorsement, it is what lawyers would call mere “dicta”:
“Dicta are judicial opinions expressed by the judges on points that do not necessarily arise in the case.”
http://www.lectlaw.com/def/d047.htm
The same link has a great line by a judge about dicta: “it is great misfortune that dicta are taken down from judges, perhaps incorrectly, and then cited as absolute propositions.”
So my guess is that the point of this “survey” is to use crowd sourcing to convert dicta into absolute (ie “consensus”) propositions.
I called it:
http://iopscience.iop.org/1748-9326/8/2/024024
well I got 3.0 and the authors 2.8!
That means you see implicit meanings where there are none.
I was trying to answer in the spirit of the survey, no messing about.
It was very odd, as none of the papers were to do with attribution, all assumed, and of coursecthe ambiguous definition of climate change or globalwarminng, somecwould assume, agw whereas others gw, or a mixture of both
I’m just commenting on how that number will be interpreted.
The abstracts you are being shown are selected from the 12000
You only get to see abstracts where the author self rated
Then they will test how you read it versus how the author read it.
Stephan Mosher –
Or perhaps it just means that there is a difference in emphasis between the abstracts that Barry Woods rated and the whole papers they represent – which were apparently what the authors were asked to rate.
Did the author’s rate the abstract or the paper? I read it as the authors’ self rating of the paper. I suspect they did not go back and rate the abstracts separately from the paper.
good point.
My point remains that I think one thing you could do with this data is compare and contrast what skeptics read in an abstract versus what the authors think, and what SkS readers think.
i recall a few instances where skeptics thought a paper was anti agw and other thought it was pro just from reading the abstract.
That suggests people bring a framework to the simplest reading task.. I happen to agree with this and you can see it in how people interprete ambiguous statements to their advantage..
Think of this. If the authors say the paper is neutral and SkS readers see the abstract as being pro agw, then consensus has been oversold..
Its actually an interseting idea.
If I write ‘ Global warming will cause a sea level rise”
And you think that is me arguing that global warming is real and human caused, then that says something about your confirmation bias.
If I write ” Wind contributed to sea ice loss”
And you think that is evidence against the consensus, then that says something about your confirmation bias..
ideally, I would just select those papers where the author thought it was neutral and then look for little shifts caused by confirmation bias..
From comments on other sites the key issue seems to be the IDs, the c=… part.
Why don’t all the sites involved just agree to use one of them, therefore removing the source site selection element.
Of course this response may also form part of the conclusions.
Lucia is discussing some oddities about Cook’s survey.
http://rankexploits.com/musings/2013/a-random-failure/
Serious question:
WHY are we interested ?
(I’m not)
My question too. Are we not just helping to produce more Lewpaper?
I am interested as to the purpose of the survey. From John Cook’s track record, the implication that he will draw from this survey are that the climate consensus is on the side of truth, as the vast majority of scientific papers are either pro-AGW, or neutral.
Clearly, Cook’s survey will be used in an attempt to embarrass skeptics. He’ll say it took longer for skeptics to answer the 10 questions than non-skeptics based on the Web stats, implying some defect in comprehension.
To me, someone taking longer to respond would be a sign that the person was actually take the time to think about what the abstract claimed. To me, that would be a sign of a more rational person rather than a more reactive person.
Don’t listen to Ian, I beg of you:
http://rankexploits.com/musings/2013/a-random-failure/#comment-112471
Another erudite climatologist professes to measure the temperature of the stratosphere with a clickon stratified random sample in order to cherrypick the census. Weak minds coupled with modern computing power are a lethal combination for the scientific method but it’s not just weak minds that are problematic-
http://www.adelaidenow.com.au/technology/tech-savvy-kids-cant-push-a-pencil/story-fn7bfu22-1226634952869?from=public_rss
Welcome to the rise and rise of the clickons!
Its interesting to watch people try to figure out what they are up to. So, beware of imputing Nefarious Intent.
I beg to differ. I’m bored already.
Drama projected on the silver screen? I’m rapt.
===========
I tried the survey.
I don’t usually find abstracts all that reliable a guide to the actual interesting points in an article.
Whether John Cook is testing some hypothesis about the defective cognitive processes of CAGW skeptics, or crowd-sourcing some piece of a dissertation project, my recommendation is not to waste your time.
People seem to have misunderstood my allusion to Lewandowsky’s respondents. The beauty of a Lewandowsky-quality response is that it is not necessary to read the abstracts to give a Lewandowsky-quality answer. Saves a lot of time.
Rest assured that some of us got the message. And few people can be said to deserve Lewandowsky-quality answers more than John Cook.
I laughed out loud!
What a riot. I actually had a very similar comment in my post. I did edit it out before publishing tho.
Coffee, meet monitor screen and keyboard…
😉
I did get it, but I was on another tack already because I’m so tired of people like Oreskes doing vast literature surveys to pretend that every scientist who parrots some words about warming or climate change has thereby provided any new evidence. But all that Cook needs to pretend to “study” with this survey is some nonsense about skeptic cognitions compared to the scientists who wrote the papers or compared to enthusiasts for the “Cause” however defined.
Yes, anyone who wants to bother with this survey could just click ratings at random or according to some pattern, etc. I just did that a few more times with various “hidemyass” IPs…… now we can see whether Cook’s analysis can attempt to cope with all the derisive and non-serious “Lewandowsky” type respondents.
Skiphil–
Heh. You could pop a brewskies before dinner and repeat that every day from now to the end of the study. But be sure to use the “Anonymous” ones. My blog gets hit by the transparent ones pass their real IP in the headers. If those try to comment, I ban them.
Does anyone remember how many responses Lewandowsky got?
Steve: 1145 were used. From recall, he excluded 150 or so.
Why would anyone in their right mind give any answer at all to Cook or Lewandowsky, no matter how well you hide your identity? You can almost depend on any information being willfully misconstrued, so why give them any data to misrepresent?
If I met either one of them in the gym shower room I certainly wouldn’t reach for the bar of soap they dropped.
My comment to the first survey stands for the second. If one is asked to agree/disagree with a proposition, there has to be irrefutable or very strong proof that a unique answer to the proposition is known. I used the example of one-world government and gave references that there were heavy people writing heavy articles about it.
I suggest that the same weakness is back again, as others have already observed. One cannot be classed as a disbeliever (in an absolute sense) if there is not a single answer known for the question, irrespective of ones response. So, the survey should, as a minimum, have a sealed envelope lodged with an official, containing the “correct” answers to the propositions and supporting evidence. Otherwise, you are a disbeliever not of the best answer, but of the preferred answer that the surveyor chooses to put; which can of course be laced politically.
There’s no point in doing the survey without this safeguard against post-poll manipulation.
There’s no point in doing this survey because the questions have no absolute answer.
It’s just publicity-seeking and best ignored.
Cook is “The boy who cried wolf” This person can’t be trusted. I say keep well away as this is just another dirty trick!
Luckily, there can’t be any danger that the results will be subjectively analyzed!:-)
Given its Cook the cartoonists , the ‘results ‘ are already in before they even asked.
An approach which certainly speeds the process up and means you don’t have to worry about ‘facts ‘ failing to support your ‘desires’
But has with much in climate ‘science’ this has nothing to do with actual science.
Spot on knr. I look at this as I would a consensus meeting;as in the result is already predetermined. So what is the point? Well you see as is a consensus meeting Cook needs to show that the end result is a vote. We already know the outcome irrespective of whatever the data.
Like the WC (Lew) paper it will just be that: brown stained toilet paper. Sorry to be disgusting but it is the truth.
Steve
Typo first line Climat
My response to Cook LeBooks survey is below :
I started to do it but after three questions I started to get flashbacks of year eight English comprehension tests. Then I thought, why I am bothering with this at all? This study will add zero to the sum of human knowledge.
I doubt if they have set themselves such an ambitious target.
What, adding greater than zero to the sum of human knowledge?
Yes.
‘Course not – they’ll simply use whatever pre-determined twisted result for propaganda purposes
For once, I might have something Steve thinks useful to contribute regarding this discussion http://rankexploits.com/musings/2013/a-random-failure
John Cook says that he is using this SQL select to choose the abstract list for each participant. “So the SQL query used was this:
SELECT * FROM papers WHERE Self_Rating > 0 AND Abstract != ” AND LENGTH(Abstract) < 1000 ORDER BY RAND() LIMIT 10"
Assuming that this is the actual SQL statement Cook is using, the above use of an unseeded RAND() function will result in all participants using the same pseudo random number series, which is a big no no if you want each participant to get a random abstract list. A proper use of the RAND() function would require a unique seed value to be provided to the RAND() call for each and every participant. I would not be surprised at all if most if not all participants are getting a very similar abstract series presented to them because each participant causes an unknown pre-initialized seed value to be used for the RAND() call which results in the exact same pseudo random number series being used for all participants.
Hmm, turns out I am probably wrong – as usual.
A few seconds with Google found the following thread regarding selecting random records from a table (it assumed MS SQL, which is what it appears that John Cook is using):
http://stackoverflow.com/questions/848872/select-n-random-rows-from-sql-server-table
The “order by rand()” option is not recommended….
And it’s my turn to be wrong 🙂 It looks like he’s using MySQL, not MS SQL, and the behavior is different. The approach he’s chosen appears to be the recommended approach, though with the caveat that RAND() is not a perfect random number generator. See the RAND() function from the MySQL site:
http://dev.mysql.com/doc/refman/5.0/en/mathematical-functions.html
I don’t claim to be an expert on MySQL by any stretch, but sorting operations generally occur after you have a result set. Given the query:
SELECT * FROM papers WHERE Self_Rating > 0 AND Abstract != ” AND LENGTH(Abstract) < 1000 ORDER BY RAND() LIMIT 10
… Most optimizers will run a count stopkey on the first 10 records the select hits (which is not predictable, but not truly random either), and then the resulting set of 10 would get ordered according to the function. So, probably not all that surprising that the same records are coming up unusually frequently. A better way to get a more distributed result would be to hash the papers by some function using data specific to each session (ip + session id or similar) and then select out of that.
Levi, you’re right that sorting operations generally happen after getting the result set. The problem with your idea is LIMIT is not part of the command to get that result set. It is a filter applied after.
The way it should be applied is as a parameter of the sort function. As in, only sort until the first 10 values are found. I don’t think MySQL is that clever though.
Looking into it a bit more, I see that you’re right. But I also see lots of sites complaining that “order by rand() limit x” is a performance killer since by definition it forces the rand() call to be processed for each row. So, if the survey is loading fast, then it would suggest that the source set is smaller than described, as I think you have argued.
I still think a hashing solution would probably provide more “randomness”. I see from your threads at Lucia’s that sessions and IPs are easy to spoof here, so maybe hash with a timestamp.
Indeed. If you run that query on a database with 12,000+ entries, it will kill your performance. That doesn’t necessarily mean a fast load time disproves the use of 12,000+ entries. After all, it’s possible the database server is simply fast enough that it could accept the performance hit. That’s unlikely, and given John Cook has already said he isn’t using all 12,000+ entries, there’s really no reason to think he is.
As for your idea of using some sort of hashing solution, that’s not a bad idea. A commenter at lucia’s directed me to a copy of the source code MySQL uses for its RNG. It’s one known to be pretty bad. It will definitely create patterns. Finding a better solution might be worthwhile. It’s just a question of how much we care. The patterns introduced by the RNG likely won’t be strong in the final sample sets, and a lack of total randomness probably won’t hurt the end results.
Why should anyone partcipate in this? I.e., what can we possibly learn from this excursion into internet surveydom?
Perhaps we misunderstand poor John Cook, his buddy the mass manufacturer of Lew Paper, has run off and left him, traffic to Sceptical Science is most likely down to members only and Cook is rightly ignored by rational people.
This “survey appears to be nought more than a desperate attempt to increase traffic at his SS site, before the funders thereof abandon ship.
We are seeing a trend here as propaganda sites are being defunded and abandoned. I wonder how long the new management at NASA will tolerate Real Climate?
Now that James Hansen has gone poor Gavin Schmidt may have to find honest work.
I would not endorse this survey on the grounds that it uses poor language to formulate the scales.
I am sure they don’t mean to asked the surveyed if they believe an abstract endorses global warming or endorses global warming caused by people. I personally would endorse global warming, given the alternative, but I don’t think that is what they mean.
I expect they mean to ask the surveyed if they believe an abstract endorses a theory of global warming. And therein lies all kinds of confusion; which theory of global warming, the one with high sensitivities or the one with low sensitivities, etc?
At least they are demonstrating a certain consistency in their approach.
I took the survey without reading the abstracts. My answers were: 1, 2, 3, 4, 5, 6, 7, 8, 1, 2. I wanted to give the authors some practice separating noise from signal. They seem to need the practice. Next time I take the survey, my answers will be 8, 7, 6, 5, 4, 3, 2, 1, 8, 7. And the time after that…..
I can’t for the life of me figure out what possible useful purpose this survey is meant to achieve. So we are supposed to decide, based on the abstracts alone, of some supposedly random (although clearly not) group of papers, what level of AGW endorsement the paper reflects? So what? What do we learn from that? Is there some reason to believe that random respondents on climate blogs are good at that sort of thing? What if they aren’t? What if they are?
Pass.
As Lucia noted, R.A. Pielke Sr. provides more objective survey questions: Is there agreement amongst climate scientists on the IPCC AR4 WG1?
Interesting survey – it would be even more informative if it was repeated again now that more information is available after the various Climategates!!
@Jeff Norman, I too think the survey is poorly constructed.
I can’t get no definition.
========
.
I don’t get no …. i d e a t i o n,
I don’t get no …. i d e a t i o n,
I can try, I can try, I can try …. to trend high!
I DON’T GET NO …. i d e a t i o n.
But if you spy some time, you might find, you trend where you neeeeeeeeeeed.
=========
Has anyone considered this survey is a honeypot, that Cooks’ real goal is something else, such as gathering evidence to support the hypothesis that climate skeptics will trash real science just out of spite)?
I tried to, but my comment is still in moderation.
Sure. The real study could be the reaction to the study, except that there isn’t much actual reaction beyond mocking the old study.
I used to comment on John Cook’s blog and even indulged in an exchange of off-line emails for several months but when he introduced “Dense Moderation” (aka censorship) I realised his ship was sinking and abandoned it along with the rest of the “Denier” rats. Since then his Alexa rank has declined although not as dramatically as I expected.
So what kind of sense does it make for him to solicit our input almost three years later? Does he want us rats back? Is he trying to “Make Nice”?
Nah! As most of the commenters here understand, John Cook can’t be trusted. He would not need censorship if there was any merit in the Voodoo “Climate Science” he is pushing. The poor guy is desperate so let him twist in the wind.
SKS keeps good records but the security on the site is not impregnable so I retrieved some of my comments that were “Moderated” along with the moderator’s reasoning. The “Moderators are mostly lightweights who dare not debate on any open forum with the exception of “dana1981” who participates but performs very badly. Arrogance and youth are a dangerous combination. FWIW:
http://www.gallopingcamel.info/Docs/DeletedCamel.doc
“Dense Moderation” – exactly.
Well my response average was 7. Can’t say I read the abstracts but I moved the mouse over them and waited a few minutes before responding. This survey is likely to be as accurate as the drug use surveys my classmates and I anonymously filled out in 7th grade – there was a lot of cocaine use among the participants.
I took a survey like that in high school. It was remarkable how many people never smoked in their life yet somehow smoked two packs a day.
(Contradictory answers like that existed in Lewandowsky’s survey. He didn’t filter them out.)
http://phys.org/news/2013-05-extensive-antarctic-campaign-cold-bias.html#nRlv
It seems that you and Anthony are right about temperature measuring bias, Steve.
even better
http://news.yahoo.com/greenlands-glacial-melt-may-slow-study-suggests-170441308.html
Re: Mike Roddy (May 8 08:43),
Mike, you’re pretty good at demonstrating your lack of understanding. But I’m always hopeful you can learn.
As noted in the article you linked,
I.e, the difference between your link and what Anthony has pointed out: your link is about a consistent bias/offset; Anthony has been showing site placement errors that introduce errors in the temperature trend.
I don’t think a lot of people appreciate what Cook is doing here. It doesn’t matter what the papers say about climate change it doesn’t matter what the authors think.
It is clear that Cook is planning on calculating a score from those who take the survey from a link at a “consensus” site with those who take it from a “sceptic” site.
He will then simply have two bins with average scores for each. He is expecting (wants) to see the sceptic bin to find less evidence of the “consensus” in the papers than the consensus bin.
Then he will say there is a “statistically significant difference ” between the two bins and conclude that sceptics (deniers) have some sort of psychological disorder that makes them blind to the evidence.
Of course the fallacy will be that hthey well have proven that alarmists ar edeluding themselves about the presence of a support for alarmism in the literature – somethnig that could also explain in discrepency between the two bins.
This is going to be a piece of garbage research, with insufficient controls to have confidence that you have put scepitics and alarmists in the correct bins, or that you have properly controlled for the ACTUAL content of the papers – which is clear they aren’t because it wouln’t be feasible to do so for 12,000 papers, even if dealing with the abstracts only.
Certain questions you can be absolutely sure John Cook will not be asking of his university peers, let alone climatologists are-
https://theconversation.com/whose-views-skew-the-news-media-chiefs-ready-to-vote-out-labor-while-reporters-lean-left-13995