Website update: Spam defenses improved

Noticing that some spam comments were getting through, I decided to upgrade the Spam Karma plug-in to version 2.1 (from 2.0) and Bad-Behavior plug-in to 1.2.4 (from 1.1.4).

As you can see from the statistics at the bottom of the page, a popular weblog like this gets around 5 times more attempted spam comments than real comments, and its an escalating conflict. Pretty soon, (if Steve will let me) the WordPress software itself will be upgraded and we’ll probably have some sort of CAPTCHA plug-in implemented to keep the garbage at bay.

I have relaxed, slightly, the number of comments allowed to be made in a certain time before Spam Karma starts getting interested, but we’ll have to see how it goes. If there is a problem then let me know.

I’ve no doubt that RealClimate probably has similar protection (although it’s also still running the scientific skepticism plug-in as well 😉 )

It’s all about the reality of the Internet in the 21st Century – while there is an economic incentive to do so, there will always be people ready, willing and able to subvert the purpose of business for their own ends. I never realised until recently, for example, that people in the Far East are recruited to play in multi-player games like World of Warcraft for 12-hour shifts 7 days a week, simply to collect virtual gold and magical artifacts that can be traded for real money by desperate cheats.


  1. Paul Penrose
    Posted Jan 19, 2006 at 7:42 PM | Permalink

    That can’t be such a bad job. I’m sure it beats telemarketing!

  2. John A
    Posted Jan 20, 2006 at 2:58 AM | Permalink

    Re #1

    What you mean staying in the same part of the game, killing monsters over and over again for 12 hours at a stretch? That’s got to be worse…no it’s about the same.

  3. Mike Rankin
    Posted Jan 26, 2006 at 3:54 PM | Permalink

    A couple of the latest comments appear to have crossed over the boundary and offer commercial links to off topic sites.

  4. nietsnethcil nhoj
    Posted Jan 26, 2006 at 11:58 PM | Permalink

    I the last couple of days I’ve been locked out of reading the site while at work. I get a complaint that my proxy server is configured wrong, and that i should talk to my administrator about it. Yeah, right.

    “So you are reading ‘climate audit’ at work?”

    “Only while I’m running hour long queries.”

    “You are running hour long queries!”

  5. Steve McIntyre
    Posted Jan 27, 2006 at 12:09 AM | Permalink

    A few other people have had problems as well – John A is looking at it. It’s amazing the problems that turn up – the weirdest was earlier in the year when people were getting transferred to an FBI site.

  6. Posted Jan 27, 2006 at 4:17 AM | Permalink

    nhoj, you must have gotten that message because John A deliberately blocked you. I know, it seems petty, but that’s the way it is.

  7. John A
    Posted Jan 27, 2006 at 4:36 AM | Permalink

    I blame global warming.

    Just recently a new paper announced that sea level rise had accelerated since 1950 from 1.7 mm/year to 1.75 mm/year. I was so startled that tide gauges around the world could be accurate to 50 micrometers, that I decided to ban someone at random.

  8. mikep
    Posted Jan 27, 2006 at 5:01 AM | Permalink

    Re 6, I was one of the ones who also got this message. I emailed Steve, who was at a loss and in any case the problem now seems to have been cured. Tim’s allegation is unfounded and seems to me malicious. Tim may want to consider that it is also counterproductive. I think I came to this debate with an open mind. I am now convinced that Steve’s criticisms of MBH are well founded. One of the factors that has persuaded me is that arguments are laid out here clearly and openly so I can assess them, whereas the “oppostion” refuse to state their case clearly (or even at all) on this site our on others, but do indulge in rhetorical tricks and name-calling.

  9. Posted Jan 27, 2006 at 7:40 AM | Permalink

    mikep, if you had clicked on the link on my comment that you would have discovered that when John A was blocked in the same way that you were, he accused me of doing it because I was afraid of him and Steve endorsed this absurd accusation. And then when I explained that he had been blocked by Bad Behavior, he accused me of lying.

    As for MBH, I don’t know whether or not Steve is correct about it, but his namecalling and dishonesty about matters that I am familiar with leave me absolutely no confidence in anything he writes.

  10. Steve McIntyre
    Posted Jan 27, 2006 at 8:37 AM | Permalink

    Tim, please don’t put words into my mouth. My interest in blocking has arisen because Mann blocked me from his FTP (as has more recently Rutherford). Per reported that some of his posts had been deleted from your blog; John A. asserted that he’d been blocked from your blog. With respect to John A, you said that operating issues (which sound a bit like ones that we’ve had) caused him to be blocked, but, as I recall, did not deny that you had deleted Per’s posts. My conclusion, and I did not initiate the debate and made it very clear that I was uninterested in it, was that it was unproven that you had blocked John A, but that you had deleted Per. I summarized your defence as follows: I shot the sherriff, but I did not shoot the deputy. I’ve attached my comments on this, which do not appear to me to “endorse” any position and certainly do not accuse of “lying”, which is not my usual form of expression. The original thread is here.

    I’ve got other things to do than to debate Spam Karma with you and would request that, if you’ve got nothing new to say on this, you give it a rest.

    Steve adds: Lambert says that John A. was not blocked [by Lambert – added Oct 6]. He says that there were server problems at his end which prevented access to everyone. We will of course take Lambert at his word, although I will note that I did not experience any access difficulties in the period in question. Anyone who want to discuss this further will find a convenient forum at Lambert’s blog, since he has devoted an entire post to this issue.

    Steve Aug 23: Lambert first said that other people had the same problem as John A and later said: “Anyway, I figured out what happened “¢’‚¬? a spambot has been spamming my blog using the same IP as you [John A], so Bad Behavior blocked access from your IP. I’ve removed the block, but if the spambot does it again, it will be automatically blocked again. If you are the only person using that IP address than your computer is infected.”

    Steve: Oct. 6: If I may summarize this boring thread . Lambert’s position can be neatly summarized, in the immortal words of Bob Marley,
    I shot the sheriff, but I did not shoot the deputy.
    This post was originally about Lambert deleting posts from correspondent Per. Lambert admits that he shot the sheriff. John A. claimed that Lambert had, in addition, blocked him from the site. Lambert claims that this was due to Bad Behavior. Lambert stated:
    I explained in detail why John A’s IP was blocked. Bad Behavior decided that a spambot had visited my blog from his IP. I looked at the log and found it looked like a visit from a spambot. If it wasn’t that then it was misconfigured software on his machine. Other people have been blocked because of misbehaving aggregators.
    In my opinion, there is reasonable doubt as to whether Lambert shot the deputy. In light of this,
    I have edited out some claims by John A. in the above post. Bob Marley’s decision stands.

  11. mikep
    Posted Jan 27, 2006 at 9:15 AM | Permalink

    Tim just confirms my opinion og him.

  12. Posted Jan 27, 2006 at 10:52 AM | Permalink

    Steve, I did not say that you accused me of lying. I said that you endorsed John A’s paranoid claim that I deliberately blocked him. Which you did endorse. On his claim that I am lying about the matter you have just said that there is “reasonable doubt”. I take it you would have no objection if someone wrote “It is likely that Steve McIntyre is secretly funded by Exxon, but there is still reasonable doubt on this question.”

  13. john lichtenstein
    Posted Jan 27, 2006 at 12:05 PM | Permalink

    Steve I am a little skeptical about Rutherford blocking you from the ftp site. Can I ask you to please try again, first from your office, and, if that doesn’t work, again from Starbucks. I have a hard time believing he even knows how to put a block like that in place.

  14. ET SidViscous
    Posted Jan 27, 2006 at 12:52 PM | Permalink

    More importantl;y if you are still having problems let us know where, we can grab and forward the data.

    I did a quick search and found

    Little bit long for it to respond (checking something) but I’m happy to grab and post it for download me-self

  15. fFreddy
    Posted Jan 27, 2006 at 1:01 PM | Permalink

    Re #9, Tim Lambert

    As for MBH, I don’t know whether or not Steve is correct about it…

    Tim, why don’t you go and spend some time studying mathematics ?
    Then you could read and actually understand MBH and MM, and make an informed judgement about whether Steve is correct or not.
    Don’t be scared – maths and statistics are good fun when you get into them.

  16. Steve McIntyre
    Posted Jan 27, 2006 at 1:07 PM | Permalink

    #12. Tim, you said:

    Steve endorsed this absurd accusation. And then when I explained that he had been blocked by Bad Behavior, he accused me of lying.

    Now you say:

    Steve, I did not say that you accused me of lying.

    It’s a little hard to keep up with you sometimes. I can see how your pronoun could have referred back to John A; I’d recommend that you use nouns more frequently if you’re trying to write precisely.

    I said that there was reasonable doubt about John A’s claim that you had intentionally blocked John A. I did not endorse John A’s allegations about you lying; I snipped out all such verbiage. I’m not sure what more you expected. However, your position still remains: I shot the sherriff, but I did not shoot the deputy.

    I really can’t be bothered debating the illogic of your last sentence. If you can’t see the problems with it, then there’s little point discusing it with you.

  17. Steve McIntyre
    Posted Jan 27, 2006 at 1:14 PM | Permalink

    #13 – At Mann’s FTP site, I could download from the University of Toronto library but not from my usual computer at home. I’m on cable and one of my neighbors on the same cable tried Mann’s FTP site and was blocked as well. So the block was real and was done on some code that was wider than simply my computer.

    Rutherford was involved at Mann’s FTP site so he’d have access to the same info. I’m definitely blocked at his site in the same way that I’m blocked at Mann’s UVA FTP site.

    So while you may have your doubts, it’s true nevertheless.

  18. Posted Jan 27, 2006 at 8:38 PM | Permalink

    We cannot be certain that Steve is lying about being blocked from Mann’s FTP site. There is reasonable doubt on that question.

  19. nietsnethcil nhoj
    Posted Jan 28, 2006 at 1:16 AM | Permalink

    JohnA continues to block me at work, but, as he pointed out, global warming has been blamed for almost everything, so this is fair game. I think it’s likely JohnA is doing this by accident, but I have a way with people, and maybe JohnA is just skipping the preliminaries.

    Steve it sounds like a few of your neighbors or people using your same ISP are running an unpatched Windows cable spambots and getting a range of IP addresses blocked that includes yours. This sounds like an issue for network admins, not engineers or statisticians.

  20. John A
    Posted Jan 28, 2006 at 3:31 AM | Permalink

    Re: #19

    Not deliberately blocking you. Bad behavior (the plug-in) is not exactly programmable, which causes me to reconsider whether bad behavior (Tim Lambert) could have been responsible for my being blocked on his weblog. Since this blocking happened just after I had upgraded the plug-in (and assuming that I had been blocked from Doltoid after Tim had upgraded his), then I have plausible deniability – but so does Tim.

    In which case, I withdraw my accusation that Tim Lambert deliberately blocked me from his weblog via such a flakey mechanism.

    My apologies. It looked sinister, and Tim’s explanation at the time (problems with the Internet) was implausible, but the reality is that in the brave new world of spambots sending fake messages to weblogs so as to boost the rankings of websites hawking dodgy pills and money making scams, there can be collatoral damage of innocent posters being caught in the crossfire.

    John A

  21. Dave Dardinger
    Posted Jan 28, 2006 at 9:49 AM | Permalink


    How does a spambot increase website rankings? Is the intention to get the Blog owner to return a message saying, “Welcome to my blog”? Or is there some automatic pingback which is generated when someone posts to a blog? I’d think that a decent way of counting hits would be able to tell true hits from such fakey ones. OTOH, I’m not a web-master so I don’t know how counters work.

    I have noticed occasional random “Great website” messages here, but didn’t know what they were about.

  22. Steve McIntyre
    Posted Jan 28, 2006 at 10:10 AM | Permalink

    #19 – the only sites that I’ve been blocked on are Mann and Rutherford. Maybe it’s coincidence arising from someone on the network, but then I’d have the problem elsewhere. I’ve been told that blocks are not hard to implement and I see no plausible explanation other than that I’ve been blocked.

  23. John A
    Posted Jan 28, 2006 at 10:11 AM | Permalink

    Re #21


    The answer to that appears to be that when Google spiders the site, it collects the links assuming them to be for popular sites. Thus the sites get moved up in Google’s ranking.

    What it does for a weblog is fill the comments field with useless rubbish which stops real discourse.

    One of the more popular ways to do this is to produce an aggregation site of weblogs, add Google Adsense, and then get high ranking for your worthless site, and collect money from Google. Another way is to scrape content from Wikipedia and do the same thing.

  24. Dave Dardinger
    Posted Jan 28, 2006 at 11:01 AM | Permalink

    re #23 I see. So when you delete, semi-automatically, these junk messages, you eliminate the links back to their sites.

    Now, for instance, looking at just this thread, only messages from you, Steve and Tim Lambert have links in their ‘Comment by’ line and it looks like both yours and Steve’s don’t lead anywhere (though I haven’t tried clicking them right now since that means I’d have to copy what I’ve just written, do it and then come back here and repaste and it isn’t worth it.) Anyway, if I get what you’re saying, Tim would get “credit” for his link if Google came by and traced all your links just now. So, can/do you at least only allow links to show for vetted posters? That would seem to be a partial solution, though I suppose a clever spambot would insert links in the body of the message as well as into the “from” line. And, of course, even a dumb spambot wouldn’t know what you do with his message and so would still send them hoping the link will show.

  25. John A
    Posted Jan 28, 2006 at 11:44 AM | Permalink

    Most of the time it is automatic. The problem is that like computer viruses, comment spam evolves, and new tricks must be learned.

    What Google did was invent a new tag called nofollow which means that links with this tag do not get any ranking boost for Google. This is the default for WordPress and (I think) MoveableType. There is a plug-in to turn it off.

    There is no way (other than by playing with the scripts) to distinguish between righteous and unrighteous commenters. One of the things we could do would be to only allow registered users to comment on the blog, but Steve would probably not want to raise the bar too high to people wishing to comment.

  26. McCall
    Posted Jan 28, 2006 at 6:32 PM | Permalink

    Re: 15 Tim, why don’t you go and spend some time studying mathematics ?
    Contrary to his Phys-Thermo weaknesses, Dr Lambert does have a BMath. I believe he also has a material education in statistics; but since his blog move, his UNSW and other supporting links don’t work (or I just haven’t found them).

    None of this answers your point of his lack of grasping the MBH errors. So there must be something else to explain his statistical disconnect on MBH — perhaps ideological?

  27. ET SidViscous
    Posted Jan 28, 2006 at 8:54 PM | Permalink

    When did Timmy get his Phd?

  28. McCall
    Posted Jan 29, 2006 at 1:58 PM | Permalink

    Haven’t found that, but based on the dates/locations of “Recent Publications” and the photo here: , mid-1990s at U. of Manitoba might not be a bad 1st guess.

    Still doesn’t help explain his statistical AGW blind spot for those he agrees with, vs his unbalanced penchant for sensationalizing things like a (common) subroutine input error, into an extended call that one doesn’t know degrees/radians? Maybe that’s all just the Timmy one must endure within a talented Dr Lambert.

  29. John A
    Posted Jan 29, 2006 at 2:40 PM | Permalink

    I always assume that at the root of it all, he knows that the case for AGW is much weaker than people assume, that MBH screwed up royally and that adding lots of tree-rings together doesn’t make a thermometer record. I suspect that to let go of such shibboleths would be a disaster for his ego, like giving in to the Bush Administration or something.

    Perhaps if you want to save the world, or have become convinced that the Earth needs saving, then nothing less that supporting those who work toward that aim (or apparently so), and pronouncing anathemas to those who oppose you, will do.

  30. Posted Jan 31, 2006 at 8:09 AM | Permalink

    John, I can’t disagree with you, but I don’t think it becomes you to make fun of Tim or anyone else in that manner. The Hockey Team make themselves look like idiots with their petty character assassination and poor science. I think you will all come across better if, not only is your science good, but you are polite too. In other words, don’t go ascribing motives to people who disagree with you, even if you think you are right, since we are here to discuss the science and not take cheap shots.

    I find the psychology interesting, but basically, I don’t think this is the the right place really. I suppose it would be even more rude to talk about him like that behind his back rather than in front of his face, but neither really contributes to the climate discussion IMO, in fact it detracts – it really bothers me when people at RealClimate etc. are dismissive or even ridicule “skeptics” so I think it’s best if we steer clear of such behaviour.

    (Hope I’m not out of line, as I am new around here, but thought I could offer some advice…)

  31. ET SidViscous
    Posted Jan 31, 2006 at 10:13 AM | Permalink


    The thing is, until you’ve bopped around this discusion (and I don’t mean just here, I mean the topic in general) and you’ve run into Timmy and his rantings over and over again in so many places, you just kind of have to.

  32. john lichtenstein
    Posted Feb 2, 2006 at 11:06 AM | Permalink

    “You have a great blog” == Spam. Except just here.

  33. McCall
    Posted Feb 4, 2006 at 8:13 PM | Permalink

    Re: 31 — perhaps your right, Nicholas. Strike my “ideological” speculation of my post 26 (as inappropriate, as well as unbecoming). Why given his training and fine analytical record, Dr Lambert has this statistical blind spot on MBH should be answered by the principal himself.

    My largely high regard for Dr Lambert (except his thermodynamics) has been consistent; and that is the reason why I commented on post 15 in my post 26. Unfortunately, I’m still unable to continue my direct blog discussion with Dr Lambert, as I was notified in a personal thread reply 4 (responding to my thread blocked inquiry in reply 3). I have response posts to ongoing Deltoid threads (including “the disinformation cycle”), awaiting Dr Lambert’s reactivation of them on his new blog — maybe some of my “unbecoming” posting on this thread was influenced by the delay of their reactivation? For that, I apologise.

  34. McCall
    Posted Feb 4, 2006 at 11:52 PM | Permalink

    Oops; sorry. Re: 30(31) & 33 — just realized too many Johns here.

  35. Posted Feb 6, 2006 at 9:19 AM | Permalink

    Great blog. Found your blog while searching for more information at yahoo about ftp. Your blog has quite a lot of interesting thoughts. Keep up the good work, :).

    free ftp hosting

  36. Posted Feb 6, 2006 at 9:40 AM | Permalink

    Sorry for the confusion. I guess my point is, if someone is not performing good science, admonishing them for that ought to be embarassment enough. Granted, it’s infuriating when they try to “wriggle out of it” by changing the subject, or other tricks, but I don’t think stooping to insulting them is going to help any. I wouldn’t want for people who come in from outside, curious about the science, to be put off by personal attacks. The science should be able to convince them on its own. And as far as I am concerned, anyone looking at this blog with an open mind will quickly work out what’s going on and who has some questions to answer, without us having to resort to character assassination – as satisfying as that might be.

    Please consider this some friendly advice only, from someone relatively new around here.

%d bloggers like this: