Garbage Characters in Old Posts

As a result of the migration between servers, specialized characters in old posts are now being rendered as garbage quotes. The characters that are affected are mainly “, -, ‘, …, š, ü , … Each of these renders into long garbage strings. To fix the problem, one needs to do a bulk find-and-replace over the entire database. I don’t know how to do it. There’s probably some information in WordPress but I don’t have the time to figure it out and it’s probably not very efficient for me to do so anyway. If someone more knowledgeable can volunteer a method, I’ll see if it passes muster with Pete Holzmann and/or Anthony Watts and see if either of them can fix things without screwing up something else.


  1. John A
    Posted Nov 22, 2007 at 8:52 AM | Permalink

    I do know how to do it. The weblog will need to be offline for 30 minutes in order to do it, though.

  2. kim
    Posted Nov 22, 2007 at 8:53 AM | Permalink

    Thirty minutes? Intolerable.

  3. Steve McIntyre
    Posted Nov 22, 2007 at 8:56 AM | Permalink

    #1. Thanks, John A. We’ll coordinate offline.

  4. Geoff Olynyk
    Posted Nov 22, 2007 at 8:59 AM | Permalink

    Adding LaTeX markup support would be something that would be really useful here. Also a hack to make WordPress not mung up the quotes when inside “code” tags. (if only one had unlimited time, eh)

  5. Bernie
    Posted Nov 22, 2007 at 9:15 AM | Permalink

    Talkingof housekeeping, any insight into the traffic in the post weblog award world?

  6. Posted Nov 22, 2007 at 9:32 AM | Permalink

    For a minute I thought you were talking about some of the chaps who leave comments here! 🙂

    Happy T-Day!!!

  7. Anthony Watts
    Posted Nov 22, 2007 at 9:38 AM | Permalink

    John A – I welcome your help on this, this may also be an opportunity to update WordPress to the latest version.

  8. Skip
    Posted Nov 22, 2007 at 9:40 AM | Permalink

    If you get this fixed it will make me very happy. I only recently started following this, probably three or four months ago after I saw a link here from NRO, and I realized that quite a few of the posts assume knowledge of the issues I didn’t have. So I’ve been trying to read the archives, and in many cases they’re tough sledding with the munged special chars.

  9. Anthony Watts
    Posted Nov 22, 2007 at 9:40 AM | Permalink

    John A – Of course, follow rule#1 do a database backup, offbox from the physical server first.

  10. Skip
    Posted Nov 22, 2007 at 10:32 AM | Permalink

    Will updating to the latest WordPress get rid of the large horizontal scroll? That would be nice as well. If not, I can give you a very small style sheet adjustment that mostly fixes it, although for the I don’t actually understand why it would.

  11. Anthony Watts
    Posted Nov 22, 2007 at 10:51 AM | Permalink

    Steve, perhaps we should kibbitz on an agreed upon “down for maintenance” period and post it so as not to catch many people off guard. JohnA says 30 minutes, but I’ve learned that in computer work, any stated goal often takes 5 times the original time estimate.

    If we advertise a 3 hour down period, and it gets done in 1-2 hours, all the better. If we attempt a WordPress upgrade or stylesheet fix, figure 6 hours.

  12. John A
    Posted Nov 22, 2007 at 12:32 PM | Permalink

    John A – Of course, follow rule#1 do a database backup, offbox from the physical server first

    Come on, where’s your sense of adventure? Real engineers don’t read manuals or take backups – which is why I recognize so much of climate science being after my own heart.

  13. Peter D. Tillman
    Posted Nov 22, 2007 at 2:28 PM | Permalink

    Whoever does this fix might also want to run a dead-link finder utility over the site, if you have one handy.

    If not, I’ll write my webmaster pal & see which one he uses.

    Cheers — Pete Tillman

  14. Andy
    Posted Nov 22, 2007 at 3:18 PM | Permalink

    Re #10 Skip, I only have the horizontal scroll problem with IE. Firefox seems to like CA much better.

    Seems fitting, since the Firefox source code is available, but IE’s isn’t 🙂

  15. John A
    Posted Nov 22, 2007 at 4:26 PM | Permalink


    I’ve no idea if the mooted upgrade will fix the problem but it is definitely an IE-only problem.

    If it doesn’t fix the problem (and the upgrade won’t happen until Dec 1st or 2nd at the earliest) then send me the stylesheet upgrade to climateaudit AT

    [And no, its not that I’m stepping back into the fray, just that I happen to have done a test with an old copy of CA upgrading to Wp2.3 and fixing the character problem just recently and just for the heck of it.]

  16. Skip
    Posted Nov 22, 2007 at 5:41 PM | Permalink

    Alright. It’s a pretty simple fix, I just need to make sure it doesn’t screw up anything on Firefox when it fixes IE. I’ll try and get that tested tomorrow. I doubt the upgrade will fix it, since it seems to be in the theme rather than wordpress itself.

  17. Posted Nov 23, 2007 at 5:05 AM | Permalink

    I wish you a good luck because it is not pleasant to be called LuboLongGarbageStrings Motl. 😉

  18. Anthony Watts
    Posted Nov 24, 2007 at 2:56 PM | Permalink

    RE17 Hi Lubos,

    In the same spirit, Lubos, maybe some time spent fixing your blog page so that it doesn’t crash browsers would be also helpful at this time. I’d love to add you to my blogroll, but I can’t in good conscience direct people to a page that goes “boom”.

    Its seems all the add-ons are the culprits.

    I’m not busting your chops, just trying to point out something that I’ve also seen others mention. I’d really like to be able to visit your blog without self destruct mode.

  19. John A
    Posted Nov 28, 2007 at 2:39 PM | Permalink

    I’ve started going through the database looking for the trash characters and replacing them with the correct punctuation. So far 5000+ changes have been made

    One of the things I’ve also done is replace “Lubo+(weird string of characters) Motl” in the comment authors with “Luboš Motl” (weird string theory character)

    As people encounter posts or comments with the strange characters, can they post the links to the posts or comments and I will run a MySQL search/replace to deal with them. Don’t post the characters themselves, just where you encountered them. (The link to the comment is in the number of the comment).

    After a week or so, I’ll attempt the upgrade of WordPress to 2.3 and see what other things break!

  20. Sam Urbinto
    Posted Nov 28, 2007 at 3:15 PM | Permalink

    Lubos, yeah, your site often has extreme trouble with IE. Firefox seems okay tho.

    John A, thanks, you rock!

  21. Larry
    Posted Nov 28, 2007 at 3:35 PM | Permalink

    18, it can get bad sometimes. About a half hour ago, I went to RF, and the “earn your degree” thing took over, and not only crashed the browser, but actually signed me out of the user session. That’s pretty bad.

  22. Peter D. Tillman
    Posted Dec 9, 2007 at 6:21 PM | Permalink

    When John A. (or whoever) fixes this glitch, another minor tweak that would be helpful would be to switch the default in the search box to CA (instead of the whole web), as this is where most people will use it.

    Thanks in advance, PT

%d bloggers like this: