Duke C Punctures More Attempted UEA Obtuseness

CA reader Duke C has some results from his FOI request that look like they bear directly on my longstanding appeal for the Wahl attachments that the UEA purport to be unable to locate on the backup server.

The UEA has a history of wilful obtuseness in carrying out FOI searches. Unfortunately the UK Information Commissioner has weakly and unjustifiably acquiesced in UEA’s obstruction.

Don Keiller has elsewhere recounted his story of UEA’s obtuseness in locating a single covering email. They originally made absurd estimates of the cost; then denied that the email was on the server. Only after Keiller threatened to appeal further was the server searched properly and, needless to say, the email had been there all along. The footdragging cost 7 months and dozens of emails.

My Appeal of 10-051
Duke C’s request touches on my own outstanding appeal to the Tribunal in connection with the Wahl attachments. In August 2012,following the return of the backup server, UEA informed me that they had been unable to locate the long-sought attachment to Wahl’s 2006 email to Briffa on the backup server. Perplexed by this failure, I asked UEA whether they could locate the underlying email on the backup server – as a means of benchmarking whether they were looking in the right place.

They refused to answer this simple question as follows:

You pose a number of queries which relate essentially to whether the emails to which documents 5-8 were attached can be found on the server. We repeat that the issue in the present case is not whether the emails are on the server but whether the attachments can be located following a reasonable and proportionate 2search. We would hope that you would accept that expanding the search for the attachments to the entire server answered any outstanding concerns you may have about the quality of UEA’s searches.

This has led to extended correspondence with the Tribunal about the adequacy of the UEA search. Despite UEA’s pointblank refusal to answer this and other questions relevant to the adequacy of the search, the UEA’s lawyers, Mills and Reeves, falsely told the Tribunal that “the University responded in full to Mr Mclntyre’s additional queries on 25 September 2012″.

The UEA also purported to explain their inability to locate the attachments on the basis that the backup server removed older backups on a rolling basis (seemingly in contradiction to earlier statements by IT personnel to the Muir Russell panel):

It is right to say that the earliest backup that is held for Professor Briffa’s work PC is the 2 August 2009 backup. However, that is not to say that that backup does not store emails dating back to a period before 2 August 2009. It is merely to say that there are no earlier backups. UEA’s position is that the 2 August 2009 backup would have included copies of all emails and attachments stored on Professor Briffa’s PC as at 2 August 2009 and this could easily have included documents and emails dating back to 2005/2006. You should in any event note that the backup server had an automated function that operated so as to remove older backups on a rolling basis.

I reported briefly on these matters a few months ago.

12-139
Unknown to me, CA reader Duke C (Patrick Reeves) had submitted his own FOI request for 2006 email correspondence between Wahl and Briffa. UEA answers are online here – see 12-139. In the notes below, I will draw on these answers, together with the original questions provided offline to me by Patrick.

On October 11, 2012, Patrick sent an FOI request to UEA regarding the Wahl-Briffa correspondence asking for 7 emails, 5 from Briffa to Wahl and 2 from Wahl to Briffa. One of the requested emails (Aug 12 – Wahl to Briffa ) was one of the underlying emails in my appeal. Patrick:

I wish to request the complete email documents associated with the following 7 email headers located at the Climatic Research Unit [lists 7 emails] …

I will amend this request to include all emails that include ” Keith Briffa” or “Wahl, Eugene R” in either the “to:” field or the “from” field between the dates of July 1, 2006 through August 31, 2006, if this would simplify the task for your staff

On November 12, not having heard from UEA, Patrick sent a reminder noting the passage of 20 working days, and added an inquiry about attachments:

1. Email correspondence between Prof. Keith Briffa and Eugene R. Wahl that occurred between July 1st, 2006 through August 31st, 2006, located on the CRUBACK3 backup server.

2. Documents/attachments related to the above emails, which my be in the possession of Prof. Keith Briffa and not located on the CRUBACK3 backup server.

The same day (Nov 12), UEA official David Palmer replied, noting that attachments had not been requested in the original request and that a separate search would be required for the attachments. Palmer told Patrick that UEA would be replying very soon on his request for emails. Palmer confirmed that UEA had “searched all possible locations” for the requested information:

I can also assure you that, as per our statutory obligation, we have searched all possible locations for the requested information including material on networked resources, and also on stand-alone resources such as the holdings of Dr Briffa himself and the contents of the server from which the emails now publicly available apparently originated.

Later on the 12th, UEA responded stating that they had located the 5 requested emails from Briffa to Wahl (which were provided in an Appendix), but stated that the two listed emails from Wahl to Briffa had not been located despite their search (previously said to have been of “all possible” locations:

It is, however, not possible to satisfy all elements of your request. Pursuant to your rights under section 1(1)(a) of Freedom of Information Act 2000 to be informed whether information is held, I confirm that the University does not hold some of the requested information. Specifically, we do not possess two of the requested emails, both from Dr Wahl to Dr Briffa, one on 21 July 2006 and the other on 12 August 2006

This failure was also reported as follows:

12-139 excerpt 2

On November 20, Patrick sent a follow-up email to UEA deferring the request for attachments, but questioning the UEA’s failure to locate the Wahl-to-Briffa emails located in the threads. Patrick then described how Eudora handled backup and directly asked whether UEA had searched “all themed inboxes with consecutively numbered mbox”. Read this carefully:

… it is somewhat perplexing that two of the emails are not held, when segments of these emails are contained within the threads of the emails that were released.

After familiarizing myself with the Eudora file management system, these emails should reside in the \Eudora directory within files named in.mbx or out.mbx. However, when a user deletes and/or empties his Trash folder, a new iteration of the in.mbx or out.mbx is created which contain the deleted items, and each is consecutively numbered.

Example:

11/16/2012 04:44 PM 0 In.mbx
11/16/2012 04:29 PM 845,516 In.mbx.001
10/15/2012 06:33 PM 481,012 In.mbx.002
11/19/2012 06:37 PM 517 Out.mbx
11/19/2012 06:19 PM 769 Out.mbx.001
10/08/2012 04:33 PM 0 Out.mbx.002

I would ask, were all themed inboxes with consecutively numbered mbox files searched, including email or document resources that may have been transferred to portable media by Prof. Briffa?

Palmer acknowledged Patrick’s request on November 22, noting that the UEA’s inability to locate some emails “may be perplexing” but was still the result of the searches. Nonetheless, Palmer passed Patrick’s query on to UEA technical staff:

As to emails that are not held, whilst it may be perplexing, this is the result of the searches we carried out. I am not technically proficient enough in Eudora to address your comments regarding Eudora file organisation or the searching techniques employed but I have forwarded your concerns to the appropriate technical staff within the University and I will pass along any comments they have in response to your concerns.

A week later, on November 30, following Patrick’s procedures, the UEA were suddenly able to locate the Wahl correspondence that they had only a few days earlier been unable to locate. Although they had claimed to have searched “all possible locations”, their search did not include a search of the “numbered mbx backup files.”

As noted in my email of 22 November, I have forwarded your concerns regarding the extent of our searching to the appropriate technical staff within the University and can report that in response to your question; no, we did not initially search the numbered mbx backup files.

Palmer reported that they were now able to locate three files that had not been previously retrieved or searched.

Having concluded that the requested information could, in fact, be held within these files, we conducted further searches late last week. Specifically, we identified three files which were not previously retrieved or searched: In.mbx.001, In.mbx.002 and Out.mbx.001. There was no Out.mbx.002 file.

They then searched the three files and found the two requested emails from Wahl to Briffa, together with nine others. These were all included in an attachment to the November 30 response.

All three files were decompressed and searched for any references to ‘wahl’ and all search matches were checked to determine whether they related to emails exchanged between Keith Briffa and Eugene Wahl between 1 July 2006 and 31 August 2006.

As a result of this revised search, we not only discovered the two emails previously reported as ‘not held’ but nine (9) other emails that fall within the scope of your request of 11 October 2012. These are contained within the attached document entitled ‘Appendix A_Additional material. pdf’.

I would like to take this opportunity to apologise for not providing this information at first instance. We do take our obligations under the Act very seriously and are very sorry that our original search strategy overlooked this information. I would like to thank you for bringing this to our attention so that we can fulfil your request in its entirety.

There is an obvious knock-on impact on my case: were the numbered mbx files decompressed and searched in the previous unsuccessful UEA search for the Wahl attachments? If they were searched, why wouldn’t the UEA have searched them in response to Patrick’s 12-139 request?

If they weren’t searched, then they should obviously be searched for the Wahl attachments, now that UEA is aware that the underlying Wahl emails are located in the numbered mbx files. On December 20, I had a Case Management meeting, much of which was devoted to the adequacy of UEA’s search of the backup server. However, none of these new developments were disclosed by UEA at that meeting.

43 Comments

  1. Brent Buckner
    Posted Jan 11, 2013 at 2:00 PM | Permalink

    As I understand it, Eudora does not store received attachments in the In.mbx file, and similarly it does not store them in the subsequently numbered files (e.g. In.mbx.001). It stores all received attachments as separate files in a folder.

  2. Posted Jan 11, 2013 at 2:42 PM | Permalink

    Great work, Patrick Reeves. Anything interesting in the additional Wahl emails?

    • Duke C.
      Posted Jan 12, 2013 at 12:09 PM | Permalink

      Nothing earth-shattering yet, Richard. There is additional content not found in CG1-CG2. I haven’t had time to fully parse the differences.

  3. Posted Jan 11, 2013 at 2:52 PM | Permalink

    Does the fact that the emails were in the numbered mbx files imply that they had been deleted?

    • Duke C.
      Posted Jan 12, 2013 at 12:44 PM | Permalink

      I ran a good number of simulations w/ Eudora 7.1, Filling up the inbox, deleting the individual emails, then clearing the trash folder. In every case the emails deleted from the trash folder were still present in one of the .mbx backup files. Briffa, more than likely, was unaware of this function when he cleared out his IPPC correspondence.

      • Steve McIntyre
        Posted Jan 12, 2013 at 1:32 PM | Permalink

        In a letter to me on 25 Sept 2012, Mills and Reeves, UEA’s lawyers, stated (purporting to explain the very late date of August 2, 2009 for the ‘earliest” backup”):

        You should in any event note that the backup server had an automated function that operated so as to remove older backups on a rolling basis.

        This, of course, is a different story from what the IT staff told Muir Russell in December 2009:

        Configuration of back-up server was unfortunate as it did not remove deleted emails.

        I requested the following information (without getting an answer):

        To clarify these inconsistencies, I request that UEA confirm whether backup configuration in 2008-9 was full or incremental and to provide the BackupPC configuration file in effect in 2008-9, including $Conf{FullKeepCnt} parameters.

      • Steve McIntyre
        Posted Jan 12, 2013 at 1:38 PM | Permalink

        In every case the emails deleted from the trash folder were still present in one of the .mbx backup files.

        Simulations. Nice way to resolve the speculation.

        BTW what happens to attachments when an email is deleted? Does they live on the Attach/ directory or is the Attach directory modified. What happens to the Attach/ directory over time as BackupPC continues in operation?

        • Duke C.
          Posted Jan 12, 2013 at 7:17 PM | Permalink

          Once the email is delivered and the attachment is converted they become separate entities, for the most part. One can be moved or deleted without effecting the other, i.e. they are not joined at the hip.

        • Armand MacMurray
          Posted Jan 12, 2013 at 10:29 PM | Permalink

          IIRC, there’s a configuration setting in Eudora which specifies whether or not attachment files are also deleted when their parent email is deleted. Note that this would only affect attachments with unchanged names still in the Eudora attachment directory. Thus, renaming the specific attachment file, or copying it to a different directory, in essence severs the link to the parent email as far as Eudora is concerned. From that point, Eudora won’t be able to affect the attachment under its new name or location.

          BackupPC has no special knowledge of Eudora files and directories, so a Eudora Attach/ directory should be unaffected by running BackupPC.

    • Anthony Watts
      Posted Jan 12, 2013 at 10:46 PM | Permalink

      When trying to understand how something works, the best way to learn is to play with it. I would suggest setting up and identical version of Eudora on a machine, apply an email acount, and put it through various tests for receiving attachments and then deleting files.

      Document the results, use that for future FOI requests.

    • bmcburney
      Posted Jan 13, 2013 at 4:41 PM | Permalink

      I think Professor McKitrick’s point is that if the presence of an e-mail in a numbered MBX box means that the e-mail was deleted, this may provide evidence on the question of whether e-mails requested by Mr. Holand’s FOIA were deleted at UEA.

      • Steve McIntyre
        Posted Jan 13, 2013 at 7:26 PM | Permalink

        does anyone know what properties are available for the numbered mbx files? do their properties show date information? or would one need to know the range of dates of the contained emails?

        • Posted Jan 13, 2013 at 10:55 PM | Permalink

          I think the properties will only show the date of the last modification and at best, modification and creation, which are probably the same as these are backup files. If you haven’t seen my reply email yet, please check your spam folder. Sometimes my emails land there.

        • conard
          Posted Jan 13, 2013 at 11:29 PM | Permalink

          mbx files have no internal properties– the only properties are preserved by the hosting OS i.e. created, modified, perms, etc. T[able] O[f] C[ontent] files do have an internal structure which varies across platforms and version; but these are generated from mbx files.

          If you are looking for date ranges of email contained in an archive decompress the files, grep for from: cut everything but the date and year, sort and extract the first and last lines.

      • Latimer Alder
        Posted Jan 14, 2013 at 2:33 AM | Permalink

        Re: bmcburney (Jan 13 16:41),

        Good point…but it might be a long stretch to show exactly who deleted them.

  4. LearDog
    Posted Jan 11, 2013 at 3:32 PM | Permalink

    Do you mean to say that UEA DIDN’T say to you “Gee, do you know what? We’ve just learned about consecutively numbered in and out boxes, and we’re going to go back and look for the Wahl attachment there, too. I hope we find it in order to bring this to a positive conclusion!”

    I gather another letter is in order,, these guys are incredible. Keep at them, sir. You are my hero…

  5. Steve McIntyre
    Posted Jan 11, 2013 at 3:53 PM | Permalink

    Re-visiting the Eudora documentation in light of the new information: Eudora emails are stored in .mbx files while attachments are stored in .toc files. The toc files are binary while the mbx files are ASCII. [Update- as pointed out by a commenter, this isn't right. The toc files are merely table of contents, pointing to Attach directories.]

    It appears that there would be numbered toc files as well as numbered mbx files.

    I wonder if the missing attachments are in a numbered toc file that hasnt been decompressed yet.

    Comments in earlier post are worth reviewing as well:

    http://climateaudit.org/2012/08/10/uea-cant-find-wahl-attachments/

    some comments by specialists make a little more sense to me now.

    • Brent Buckner
      Posted Jan 11, 2013 at 4:40 PM | Permalink

      I think it may be that information about attachments are stored in .toc files, but not the attachments themselves.
      c.f. http://users.starpower.net/ksimler/eudora/toc.html

    • Posted Jan 11, 2013 at 4:59 PM | Permalink

      >Re-visiting the Eudora documentation in light of the new information: >Eudora emails are stored in .mbx files while attachments are stored >in .toc files. The toc files are binary while the mbx files are
      >ASCII.

      That doesn’t sound quite right. The .mbx file will be a mailbox file. It will contain the entire text of every email in a given mailbox/folder. It’ll be one very large text file with the emails appended one after the other. The .toc file is a binary file, toc probably stands for table-of-contents. It’ll be a list of numbers identifying where each email is located within .mbx. It’s not useful except to the eudora client itself. And I’d guess if you deleted it eudora would be able to regenerate it from the .mbx file.

      There are two technical details that make the climategate email story confusing.

      Firstly, eudora did something slightly unusual. When it received an email containing an attachment, it would decode that attachment and story it somewhere else, say C:\eudora\attach\. It would also remove the attachment from the email it had received and replace it with a simple text link. That means once eudora has opened an email the attachment no longer exists inside the email. There is nothing to stop the users deleting the attachments or moving them somewhere else.

      This all seems a bit daft now, but back in the days when Eudora was written PC were a lot slower and hard disks much smaller. You wouldn’t really have wanted to keep all the attachments, forever, in the eudora client emails.

      The other thing that trips commenters up, is that the backups are not a backup of an email server but are backups of people’s desktop PCs. Today many of us have all our emails held on an email server (IMAP or Exchange) and this server keeps our emails forever. So a backup of an email server’s disks might be useful.

      The CRU setup was like email used to be 20 years ago. Emails arriving at CRU or UEA mail servers would have sat there until people downloaded them to their email client. And then the server would have deleted the server copy.

      Cheers,

      Nick

      Steve: your point about toc files makes sense. It seems the attachments are in Attach directory. They used BackupPC.

    • Posted Jan 12, 2013 at 5:53 AM | Permalink

      I routinely find EUDORA attachments in a directory named “attach” and pictures embedded in emails in a directory named “Embedded”. Other versions of EUDORA may vary.

      Thanks
      JK

  6. None
    Posted Jan 11, 2013 at 3:58 PM | Permalink

    So really these guys are either totally incompetent, or totally untrustworthy.
    If we must assume it’s incompetence, surely all subsequent requests for information MUST be investigated by a suitable independent and suitably qualified individual.

  7. Posted Jan 11, 2013 at 4:57 PM | Permalink

    Things not comprehended at UEA: Eudora, Excel, climate, revision control, ethics, honesty, science, …

    • Mike Restin
      Posted Jan 14, 2013 at 7:57 AM | Permalink

      you forgot excel
      mr

  8. Neil fisher
    Posted Jan 11, 2013 at 5:19 PM | Permalink

    “So really these guys are either totally incompetent, or totally untrustworthy.
    If we must assume it’s incompetence, surely all subsequent requests for information MUST be investigated by a suitable independent and suitably qualified individual.”

    I hope Steve has emailed them, requesting another search in light of “the new search location brought to light by FOIA request 12-139 and the subsequent correspondence with the requester”.
    In any case, it is beyond incompetence IMO for them NOT to have uncompressed and searched every file in the Eudora folder(s), or even the entire backup – as a sysadmin for a hosting company, I can say that the failure to do so is not in any way a technical issue or limitation, and almost certainly was the result of direction to the technical staff along the lines of “don’t look too hard” or in some other way limiting the search by design.
    Frankly, excluding particular files from such a search would likely result in the use of more (personel) time used than a simple search of the entire contents, which could be set up and allowed to run in well less than 1 hour of “people” time. While the size of the folders to be searched would determine the run-time of the search, actually setting up to do the search and collating the results would be relatively simple unless they were specifically spending time deciding what to search rather than just searching the whole lot. Neither are the required tools to do such a search difficult or expensive to obtain – in fact, such tools are freely and easily available for all major platforms as ready-to-use installations and compilable from sources for unusual platforms.
    The only reasonable conclusion is, IMO, that they deliberately wasted the time of the technical people to “pump up” the amount of time they spent doing the search so they could then claim they spent X hours attempting to fulfill the request.

  9. johanna
    Posted Jan 11, 2013 at 10:38 PM | Permalink

    While the technical discussions on this issue are way beyond my ken, what leaps off the page is that a university, and specifically a part of it that is supposed to be in charge of cutting-edge scientific research, can claim to be so ignorant of – email IT! I mean, we are not talking about looking for the Higgs bosun here.

    As a lay person who has managed quite a few IT projects and programs, my reaction is – pull the other one, it plays Jingle Bells.

    • johanna
      Posted Jan 11, 2013 at 10:49 PM | Permalink

      boson. fat fingers.

      • Jimmy Haigh
        Posted Jan 11, 2013 at 11:51 PM | Permalink

        “The Higgs bosun”. I like it. I’m sure Josh could work on that one…

        • Posted Jan 12, 2013 at 7:30 AM | Permalink

          In the gloom of primordial soup
          A sea of lepton and quark
          What weight to give
          Dr Higgs explanation?
          Till CERN shone the light
          Five sigma bright
          Mass data, clear and loud
          “Aye aye, Captain!”

        • Steve McIntyre
          Posted Jan 12, 2013 at 8:00 AM | Permalink

          +1. “Higgs, bosun”.

          Wasn’t Higgs the bosun on Captain Ahab’s crew looking for the white whale? Who went missing. The sequel to Moby Dick was the search for Higgs, bosun. That’s what they teach at the UEA, anyway.

        • Posted Jan 12, 2013 at 8:29 AM | Permalink

          It’s not just that they get science, history, creative writing and their responses to FoI requests, under UK statute, hopelessly mixed up. It’s that it’s particularly bad creative writing.

    • Barclay E MacDonald
      Posted Jan 12, 2013 at 12:06 AM | Permalink

      Yes, all of this is in the context of a public university that states on it’s web page that ” …the Climate Research Unit is a major repository for data…” It is so reassuring that the data is stored with such manifest expertise and objectivity.

  10. barn E. rubble
    Posted Jan 12, 2013 at 8:38 AM | Permalink

    RE: Barclay E MacDonald “It is so reassuring that the data is stored with such manifest expertise and objectivity.”

    Storing the data doesn’t seem to be a problem. However, retrieving it is another story – and don’t even ask about sharing . . .

    As I read thru the last few posts I can’t help but wonder if the foxes can hear the hounds yet?

  11. Josh
    Posted Jan 12, 2013 at 12:28 PM | Permalink

    For newcomers like myself what is the potential significance of this attachment ?

  12. Latimer Alder
    Posted Jan 12, 2013 at 1:32 PM | Permalink

    One can’t help feeling that in the days since Climategate, Hapless Harry (of ‘Read Me’ fame) must have been promoted to be in charge of the ‘technical team’ responsible for e-mails. Their lack of understanding of their own systems seems to be just as overwhelming as it was in 2009.

    I knew nothing of Eudora (until 1 minute ago) but the simple Google search for ‘Eudora backup’ immediately gave access to the following public material that seemed to have eluded UEA for some time.

    It describes both the mailbox and backup numbering convention and is understandable to any technically literate person in less than twenty seconds. It is inconceivable (to me at least) that the tekkie supposedly responsible for making these searches did not avail themselves of this knowledge. Or if they didn’t they are wilfully incompetent.

    http://www.eudora.com/techsupport/kb/2651hq.html

    Steve: while incompetence is usually a simpler explanation than dishonesty, I’m disinclined to attribute the obtuseness manifest in their failure to search the numbered mbx files to mere incompetence.

  13. john robertson
    Posted Jan 12, 2013 at 3:28 PM | Permalink

    Deliberate obtuseness for the purpose of buying time.
    Rules of bureaucracy.
    Keep up the good work, looks like the dike is springing more leaks.

  14. JDN
    Posted Jan 12, 2013 at 4:03 PM | Permalink

    There may have been one bright soul amidst all the incompetence and obtuseness at UEA who kept his head down and his mouth shut. And that one might be our FOIA of CG1/CG2 fame.

  15. Posted Jan 13, 2013 at 9:34 PM | Permalink

    If people are going to test they may want to re-create the conditions of the time. Older versions of Eudora can be obtained.

    Available here: Note dates: http://www.eudora.com/techsupport/kb/2350hq.html

    Cheers

    • Duke C.
      Posted Jan 14, 2013 at 11:08 AM | Permalink

      WillR,

      Good point.

      Eudora version#s are present in a number of CG1-CG2 email headers. Grep using “Eudora” and scan the results.

      I recall seeing version 7.1 in one of the Briffa headers, which is why I used it.

  16. dynam01
    Posted Feb 5, 2013 at 4:44 PM | Permalink

    I have to admit, this whole affair is highly arcane and takes a bit of back-reading to make any sense of. Having done so, however, I’m appalled at the frank stonewalling exhibited by the U. of East Anglia. How anybody can take anything this outfit says, about the climate or anything else, at face value is beyond me. Kudos to CA for the perseverance in pursuing what must have been an immensely frustrating and time-consuming matter!

  17. Duke C.
    Posted Feb 18, 2013 at 2:56 PM | Permalink

    Director of UEA Information Services Jonathan Colam-French responded today to my Jan. 24th. request for an internal review. It makes for some interesting reading. Since Steve Mc has been “mia”, I’ll post up the response for discussion in a different venue when I get a chance.

2 Trackbacks

  1. [...] Duke C Punctures More Attempted UEA Obtuseness (January 11, 2013 01:20:58 PM)CA reader Duke C has some results from his FOI request that look like [...]

  2. [...] http://climateaudit.org/2013/01/11/duke-c-and-more-uea-obtuseness/#more-17331 [...]

Follow

Get every new post delivered to your Inbox.

Join 3,208 other followers

%d bloggers like this: