Besides the University domain there was a domain on the server where backups of the connected PC’s were stored. These included the Eudora attachments.

The suitability of a search process is easily testable, if you use the same process to search for something you know you actually do have present in a backup. It shouldn’t be too hard for anyone to document the search process as adequate and reasonable in this way.

Then you end up with a set of files that can be examined in detail using unix find and grep.

They should *not* be depending upon the backup utility build in search functionality – it could be buggy or enforce policies that deliberately exclude certain files or file types. Unix find and grep are well proven rock solid ways of searching a file system. Not the quickest but certainly complete.

The CompSci department should be able to do this quite easily; its an afternoons job for a postgrad.. No excuse to doing it properly. In fact there might well be scripts available to do this for you.

First post on your very interesting site. Background, professional engineer CEng specilising in technology in telecoms IT sectors, did a spell in management consultant used to doing due diligence, business development blah, but IT literate to say the least.

You have three very pertinent recent comments which should be put together in my view:

i) PennDragon raises several good points which match my experience. His points about PAs, individuals own policy (ie did they keep copies of attachments in separate directory which is what I do for key files especially ones you want to work with) and again what policy was used in the email system itself. Sorry I dont know Eudora but Msoft mail has default archiving settings but again what was their policy and how was it exactly implemented.

ii) the whole back up policy and tools used need to be understood and how it was actually implemented in the real world so that the correct set of back-up files can be searched. For example, full back, partial back up need to put together for the correct dates after it was sent but before it was deleted. A lot of back-up tools mangle the file names so its no good searching for a file in the back up unless you are searching through the back-up tool itself.

iii) GREP is very powerful tool but as you say is a Linux command. There is hope however, attach the server hard drive as an additional drive in a linux system and boot up using your existing system. You can also use a linux live boot system on a thumb drive and boot from there rather than windows (I understand that is the server OS) and run Grep using the key word in the file approach.

Of course the file may actually have disappeared, then you get into really nasty forensic examination of deleted files that still exist in partial form on hard drives.

The what is wrong with asking the guy to find it himself – he is willing?

Much to do to ensure a thorough systematic search is undertaken before it can be concluded the file has truly been lost.

Something like my post above for appropriate values of dot

find . -print | egrep -i 'EDITORIAL|AR4SOR|Ch06'

Trivial test case

egrep -i 'EDITORIAL|AR4SOR|Ch06' <<-EOF
c:\eudora\attach\AW_Editorial_July15.doc.
c:\eudora\attach\AR4SOR_BatchAB_Ch06_ERW_comments.doc.
c:\eudora\attach\Ch06_SOD_Text_TSU_FINAL_2000_12jul06_ERW_suggestions.doc.
c:\eudora\attach\Ch06_SOD_Text_TSU_FINAL_2000_25jul06KRB-FJ-RV_ERW_suggestions.doc.
chapter six (ch06) of this process is somewhat silly
ar4sor is also silly
This editorial brought to you by mister bob
EOF

I am a lawyer of many years experience but hold an honours science degree and I have actually performed similar searches in significant litigation myself.

“However, on the basis that the search terms were sensible, does it make sense that the emails (attested in the Climategate dossier) still exist while the attachments don’t?” Sadly, yes this can happen. I learnt this the hard way many years ago. I lost attachments myself that way. They can be stored on a separate directory.

So unfortunately attachments with emails can be deleted separately from the emails.

However the search seems,prima facie, inadequate, although without seeing the information in detail I can’t say that conclusively. If the emails were backed up through, for example Outlook, they would be in a special Outlook file and would not have been found. I have not used Eudora for many many years so I do not know how old emails may be backed up. The best way I can explain my discomfort is that in one search I performed one important email that only had a few recipients had 31 copies. Copies get inadvertently kept in many ways and in different directories. The organisation may have had automated backing up or in my case at least one secretary was able to read and save emails and backed up her boss’s emails on a regular basis in Outlook .pst files. Secretaries folders should be searched. Unless you know what people could do, searches should not be restricted to particular folders, especially if they may have been forwarded to others.

The other critical comment is I would not place much store in the date. Computer systems automatically update the so-called date and so considerable care needs to be taken when identifying the precise real date of a backup. If these things are put on a backup server, the first question is how the backing up was done etc and therefore where those files might be. eg Was some compression process used on older files or attachments? This would effectively hide them.

Looking for a particular email attachment is perhaps best done by finding a key word or term within it that is reasonably unique and searching the contents of all the files in the whole server back up. This is not hard, you just set it going, go home and get the results in the morning. Care then has to be taken to search all possibly relevant file types.

In short the process is much more complex than people think and their account sounds more like what we in Australia would often derisively refer to the “she’ll be right on the day mate” approach.

Is J grep -i AR4SOR BatchAB

Should be

ls -R / |grep -i 'AR4SOR.*BatchAB'
or better
ls -R / |grep -i '\(AR4SOR\|BatchAB\)'

Differences:
* ell-ess not eye-ess
* needs -R to search subfolders
* needs to search the whole filesystem (“/”) or at least a specified filesystem “/mnt/J”, not an arbitrary subfolder of the current location (“J”)
* needs pipe symbol between the two commands
* grep doesn’t find two words the way they showed. It’s a bit trickier than that.
* I provided two examples: the first finds the two words in sequence. The second finds either word

All they have done is demonstrate that they don’t know what they are doing. (Or else a poor front-line admin assistant was tasked with translating a technical description into human language, and understandably made a hash of it.)

The instructions given to the technician are designed to look like they are cooperating while continuing to pull the wool over ones eyes. “its not where we think it should be, therefore it doesn’t exit”.

What crap.

To think that the UEA doesn’t have an archival policy that protects all its email (inc attachments) is not credible in this day and age.

And the problem is, you can ask all the specific questions you like but it will be like trying to shoot ducks at night. You will be wasting your breath.
The IT systems administrator will know EXACTLY where to find the documents.
Ideally an independant technician should be provided access to that server. Should take no more than a couple of hours to hunt said documents down.

If UEA continue to roadblock this, the next FOI request should be to disclose their archive/backup procedures. Those documents will be sitting on a backup drive or tape…somewhere. If they don’t, it also means the UEA does not have the means to recover from the loss of its email server through hardware failure. Not only is that not believable, if true, would open the university up to a major interruption to its operations.

1) The back-up server seems to be running some UNIX variant (implied by the use of ls and grep).
2) The users’ PCs are running Windows (implied by the use of the Eudora mail client).
3) They only searched their backup for a file with a particular name in a particular directory.

Now, if someone told me that they had a back-up mail server, I would assume it was running the same operating system as the primary mail server and that they are backing up the actual e-mail files, rather than backing up the contents of users’ PCs (such a machine would be referred to as a back-up server but I wouldn’t call it a back-up e-mail server, even if the user backups included e-mail).

On a UNIX server, normally the attachments are stored along with the e-mail itself in a file with a random-looking name. Searching a back-up of such a machine for a particular file name will not find e-mail attachments with those names. Searching a back-up of the user’s PC would but only if they are using Eudora which automatically strips the attachments out. Other e-mail programs such as Thunderbird will store/cache the e-mails in a similar way to the way they are stored on the server, ie, in a database format or with the text and attachments combined in a single file with a non-human-readable filename.

The type of search they have performed only makes sense if they have a Unix back-up server which has backed up the contents of the users’ PCs and the users are running Eudora. In that case, you would have a Unix machine with (some of) the attachments stored with their actual file names, in a single directory, that you could search using an ls/grep command. This is an odd back-up situation; in any other case, those commands would fail to find the attachments even if they were present.

So in summary, this type of search doesn’t make much sense and is unlikely to succeed, based on the information that we have.

I back up the mail server at the company I work for by using the “rsync” program to make a copy of all the mail directories. If someone asked me to find an attachment with a given name in the backup, I would have to search INSIDE the back-up mail files for that text. I imagine most other organisations who have a mail server on a Unix machine and do regular back-ups would be in a similar situation.