Nor has the heat convection via water vapor to TOA with the hyperviscosity and its effect on this transfer been well defined. When you, Jerry, and I tried to pin Gavin down on this, he also reverted back to 1.) model backcasted well; 2.) it agreed with other models; 3.) and the models were right because they got the changes from major vocano eruptions correct.

That said, there are a few sine qua nons of science that are universal. One of them is that models are tested against observables, not against other models. Only climate science seems to claim an exception from this principle. There is no warrant for such exceptionalism. Modelers should be looking at observables to improve their model, not to other models. Model parameter sets ought to be rationalized strictly from physics, not from whether they give good results in someone else’s model.

You by-passed the mention of hyperviscosity. This non-physical graft is necessary to prevent model divergence. Further, Carl Wunsch has noted that ocean models do not converge. He noted that modelers brush aside his queries about that, because the non-converged outputs “look reasonable” (Wunsch’s description). ‘Looks reasonable’ sounds a lot like your “realistic,” and is no grounds for confidence.

With respect to clouds, condensation takes place across microns, not kilometers or even meters, and the microphysics is not well-understood. Nevertheless it is critical to cloud formation. Cloud opacity is dependent, in part, on the presence or absence of aerosols and particulates, and condensation depends on the nature of the particulate — silica vs. soot, for example. These differences can’t yet be modeled. The equations of turbulence can’t be solved exactly and Gerald Browning in this forum has gone through the problems of energy dissipation and upward cascades in some detail. These are not problems of resolution.

You noted that clouds are not well-modeled and stopped there, as though this problem — whatever the cause — is not important to climate projections. Only a few percent change in tropical cloudiness can obviate any warming due to extra GHGs. And yet, whether from resolution or from physics, clouds are poorly modeled. That means climate projections are not reliable. But somehow, this doesn’t seem to matter when modelers publish projected climate futures.

You mentioned “the huge variability in the last model generation’s projections of variables such as precipitation.” But precipitation problems occur in part when the models have been trained to reproduce observed temperature fields. When models are trained on precipitation fields, the temperature variables are not reproduced. These see-saw divergences show a fundamental difficulty with thermal dynamics within the models, including accounting for the latent heat of condensation – evaporation. This problem probably contributes a good part of the error models produce in the TOA thermal flux. When thermal problems are subsumed within errors in precipitation fields, projections of air temperatures must obviously be in error. But somehow, this is never admitted when air temperature projections are published.

Climate science exceptionalism. It’s self-serving and pervades the field.

This thread is about how did an obviously incorrect value occur? If such an error occurred in any software I had produced my first assumption would that there is a bug in the code. Having read a book by Mueller and Storch about modelling this should not be a surprise. I have worked for the last ten years for government in a section than ran a financial systems funding universities and students. The testing was rigorous and very necessary. I expected it should be the same for GCMs but from this thread it seems not! Why aren’t assert statements used, it is common in my experience.

My guess is that it is to do with speed. I have never used a super computer but surely the compilers on these systems are not that much different from the ones I have worked with. You can switch code optimise on or off, if the ultimate goal is speed then you switch it off. This can be only valid if you can adequately test your results are still the same. This is because you are actually giving permission for your code to be messed with. Maybe they are not testable?

My assessment so far is that GCMs are an attempt at a virtual reality of an analog, event driven world with low resolution (relatively) cells and iteration. My assumption is that this is because computers are digital and too slow for anything other than iteration. Beyond this my world had business, analysts, software developers and testers. All told about 40 or 50 people. Business in consultation with the analysts produced specifications for the developers to code from. The software was then tested against test plans by the testers. I would hope a similar process happens with GCMs. Our operational budget was in the tens of millions so I would hope the budget for climate model is much higher than that or am I wrong. I hope not.

Thanks for the direct answer. You are aware that there are rumblings to the contrary out in cyberspace. Its good to know that, at least for this model, some of this repeatability is being handled in a professional manner.

There is one caveat, though: if you’re not running the same compilation of the model on the exact same system as the original research run, you won’t get *exactly* the same results. But, this is the computer science *exactly*; there will obvious be minor differences in the model depending on what compiler you use and there are a suite of reasons why different architecture systems will yield slightly different results.

Yes, I mean “exact” up to the uncertainties wrt compiler versions, machine precision, etc.

(although I’m very interested in setting up a Tesla desktop-supercomputer and porting a simple model to CUDA…)

I know of several efforts, outside of climate science, where atmospheric models are being ported to this sort of environment quite successfully.

Ah, I see exactly what you mean now. Speaking from experience with the CCSM3 (I imagine it would be similar for other models although I don’t work with them), you would only need three things to accomplish this task: 1) the model configuration script (which is just a shell script, although I’ve experimented using Python scripts instead); 2) the same initial data files (which are often times the freely available files published on NCAR’s website); and 3) any SourceMods utilized by the researchers (rather than plugging new code into the model source, the common practice is to copy a Fortran module you’ll modify and place the modified version in a directory off the model root; the model will then use that source file instead of the identically named one in its source).

Often times, researchers will ‘spin up’ the model to perform a transient experiment. In this case, you could also ask for the branch or restart file from which the actual experiment was run. This is probably the most common practice. All of these files are available on the MSS at NCAR, so accessing them is trivial to active researchers with NCAR accounts. Also, it is good practice to make local copies of the datasets, so most researchers would probably be able to send someone copies of the files directly if someone doesn’t have NCAR access (which is limited to active researchers for obvious reasons).

There is one caveat, though: if you’re not running the same compilation of the model on the exact same system as the original research run, you won’t get *exactly* the same results. But, this is the computer science *exactly*; there will obvious be minor differences in the model depending on what compiler you use and there are a suite of reasons why different architecture systems will yield slightly different results. For this reason, the first thing everyone does once they port the model is run a validation test against control data from NCAR’s current supercomputer (bluefire). If validation goes well, you still do a full simulation (decade-century) and compare against control runs on bluefire just to be safe. To bypass this issue, most climate scientists running the CCSM3 will just queue up runs on bluefire. Bluefire is a hell of a lot faster than any other computer that most climate scientists have access to (although I’m very interested in setting up a Tesla desktop-supercomputer and porting a simple model to CUDA…)

I hope this answers your question! Let me know if you want me to elaborate a bit on any part of this answer.

Sure, suppose one produces a set of runs for a paper…what I would call “runs for record”. Are the versions of the code (source and/or binaries) and the set-up files sufficiently configuration managed such that, a year from now another researcher could exactly reconstitute that set of runs down to the exact version of any component of the code base and the inputs?

Since I’m 90% sure I was here first, I think if he comes back, he should change his name.

Now we’re only looking at the tail end of the modeling process. You need to consider the beginning end as well – the core constituents of the model. At this level, we’re not talking about ‘parameterizations’; the core of a climate model is just a suite of physics equations. For instance, since we’ve been discussing the CCSM3, you can read (in detail) about the core physics and dynamics equations in the CAM3 here. A JoC article from a special edition back in 2006 details these equations in even more detail – I can fish it up if you’re interested.

Consider your point about cloud errors for a moment. It’s widely known that GCM’s are deficient in their cloud-modeling capabilities. But keep in mind the ‘why’ to this question – model resolution is too coarse (due to limited processing budgets) to accurately resolve clouds. No one ever discovered a panacea for solving this problem without increasing the model resolution. Thus, is it any surprise that most models utilized similar cloud parameterizations? Considering this fact, is it really that surprising that most models suffer this same flaw? I think it’s far more interesting to consider the huge variability in the last model generation’s projections of variables such as precipitation.

Your point boils down to alleging that the models are unphysical. There is a huge body of literature discussing this topic, and in general, those who work directly with the models disagree with your analysis.

Really? Your workstations/servers do not have ECC memory, are using a Journaling file system and your HDs are not in a RAID setup? Are you serious? You have to be kidding me.

I didn’t mean to say that the hardware is flawed (yes, AFAIK, the systems we use have ECC memory, RAIDs, and so on); the instance I’m recollecting from old PCM runs is that the layout of the file (header + data) was corrupted at some point. It wasn’t reproducible, it was random, and it wasn’t easily detectable. Basically, fixing the files required examining the header of every single file and checking the value of a certain field in the header, and if it was incorrect, then replacing the header (after suitable modification) and rewriting the file. That was a weird instance.

The greatest non-replaceable data loss I’ve ever had (and this is over a time span of many years and literally hundreds of millions [billions?] of I/O operations) was from an archival tape that was damaged and for which we didn’t have a backup copy of the data stored upon it. I believe I already explained that one.

At one time, the model would do a checksum on any output file, write it to archival tape, remove it, read it back from archival tape, do another checksum, and compare the new checksum to the older one to make sure they matched, but that’s a bit overboard, IMHO. It also turns I/O into a major chunk of the model’s wallclock and run time, and since run time is finite, and expensive, the process was dropped. It’s still possible to do it, but unnecessary.