Climate Audit

steven mosher

http://initforthegold.blogspot.com/2007/12/why-is-climate-modeling-stuck.html

Scott-in-WA

#3 and #4:

Having spent fifteen years writing software in the nuclear industry, I have an appreciation for the issues associated with doing V&V on complex software. I’ve also been involved in several code re-engineering projects.

Referring to the discussion concerning GCM models underway in Steven Mosher’s link, it would be a most interesting (and expensive) experiment to take one of the older GCM models running in procedural language, construct a requirements document and a code design document for the GCM under current V&V practices, and then re-code it using a more modern object-oriented language.

Would the two incarnations of the same GCM model produce identical results starting out with identical initial parameters?

My guess is, not the first time around doing the re-coding, and probably not even the second time around.

Larry T

I was tasked with reproducing a statistical package on Unisys computer to replace existing one on a IBM mainframe that the company was getting rid off. This was an important tool to the company that had been in use for many years and was implemented on a statistical software package. Mathematician that I am, I went back to the original statistical equations to implement my solution in a structured programming fashion. After careful testing, we ran a parallel test of the two systems and found a serious discrepancy. Since my program was the new kid on the block, I went and did a careful analysis of the system and could not find any software errors. We then went back to the original coding in the statistical package and found an error in the original code.

One problem I have found in many of the organizations that I worked for is that they forget that when you work with packages such as SAS, SPSS, EXCEL etc. and are constructing in them data. relationships and formulas you are still programming and minor errors that have significant implications. I have had to correct many spreadsheets that were used in producing reports for higher management of organizations for large manufacturing firms, state and federal government agencies.

William Newman

#4: “There aren’t any standard test cases used by atmospheric modelers. I find this utterly astonishing!”

I don’t think this should be so astonishing, though you might want to be astonished by some other stuff. (And if my discussion of the other stuff is insufficiently close to the original topic, it’s OK with me if some moderator-type person deletes all but the next two paragraphs.)

I did some undergraduate research, and my Ph. D., in simulations of biomolecules. There are some standard sorts of checks that people tried to do — e.g., compare to the atomic positions found by experimentalists in really-well-studied macromolecules. But even with complete scientific integrity there can be vexing practical obstacles to making satisfactory “standard test case” choices. For example, the stuff which is known most clearly can be annoyingly far from the regime which is of the most practical interest. For the proteins-in-solution problem which motivated my work as an undergraduate programmer, the experimentalists had very good results for protein crystals, but they have to impose pretty weird harsh conditions (like extreme salt concentrations) to get them to crystallize, and what we care about much more in practice is how proteins behave in milder environments more like the insides of living organisms.

I am generally underimpressed by the climate science folk — e.g., yesterday I was idly wondering whether a page like http://www.realclimate.org/index.php?p=11 would look more like Darwinism or Lysenkoism to someone who doesn’t already have my cynical view. And I generally support some common criticisms of modelling, like Steve McIntyre’s remarks somewhere about how blurring the distinction between measured and modelled/extrapolated results is a well-recognized sin in mining-related advocacy and should be here too. But I don’t think you should necessarily be shocked at the lack of standard test cases.

One less-standard criticism you might want to be shocked at, or at least very seriously disturbed by, is lack of attention to other kinds of verification of predictive power. It is weird for me, coming from molecular modeling (and a general interest in the history of science), to see people paying so much attention to matching a very small number of observations given the large number of parameters in the model. It is particularly weird because I currently have _The Minimum Description Length Principle_ checked out from the local university library (for my own machine-learning reasons, not for policy advocacy reasons). In my experience, when scientists have a theory which they believe to have a lot of predictive power and only a few obvious numbers of practical importance to test it against, they look intensely for less obvious practically unimportant observations to test it against. These numbers may not come up in, e.g., testimony before Congress, since they may be honest-to-goodness very hard to present in a few pretty pictures. But they come up all the time in more technical discussions.

E.g., in molecular modelling, people got very interested in higher-dimensional nuclear magnetic resonance data. Such NMR doesn’t naturally give the same kind of pretty every-atom-in-its-fixed-place pictures as X-ray crystallography, but it gives a great volume of weird detailed little constraints and correlations which could be cross-checked against a model. And even if what eeveryone cared about in practice was some simple high-level summary like the function of a protein (e.g., something like the O2 affinity of hemoglobin), nobody would present a new model with hundreds of parameters and focus only on its fit to a few-parameter curve of bound O2 vs. partial pressure of O2. If you can’t find datasets with very large numbers of degrees of freedom to compare against (like higher dimensional NMR, or various other kinds of spectroscopy) caution is in order. And if the modeller isn’t terribly interested in finding such datasets, perhaps great caution is in order.

Without knowing much specific about climate models, coming from chemical modelling I’d expect two general things of them. First, they’d regularly refer to at least one huge family of checkable things about regional distributions and correlations and so forth, rather larger than “the number of adjustable parameters” (a vague concept, but one which can be made more precise, as e.g. in the book I mentioned) in their models. Second, if they’re confident their models are precise enough to pick out interesting nonobvious phenomena (e.g., famously, that the net temperature response to C02 concentration is considerably higher than the first-order effect) then even if the mechanism doesn’t leave clear footprints in the current experimental datasets, it should be leave footprints in some imaginable experimental datasets that the climate folk are now passionately longing to measure (I dunno: nocturnal concentration fluctuations of an ionized derivative of CO2 over the Arctic and Antarctic). I don’t absolutely know that these things don’t exist, but I’ve spent some hours surfing RealClimate without noticing any hints that they do. (And if molecular modelling had been subject to the same level of informed controversy as climate modelling, I’d’ve expected that a few hours surfing the RealBiomolecule site would have given many such hints.)

I’d be reassured to see climate modellers hammering on how detailed interesting regularities in experimental data (existing or wished-for) are explained or predicted by their model: something like the (honestly exasperated) way biologists refer to all the megabytes of detail revealed by DNA sequencing and other molecular biology which just keeps matching the constraints of Darwin’s model. So far, that honestly exasperated attitude hasn’t come through as strongly as I’d like.:-| I don’t expect the climate scientists to be angels: I’ve followed the creationism dispute for decades, and honest competent biologists are not immune to the temptation to be exasperated at distractions like their critics’ funding not coming from the holy NSF and their critics’ almighty credentials not being biology degrees. But biologists seldom get so fascinated by distractions that they forget to refer to fundamentals like the enormous volume of detailed regularities in nature that the consensus model matches.

Larry M

Craig, I had days like that.

However, I am not so sure that the GCM modelers would see that as a problem. 🙂

Ron Cram

re #6,

Geoff,

I think I am mostly agreeing with lucia here but I say it differently. The payback is in learning what we don’t know about climate. The models will never have predictive power.

stan palmer

The January 5, 2008 edition of “New Scientist” contains a report on the work described in the paper found at http://www.arxiv.org/abs/0712.1187 . The research concens the effect of the effect of using tools like Mathematica of the physics reasoning skills of students. There are two stage in this. The first is to use physics to develop a strategy to solve a problem. The second is to code it. What the researchers found is that it was the physics that was deprecated in this endeavor . Coding took on paramount importance. One set of students who chose an incorrect strategy to solve a problem in quantum theory repeatedly re-coded the incorrect physics strategy in an effort to correct their mistake.

The final quote from the “New Scientist” story is perhaps illuminating.

“We often see”, says Elspeth McKay of the RMIT University in Melbourne, Australia, “that automated tools tend to block people’s capacity to see or know the broader context of the problem they face.”

One example of this is to take the simulation as reality and believe that perform experiments on it are representative of reality.

Hank Roberts

I didn’t find contact info for the page
http://www-personal.umich.edu/~cjablono/dycore_test_suite.html

If you know them, someone, please tell them they’ve got a typo:

“… packages on the sphere if highly desirable. …”

———————–
“Opposition is true friendship”
— William Blake, Proverbs of Hell

lucia

Judy–
I agree with you that V&V is done in climate science. But, I’m in the position of knowing climate science V&V is not done quite the way it’s done in nuclear or safety work. (In some sense: thank heavens! In another sense: climate science could improve.) 🙂

But, when I say the codes aren’t generally subject to the type of V&V in nuclear applications, they generally aren’t. My husband Jim worked on ARM. (You also did a project with Sheba, right?) I’ve worked on a range of projects; he has. We both know the range of what is done.

I’m tempted to write more, but I can’t figure out how quickly compare and contrast and also explain that
a)climate science could benefit from more formal V&V procedures but
b)it should adopt the full nuclear safety V&V requirements.

The main benefit of the more formal V&V than typical of academic peer review is not scientific: it’s to communicate with technically competent non-climate scientists who must make up their minds before casting ballots during elections. I think some of the violent disagreements over V&V here and at RC come from each side not understanding what formal V&V is for.

The reason to avoid full nuclear safety requirements would be clear if you read how amazingly detailed they could be. The most formal V&V requirements are reasonable when a single mistake in a single application can cause an accident that kills someone. They aren’t quote so necessary when the results of a study will simply add to a body of literature that helps us gain understanding about science but does not lead to any particular decision immediately. Some of the V&V demands I read are too extreme– but it is regrettable that more formal V&V was not adopted by climate modelers.

But, in some sense, that’s in the past. What would be nice is if these methods of testing were adopted and some program began encouraging or requiring some formal methods of documenting the V&V for codes.

Al Fin

Judith, it is the lack of standardized tests to evaluate the cores that concerns most people here.

MarkW

Judith,

If the cores have been validated, then the modelers should have no trouble providing the documentation to prove it.
If they can’t provide the documentation, why should the rest of us just take them at their word?

Larry

Interesting discussion on V&V. The problem, in a nutshell, is that there’s never enough money to do the science right, but there’s unlimited money to react to the science that wasn’t done right. We’re missing a step in the process. The way the UN and most of the world is doing it, is according to the following formulation:

1. Survey the science.
3. Act.

What’s missing is:

2: Review the science with a fine toothed comb. The V&V, or what Steve calls the “engineering report” should be done done between steps 1 and 3. If they don’t have the money to do this, they have no business enforcing the unfunded mandate implied in #3.

I’d like to think that this was a colossal accident, but the charter of the IPCC specifically forbids them from conducting original research. That means they CANNOT influence the process of the research. It’s almost as if they designed a system that specifically forbids quality control.

Judith Curry

Re V&V, probably the model with the best documentation is ECMWF (which is the same dynamical core and mostly the same parameterizations of ECHAM climate model). Here is the link to the full archive of their technical notes. Read all this and let me know if you have confidence in the model.

http://www.ecmwf.int/publications/
specifically, the technical memos and technical reports

steven mosher

RE 17. HANK! Googlemaster! you find them and you contact them.
It’s in the white pages.

steven mosher

RE 16.

Easy question Dr. Curry. If GCMs are used to asses, base, craft, create, change, construe, delimit,
determine, establish, finesse, form, guage, influence, invalidate, inform, judge, juggle,limit, maintain,
manage…… public policy, you will agree that they should pass a standard test and actually graduate.

No GCM left behind!

Jaye

b)it should not adopt the full nuclear safety V&V requirements.

Maybe not but I would wager that most of these models haven’t even been verified. If I read J. Curry’s assessment right, that is what is being proposed a standardized set of cases that begins to touch on the subject of verification. Validation is years away….which leads me to ask: How can anybody depend on these things for any type of conclusion at all? If I used a sim that wasn’t V&V’ed to help evaluation/design the devices I’m involved with, my head would be metaphorically taken off at the shoulders. Guess that’s the difference between engineers that have to actually make something work and researchers whose work is confined to bickering back and forth in journals.

Mike B.

#31

This one caught my eye. Ahhhh. The Cooley-Tukey FFT.

How is the model performing at prediction?

lucia

Jaye– I’m an engineer. There have always been engineers who use codes and software that don’t go through formal V&V. There always will be. It depends what they are doing and how it’s being applied.

V&V is rarely emphasized on exploratory research.

Judy — “After reading all these (going back to about 1980), tell me whether or not you have confidence in the model.”

With regard to the skeptics screaming for V&V you posed an irrelevant question. The question should be: “Tell me whether this is V&V?”

Alas. While the things you point to are interesting, and show that good work is done in weather prediction, it does not resemble any sort of formal V&V. Some I saw would fulfill part of Verification. I saw absolutely nothing that would remotely fulfill validation in any formal way. The validation is stuff you would have expected all those people to do, but would normally find boring to read. If it were in those documents, you’d consider the material filler. If you put it in a peer reviewed article, the peers would either reject your manuscript or tell you to take out the filler.

But, unfortunately, this validation is important when things switch to guiding policy. One reason this is important is simple: a sizeable fraction of skeptics are unconvinced because they want to see this stuff in black and white. They know the are required to provide it when they need to convince regulators. They didn’t make up those rules — they know skeptics won’t just take their word they did the validation if they didn’t document it. (They know some people won’t do the Validation and they know sometimes things go wrong do to lack of validation.)

The people who will not feel confident without V&V need to see V&V documents. An infinite number of good science documents won’t make them feel confident.

Larry T

#34

Boy do I need to review my matrix algebra texts

Kenneth Fritsch

Re: #36

But, unfortunately, this validation is important when things switch to guiding policy. One reason this is important is simple: a sizeable fraction of skeptics are unconvinced because they want to see this stuff in black and white. They know the are required to provide it when they need to convince regulators. They didn’t make up those rules — they know skeptics won’t just take their word they did the validation if they didn’t document it. (They know some people won’t do the Validation and they know sometimes things go wrong do to lack of validation.)

Would not the people charged with running these models want to validate them in some formal manner regardless of what potential skeptics might say or want? Or would they argue that validation is a waste of time like an unfortunate TV network news reader did a few years ago.

lucia

Ken Fritsch

Would not the people charged with running these models want to validate them in some formal manner regardless of what potential skeptics might say or want? Or would they argue that validation is a waste of time like an unfortunate TV network news reader did a few years ago.

Competent responsible people running these models would validate things. The question is the level of formality. If an unfortunate newscaster suggested validating itself is a waste of time, that unfortunate newscaster possibly doesn’t understand what’s can go wrong when coding.

To give an example:

Supposed, for some reason, I write a subroutine that to calculate

T= T(o) exp(-t/tau) in excel.

After I write it, I might want to validate and do that several ways. One might be to enter time in one column of a cell, enter various the formula in a cell, passing both time and tau. Then I might plot.

That’s sort of a validation.

So, do I do this informally: I just do this routinely and look?
Or do I do this formally at the thermo-nuclear level which involves:

a) I print the chart out,
b) I print the data out.
c) I write a 4 page document discussing the equation, and printing out the VBS macro in excel.
d) Format the docunett nicely so a human can read it.
e) Hand the document to a competent individual with at least a BS who will read it and check that my formula correctly calculate the intended value. The other engineer is required to pick a few values and calculate, and attach his results.
f) Have then initial this, put it in a file, which will later be stored by the Richland Field office of DOE.
g) Have the field office guy come by and pass his geiger counter over the files before storage. (Not making this up. It has nothing to do with V&V. Evidently, radioactive stuff once ended up in storage.)
h) maybe, depending on the project, have this be subjected to additional reviews.

a-f is the thermo nuclear validation and verification. My simply checking is informal.

I’ve obviously picked a simple example involving software and I’m describing extremes. (It’s also unlikely that one equation by itself would go through V&V on in isolation.That would only be V&V’d if that two line macro were used in something else. The whole thing gets v&v’s collectively.)

That said, sometimes just me checking my work is enough. Other times, the thermo-nuclear V&V is required.

Climate science falls in between. In many cases (particularly many documented in the efforts Judy pointed to) the proper level is closer to the informal method. The researcher checks.

For actual GCMs, now that it’s apolitical issue, something more formal is required.

The thermo-nuclear V&V would be overkill.

Argonne National Lab actually has guidelines, and the “thermo-nuclear” V&V would not apply to GCM’s.

John Creighton

#43, I wonder at what point they really start to fail? Do they start to fail when they start doing ideological things like putting in too much positive water vapor feedback and then trying to fix the model by adding aerosols that aren’t really there.

I think as far as basic circulation goes we should understand the physics enough to produce reasonably accurate models given that much more energy is transfered trough radiation then trough convection.

jae

I’m late to this party and have not read all the responses. But, haven’t the models already been tested and shown to be way off target? The SH is not heating as “required” by the models. The troposphere is not heating faster than the surface as the models suggest. The models have not tracked temperatures since the 80’s, especially during the last 9 years. How can anyone support these things with a straight face?

Judith Curry

Hansen goes beyond the IPCC. The IPCC is much more conservative. Personally I agree with Smith’s interpretation, we have not done the full analysis of model uncertainty. But again the IPCC is pretty conservative, it states that the range from its scenario simulations are “likely”, implying > 66% (and therefore that there is 24% probability that the temperature change will be outside the range. So I don’t think the IPCC is inconsistent with what Smith is saying, but some of what Hansen says is.

Jaye Bass

RE:42

Looks like A-F is just verification.

Mosher is right. The sort of code that potentially influences trillions of dollars of public spending should be developed to a standard that is much better than the Level 1 amateurish stuff I see from most academics. Requirements, design, coding standards, documentation, peer review, unit/integration test, etc and at least verification should be the minimum standard for this type of project. Making policy decisions based on code that does not reflect that sort of pedigree is foolish.

Frank K.

Back to my question, which wasn’t answered…

“The bottom line is that I think the U.S. climate models that are been run at resolution 250 km or higher almost certainly have robust dynamical solutions (the real problems are with things like the convective parameterization).”

Again, is there a report or paper which provides proof of the foregoing statement? And what is meant here by “robust”?

The reason I ask these questions is that I am convinced that there is a fundamental limit to the predictability of the dynamical “core” equations (continuity, momentum, energy) due to the ** nonlinearity ** of the system. Depending on what is meant by “robust”, I would venture that modern schemes for integrating the dynamical core equations in climate models are no more “robust” than in those in the NWP codes. That is, of course, if you believe that climate prediction is an initial value problem (as I do).

I looked through the archive of technical papers and reports at http://www.ecmwf.int/publications/, but didn’t find anything to support Judith’s claim above. Perhaps she can provide a mathematical proof of this herself to settle this once and for all.

Finally, are we being too hard on the climate modelers? After all, CFDers like myself can publish papers on esoteric topics like numerical prediction of steam droplet nucleation, and no one asks us for stability proofs or verification studies. Then again, most engineering CFD analyses are not used to construct apocalyptic fantasies such as the infamous “Climate Change and Trace Gases” by Hansen et al.

Click to access 2007_Hansen_etal_2.pdf

If we are to submit to the will of the IPCC and make the costly changes to our society and economy recommended in the policy documents, I believe it is not asking too much for rigorous validation and verification of the software upon which these policy recommendations are based.

Scott-in-WA

#54 Tim Ball: All this is a very nice sophisticated discussion about testing the models but in my opinion ignores the much more fundamental issue – the data base for the model construction. Surface data is totally inadequate in space and time for almost every part of the Earth. There is almost no upper air data of any consequence. There is talk of testing one column, but which one. I doubt there is one with adequate data in all dimensions even in the 250 km scale grid.

The GCMs are not going away and will eventually act as the recognized formal basis for justifying many trillions of dollars in anti-GHG measures. So these discussions concerning GCM software V&V issues are far from being an academic exercise in software V&V theory.

Implementation of a successful V&V philosophy for the GCMs implies that a full lifecycle approach must be taken in developing and managing the GCM software and its employment while in production operation. Elements of the full life cycle approach include a Conceptual Design Document, a Functional Requirements Document, a Detailed Design Document, Code Construction documentation, a V&V Testing Document including unit tests and all-up integration testing, Production Implementation documents, Ongoing Software Maintenance and Enhancement documents, and Code & Documentation Configuration Management.

If a successful V&V program is ever to be established for the GCMs, then there has to be a precise, foundational decision made as to how that V&V program will be defined and structured for GCM software applications, and as to how the V&V program will be applied in actual practice in terms of integrating specific software V&V activities with the other software lifecycle processes as listed above.

As a starting point in bringing some kind of order to this rather chaotic V&V discussion, let’s define some basic terms. Verification and Validation (V&V) can be thought of as two related fundamental questions: “Did you build the right thing?” and “Did you build the thing right?”

For the GCM’s, the question of “Did you build the right thing?” has to do with whether or not the design basis assumptions for the software are appropriately chosen so as to achieve the desired functional utility of that software. For GCMs, these design basis assumptions might include first-principle physical equations, assumed IR spectral behavior of gases, fluid dynamics principles, thermodynamic principles, and more technically-complex assumptions concerning the behavior of clouds, aerosols, non-C02 greenhouse gases, etc. etc. One also has to precisely define just what the “functional utility” of the GCM software is expected to be. Is it to be a simulation of actual physical reality? Is it to have predictive capabilities? These further questions are all part of the fundamental question “Did you build the right thing?” All of this information must be documented in their appropriate places within the GCM design documentation, otherwise there is no practical basis for implementing a successful V&V approach.

The second fundamental question of “Did you build the thing right?” has to do with whether or not the software properly implements the design basis assumptions and the specified software functionality, as these are described within the software design documents. I see no alternative but that the mathematical equations as implemented in the GCM software must be unit tested to ensure they are properly doing their calculations. I see no alternative but to design tests which indicate that the sub-module interactions are operating as designed. I see no alternative but to design load tests that give some indication of the software’s behavioral stability on the hardware platforms it actually resides on. I see no alternative but to devise all-up tests that evaluate the software’s overall ability to fulfill its design basis criteria, as that criteria is described throughout the design documentation.

Generally, in the world of nuclear-grade software, the twin questions of “Did you build the right thing?” and “Did you build the thing right?” both have equal levels of applicability as they affect a Pass/Fail evaluation decision.

But is this so for the GCMs?

In nuclear, we usually know fairly well what the question “Did you build the right thing?” means for any given technical situation, and so managing uncertainty in the design basis requirements is a fairly straightforward process. The GCMs are a very different story, in that managing uncertainty in the design basis is a much more difficult and contentious process—a process which will always be difficult and contentious because of the inherent uncertainty of the climate science on which the CGMs rest.

This implies that climate science uncertainties which affect the GCMs design and operation must also be documented as a formal part of creating the design basis criteria against which the V&V processes will operate. Such uncertainties might well include the issue that surface data might be totally inadequate in space and time for almost every part of the Earth, and that there is little upper air data of any consequence.

bender

#56
Do you have a link? More context is needed.

Pat Frank

#49 — A 95% physically accurate climate model couldn’t predict 20 years on. One that’s 66% accurate couldn’t predict two years on. But the conservative IPCC extends its predictions out a century. It’s all so wildly improbable that it’s no wonder no one pays attention. It’s too embarrassing. And so objective nonsense is discussed with sonorous gravity.

Geoff Sherrington

Re # 43 lucia

What you write often strikes a chord with me, perhaps because we both have nuclear safety backgrounds and a better idea of the consquences of error.

Three matters concern me, as I’m sure they do you. First, there is a paucity of reliable data upon which to build any framework for testing, V&V or even simple proof of physics and chemistry. Like others above, I suspect a cell size of well under 200 km will be needed, maybe even smaller to allow an extrapolation to a limiting case (like the missing exponent in how long is the coastline of Britain, Mandelbrot). In the non-temporal case of mineral resource work we used weighted search ellipsoids in the tens of meters range.

Second, the danger in providing a standard set of tests that all models can pass is that all models will be tweaked to pass the test. Maybe urban myth, but whem Chairman Mao decreed there were too many rats in China, he ordered each person to take a rat tail to a repository every week for a payment. In a short times, farms had sprung up to breed rats to take to the repository. Even now there are data comparison projects between GCM modellers to see how close they are to each other. The real question is, how close are you to truth? Look at this set of numbers for Australia’s CSIRO model performance compared with the average of 22 models (data for temp change at various altitudes, espressed as millidegreees C per decade).

Aust. Mean of 22 GCMs.
163 156
213 198
174 166
181 177
199 191
204 203
226 227
271 272
Then the figures start to diverge a little more.
307 314
200 320
255 307
166 268
53 78

See how close the models can be?

Third worry is that modellers full of confidence after passing tests will be more insistent that they are right and push the political cart. Not much we can do about that.

I’m all for improving the quality of data, but the numerous instances of wrong data even reported in CA would make an aircraft unsafe to fly and a nuclear reactor a good place to keep at a distance. I keep repeating that the scientific method is in places and at times being used in the wrong sequence; and that failure is a high probability because of this. There have to be more microexperiments to put better estimates on variables and shapes to equations.

MarkW

I would also like to see a rigorous justification for every parameterization put into the models.

The statement “it works when I set it this way” doesn’t cut it.

Judith Curry

#50 The IPCC never said the science was “settled”. Their main conclusions are:
Most (implying >50%) of the warming in the latter half of the 20th century can very likely (>90%) be attributed to anthropogenic burning of fossil fuels.
Re the climate model simulations, the state
It is very likely (>90%) that there will be more warming in the 21st century than in the 20th century.
The ranges they give for the climate model temperature increases in the scenario are ascribed as likely (>66%)

When people say “the debate is over”, etc., this is not the IPCC saying this. the debate is over in terms of whether humans are contributing to climate change. the magnitude of the contribution, and what that means for the future of course has uncertainty

Scott-in-WA

#58 Tim Ball: All very nice but in answer to your questions “Did you build the right thing?” “Did you build the thing right?” I would have to say you can’t build anything if you don’t have the building material, in this case data.

Dr. Ball, based upon your own understanding of how the GCMs are constructed, would it be accurate to say that the models—as a matter of necessity—internally generate their own “observational” data as a substitute for the observational data which is not currently being collected from natural climate processes?

Peter Thompson

Dr. Curry #65,

When people say “the debate is over”, etc., this is not the IPCC saying this. the debate is over in terms of whether humans are contributing to climate change. the magnitude of the contribution, and what that means for the future of course has uncertainty

I agree with part of your statement, i.e. humans are contributing to climate change. Yes but so are termites, pine beetles, codfish, cows, pigs and every other living thing on this planet. What the models can’t do, let alone get near magnitude, is get the sign right. If computing sophistication was at today’s level back in the 70’s, GCM’s would focus on human caused cooling through aerosols and such. It was getting colder. Models would have predicted continued cooling. They are bent after the fact to match what already happened.

Boris

Radiative transfer codes are being evaluated very thoroughly in the context of the ICCRM program (not exactly sure what the acronym is

ICRCCM

Also, the more recent I3RC

Dan Hughes

re: #68

“First, no one has any idea how to do V&V on a model of the complexity of a climate model.”

and

“So climate models are much more complex than other models being discussed here, and …”

Both of these statements are not quite correct. Especially, the first V in V&V, meaning Verification, can be successfully carried out for any piece of software no matter how complex the physical phenomena and processes modeled. In its most simple application, the coding can most certainly be Verified. That is, the coding can be shown to correspond to the mathematics and logic that were specified to be implemented into the code. In its second application, Verification that the numerical solution methods do in fact lead to solutions of the continuous equations, is also possible. More difficult, but not impossible. Of course either or both of these Verification activities might lead to the conclusion that the null hypothesis, correctness, is shown to be false. Then the problems must be fixed.

It is extremely depressing to see someone in the climate-change community trying to argue that Verification of computer software is basically an impossible task. That does not favorably reflect on the attitude of the community relative to usefulness of its software products. In many engineering and scientific disciplines such an outlook would lead to immediate dismissal of all calculational results presented by the community.

Climate models are not the be-all and end-all of complexity of modeled physical phenomena and processes. They are not the ultimate and creme-la-creme of state-of-the-art mathematical modeling. The natural evolution of all modeling and computer codes is toward more and more complete descriptions of, and fidelity to, the underlaying physical phenomena and processes. Consider a supersonic aircraft releasing external stores while executing a complex flight pattern with the objective that the stores must be placed within an effective distance of their targets.

The second V, meaning Validation is another, and somewhat more difficult, problem. But to say ‘no one has any idea’ is to underestimate the extent of the work and associated successes that have been achieved in problems of equal, and sometimes greater, complexity.

And, finally

” … and their projections are not being used for deterministic decision making by major policy makers.”

ok, put a statement that significantly qualifies the usefulness of Chapters 8, 10, and 11 in the WGI IPCC reports. Or eliminate these completely. Based on my present understanding of the attitude of the climate change community relative to V&V and SQA of its software, the latter seems to be the most scientifically sound decision.

lucia

Dan Hughes–

You are correct that Verification can always be done. This has absolutely nothing to do with complexity.

You are also correct that Validation is complex confusing part. But, strangely,though Judy says V&V is impossible, she also points with pride to those aspects of validation that are done. Validation is what peer review journals appreciate; verification is not. It’s boring, boring, boring, boring from the point of view of science.

I think Judy’s incorrect impression that formal V&V is somehow impossible for GCM’s may stem partly from the fact that some posts in this thread convey the idea that that formal validation requires pre-specified procedures imposed on all. It doesn’t.

Validation provides quite a bit of flexibility, permitting specialists to decide what is validation in the context of the needs of a project.

Given their application, validation documents for GCMs would certainly be longer and more detailed than the peer review documents or the reports Judy is pointing to. You don’t get to send readers on too many goose chases pointing to bits of theory scattered all over the place; you re-capitulate a lot, while still citing where the originalsource. Re-iterating places information right there in front of decision makers and that, not novelty, is key in a validation document.

You can’t fail to document key bits simply because peer review journals don’t care about that bit or would tell you that bit is already published in another document. There are plenty of other ways to publish: laboratories have laboratory reports. Some universities have University Reports with document numbers. UofI has numbered documents for university reports. So do a number of other universities– I know because I’ve gotten them occasionally. I know I have one from Stanford. Judy will recognize that these documents are often considered “grey literature”. With regard to tenure decisions, it often doesn’t count.)

Judy suggested that GCM’s aren’t used for guiding policy– a point many dispute. This is a nebulous areas because it depends on point of view.

With regard to the funding agencies goal (NASA for example) the justification for the GCM’s in not to drive policy. So, when the scientists fill out any little checklists (as we would do at a national lab), they can simply say the goal of the work is not to drive policy, doesn’t impact safety etc. The work is exploratory from the point of view of the funding agency. This cuts out a lot of work, lets the scientist do the fun science stuff. And, technically, from the point of view of the person paying for the work, is even true.

(I fill out forms of this nature all the time. If I were working on a GCM, and my funding agency didn’t plan to directly guide policy, I’d check that it doesn’t drive policy. Because the question implies “directly drive policy”. NASA doesn’t decide how to responde to climate change, so the work doesn’t drive policy from their point of view. )

But, the reality is that, unlike a project to further develop string theory, GCM results will ultimately be used to drive policy. They are being used that way now; some scientists are putting together documents advocating political decisions. (I don’t have a problem with this, provided they say they are doing this and they don’t modify their results to promote their advocacy. But the appearance can be bad.)

But, from the point of view of funding this is an indirect effect.

In any case, because the reality is these are used in policy discussions, as a practical, political matter, the system enforcing V&V on projects that would benefit from them breaks down.

We end up in the situation we are in: the public (aka voter) aren’t provided proof of verification. And, though scientists are more interested in documenting validation than verification, members of the public are often very, very very strongly swayed by lack of formal written validation in black and white.

As someone above said: It’s a matter of trust. When I did things for nuclear work, the individual clients always trusted me personally to check things. Had they not, they wouldn’t have given me a task. But the system, and the American public, who don’t know me from Adam, didn’t leave verification to trust. I had to sign seal and deliver certain documents.

In some projects, verification must be formalized, and provided on paper, in black and white with signatures. I say this as a person who loathes paper work. But climate science has gotten to the point where task which is loathsome to people who are fascinated by physics, not paper work, must be done.

gb

Hi,

V & V is perhaps possible to ensure there are no bugs in GCMs (but probably not easy because of the nonlinearities). But I guess Judith refers to the validation of GCMs with observations in order to see if the physics and the parameterisations are correct. This is another topic and not a trivial exercise. How should you do that when there are chaos/internal variability and incomplete observations?

lucia

gb–

This is another topic and not a trivial exercise. How should you do that when there are chaos/internal variability and incomplete observations?

The goal of V&V is not to prove something is absolutely correct. The difficulty you describe is shared by all codes involving transport phenomena. It’s most certainly present in codes involving thermal-hydraulics for nuclear applications.

If the modelers said their goal was to predict averaged behavior in a particular range of circumstances, then all that would be required is to compare the predictions to averaged behavior in those circumstances.

In the event this could not be done adequately, or there were deficiencies in the fidelity of the predicitons, the validations documents call these out very explicitly so that users would know better than to expect the models to describe something they can’t describe.

In contrast, weaknesses are often skirted in peer review documents, where the goal is to describe new things we can do not reitterate the fact that we still can’t do some things well.

Models & codes don’t have to be proven perfect to be validated. However, their strengths ,weaknesses and limitations need to be stated explicitly and unequivocably.

If these validation models existed for GCM’s, skeptics would be able to find the paragraph where modelers describe what models can’t do and what you should not expect from them! (Once again: no one likes to write these things. It’s no fun to explain what your model can’t do!)

Larry

Oh, and Lucia, #75 is an excellent summary of the problem that basically puts flesh on what I said in #25. Somewhere between “let’s survey the literature and report the state of the science” and “let’s do something radical”, they left a step out. I’m not saying necessarily that the scientific community has the responsibility to perform that step; it’s up to the political leadership to make that decision. I do, however, fault certain scientists for a) failing to communicate to the political leadership that this step is necessary, and b) in many cases outright denying that it’s either necessary or even possible.

I think that’s the heart of the whole dispute here. And yes, I agree that it can be done, and anybody who claims that it can’t just doesn’t want to be burdened with the task of doing it, or the task of learning something new.

Judith Curry

I am getting very little sense that most of you have any understanding of the complexity of climate models. How to treat the unresolved degrees of freedom in any model is a huge challenge; in the case of climate models this includes things like clouds, and is at the heart of the so called parameterization problem.

The issue that concerns many of you is not technically V&V but whether or not the models are “perfect” enough to prescribe policy. The models are not perfect and are not being used to prescribe policy. They are being used in the assessment of risk that we are facing from global warming. Susann on other threads has described how policy making actually works

Re V&V, here is an interesting overview on the subject.

Click to access 016.PDF

there are many different components and ways to do this. I suggest that the V&V discussion be framed in the context of the validation techniques listed in this document. In the case of the ECMWF model, the V&V is very clear (read the technical notes, memos). If i had time, i could go through each of the ways of validating a model, then refer back to the relevant technical memos and notes.

MarkW

Models are the source of the scary scenarios that predict large increases in temperature. There is no other source for these projections.

The claim that models don’t influence policy decisions fails the laugh test.

Spence_UK

It is very likely (>90%) that there will be more warming in the 21st century than in the 20th century.

The IPCC claims are delusional here. There are so many uncertainties, ranging from the projections to the credibility of models, to assign 90% probability just to 21st century warming – let alone 21st century warming exceeding 20th century – is massively overstating certainty and is not compatible with the claim of being a conservative document. It is a narrow document, yes, an advocacy document, yes – but conservative, no.

The claim above is akin to me producing a model to predict the future of mankind, projecting forward that humans will evolve to be taller in the 21st century, and the rate of change will increase compared to the 20th century, with 90% certainty. If I were to make such I claim, I personally hope my boss would fire me and encourage me to become a mime.

The only reason the IPCC can get away with it is that by the time the extent of the delusion becomes apparent, everyone involved will have “moved on”.

Footnote: I quickly checked the SPM for context and note that Dr Curry left out an assumption in the above quote that GHG emissions continue (at least) at present rates. I do not think this assumption affects my position on the statement.

Tim Ball

I can’t get the link function to work. Simply go to Wikipedia under GHCN Temperature Stations.

Judith Curry

Dan,

Can you clarify this statement, not sure what you mean

“One major aspect that I’m having problems dealing with is the apparent lack of application of fundamental processes and procedures to these problems by the owners of the products.”

One of the interesting things about this thread is having people that are familiar with models from other fields discuss these issues. I appreciate your statement
“From what I read here there is a great deal of understanding, in depth, of the complexity of both the physical phenomena and processes, the mathematical problems associated with both the continuous and discrete equations, numerical solution methods for the latter, and designing, developing, building, testing, and applying computer software that has all this stuff in it.”

I am dismayed by some of the comments by people who are ready to reject the models based upon something that seems ” fishy” without having any real understanding.

The reason i am spending time on this thread is that I would like to see people with some technical background (and possibly even some modeling experience in another field) develop an improved understanding of the credibility of climate models, and their outstanding issues. Lets face it, some climate models are a lot more credible than others, having higher resolution, more advanced physical treatments, better numerics, and a more extensive V&V process. I put the ECMWF modelling system out there as the one that is generally regarded to score the highest in this regard (this does not necessarily mean that a climate model with the ecmwf atmosphere will provide the “best” prediction on century time scales, that is a whole different issue).

Larry

There’s an obvious cultural divide here. Here’s the rhetorical question to consider: how many dead astronauts would be strewn over the oceans, space, and the moon, if the core part of NASA did business like GISS?

Gerald Browning

All,

I am writing a comprehensive review of this manuscript that is quite different from Judith’s review. But might I suggest that you go to the site mentioned at the beginning and look in the downloadable pdf file
in the first reference at the following items.

Figure 2.2 on page 9 shows contour plots of the perturbation of the steady state zonal flow (not shown in published manuscript). Note that the perturbation is quite large scale, but also quite small in magnitude (1 m/s as compared to ~40 m/s in the jet maximum of the steady state zonal flow solution). This perturbation, as small as it is, suffices to destroy the numerical accuracy of the dynamical cores within a matter of days. Also note that observational errors in the winds are at least this order of magnitude and that the inaccurate parameterizations in a climate model can easily lead to errors in the winds of this magnitude.

Even more of interest is the discussion of the sensitivity of the dynamical cores to different hyperviscosity coefficients beginning on page 38
(and there are other forms of spectral and damping included in the EUL model). (This is a point I have emphasized over and over.) This section was not included in the published manuscript for obvious reasons.

Jerry

MarkW

how many dead astronauts would be strewn over the oceans, space, and the moon, if the core part of NASA did business like GISS?

I recall a recent Mars probe that exploded because one group was using English measurements and another group was using metric.

lucia

Judy:
Yes. Some people are arguing about whether or not the GCM’s are perfect. Other people really are discussing the V&V. These are entirely separate issues.

GCM’s can’t be perfect. The people asking for that will never be convinced by a GCM. But that’s probably only a very small fraction.

GCM’s are complex and so more difficult to validate than models describing simpler things.

However, neither complexity nor the impossibility of perfection interferes with the “do-ability” of a V&V! The goal of V&V is absolutely not to create a perfect model, and complexity is simply not a problem.

If you can write a code, you can do a V&V. Complexity doesn’t actually affect form Verification very much; it only means one requires longer document because there are more modules to test. If a model is approximate and has difficulties (like GCM’s), or the results are difficult to interpret, that is reflected in the narratives and figures contained in the Validation document. All the caveats you describe here would be included in the validation document in written from where third parties could read the caveats. That’s a purpose of the validation.

But with regard to this:

In the case of the ECMWF model, the V&V is very clear (read the technical notes, memos).

No. It’s nto at all clear in those notes and memos.

I suspect the V&V for ECMWF may well be very clearly stated somewhere. In so far as a model is used for weather prediction broadcast to the public, and it’s predictions have a direct impact on public safety, funding and regulatory agencies probably do require V&V for any model.

That said:, there are zillions of links at the site you point to. I’ve clicked on 20 or so links to reports and memos and skimmed. They look like good reports. They look like decent science. Unfortunately, with regard to the discussion going on in this thread, absolutely nothing I clicked remotely resembles a Verification. A small fraction of documents contain snippets that would belong in validations, but those would be stupendously incomplete as validations.

Could you point to an individual link that you think looks like a verification?

If you could point to a specific one, this get us all on the same page. Otherwise, right now , it looks like everyone is talking past each other.

Meanwhile– Dan, or Tom Vonk, could you supply Judy with examples of formal verification documents? (I know this is difficult since most are only available in the grey literature and they are generally a set of documents. At that, they represent such a small fraction of the grey literature that you need to know a named party did one. But if either of you was involved in one, then that might help Judy understand.)

(FWIW, in some ways, I’m agnostic on this isue. I frankly don’t give a hoot whether GCM’s are verified or validated because I don’t base any of my judgment about AGW on the results of GCM’s. I rely on simpler energy balance models supported by temperature trends, and some physical arguments. I think the balance of the evidence points toward warming caused by human activity.

Nevertheless, if documents describing formal validation and verification of GCM’s do exist (and I’m betting Quatloos they don’t,) it would be useful to the never ending discussion if someone could identify those specific documents.

Identifying alternative documents of the sort that appear in academic journals and incorrectly calling them V&V just won’t do for people who now what V&V is and want to see V&V. (It’s a bit like giving someone chocolate ice cream when they want chocolate fudge and saying “See, I gave you chocolate! And anyway, ice cream is better than fudge– you should want fudge!” Ice cream is tasty, but the customer wants fudge.)

Only bringing the V&V to table can bring this bitter arguments about whether or not V&V has been done.

Jaye

94

That’s mostly because they are either glorified hackers or just one of the rabble of grad students punching out code for their profs. Its been my experience that these grad students have a decent grasp of the science but are woefully lacking on the coding side. Now, when you iterate this process over hundreds of grad students for 10’s of years what sort of pasta does the process produce?

Gerald Browning

All,

The best way to obtain the manuscript I mentioned above is to do a google
search on NCAR Tech Notes and then on Jablonowski. The tech note is more
complete in its presentation and has the amusing details mentioned.

Now the interesting question is why were these rather revealing discussions
removed from the final published manuscript.

Jerry

Judith Curry

Lucia, the sheer complexity of the V&V for these models is pointed out in #88. V&V for any complex model is a complex process, as pointed out in the paper that I linked to in # 83 http://www.informs-cs.org/wsc98papers/016.PDF

This paper says there are 3 different categories of V&V:
1) that done by the modelling team
2) independent verification done by an independent team (very costly and time consuming for what is actually obtained)
3) scoring model. while the article says this is infrequently used in practice, no so for atmospheric models used in weather mode (e.g. ECMWF). I will get to specific references for scoring later in the message

The recommendations for validation made by Sargent (the author) are:
1) assessing the conceptual validity of the model
2) computer model verification
3) necessary data is available for building the model

Recommendations for verification are:
1) comparison to other models
2) degenerate tests
3) event validity
4) extreme event validity
5) extreme condition tests
6) “face” validity tests
7) fixed value tests
8) historical data validation
9) internal validity (stochastic runs)
10) multistage validation
11) parameter variability-sensitivity analysis
12) predictive validation
13) traces
14) turing tests (i didn’t know what this is so googled ECWMF turing test, and i got 150 hits)

All of these kinds of tests have been done for the ECMWF model. But a reminder, you are doing V&V on the dynamic core, the bottom boundary conditions (like orography), each individual parameterization (e.g. radiative transfer, convection, boundary layer, clouds, etc), and in the case of coupled models the ocean module, the sea ice module, the land process module, the aerosol module (and in future the ice sheet module), in stand alone mode as well as when coupled in the climate model.

The key issue of interest here (at least on the threads re verification of NASA GISS climate model simulations) is predictive verification. The ECMWF atmospheric module is evaluated every day in numerical weather prediction mode, and also each month for the seasonal forecasts. Evaluating on climate change time scales can’t effectively be done because we only have crude climate model simulations from the 1980’s and the more sophisticated coupled models really came in the mid 1990’s. So we are talking about a 10 year period for “verification”, and these projections make no pretense at simulating things like volcanic eruptions. So the IPCC 4th assessment report simulations can be verified say in 2030, and by then we will have much better models. so in many ways climate modelling has unique verification challenges since the model development time is much faster than the time scale of what is actually being modelled. Also, the extreme event validity is very difficult one for climate models because even in the paleo record we don’t have an exact analogue for what we might be looking at in 100 years. however, simulation of a broad range of paleoclimates (and verification against whatever paleo data is available, given all the uncertainties, can help here.

To Jae or whoever said something about climate scientists not asking for money to do this, there are incredible efforts undertaking by administrators from NCAR, NOAA, NASA and DOE to try to get more funding for this, not to mention from our professional societies, and many individual scientists. The funding for all this is peanuts (when you see big numbers being spend on climate research, most of those funds go for satellite systems, then some for computer hardware, and other observing systems, not much for actual scientific research and modelling). I probably get 10 requests a year either from the AMS or NCAR/UCAR to engage in a letter writing campaign to congress members. We conduct lots of congressional briefings that highlight the need for more resources. It seems to be more important to fight wars in Iraq, send people to Mars, etc. Don’t blame the climate scientists for this one.

I

Gerald Browning

Judith Curry,

Have you ever coded an NWP, climate model, or any numerical model?

Jerry

Jaye

Also, 98 is ancient in terms of software process think.

Gerald Browning

If there are basic mathematical problems with the unforced continuum PDE system, convergence tests of the numerical approximations of the unforced dynamical system will reveal those problems. Thus we need to stick to the issue of dynamical cores on this thread and not get into the more complicated questions of errors in the parameterizations (forcing).

If the dynamical cores do not provide accurate numerical solutions for
simple tests, then adding forcing is not going to help and in fact will
just be a tuning exercise and not science.

The points mentioned in the technical note on the NCAR site should be
read and discussed. These point to very serious errors in the dynamical
cores that cannot be overcome.

I am working on a thorough review of the Jablonowski manuscript that will discuss these and other points.

Jerry

William Newman

Tom Vonk (76): If you quote “radiative transfer is a settled science” there’s some someone who actually said that in this context, right? Could you give the reference? If you don’t see the importance of having the precise reference, look at http://www.realclimate.org/index.php/archives/2004/12/myths-vs-fact-regarding-the-hockey-stick/ . Would you agree that mike@RC would seem to be some weighted average of clueless and sinister if he didn’t have an actual op-ed reference that literally contains his Myth #0, that evidence for modern human influence on climate rests entirely upon the “Hockey Stick” Reconstruction of Northern Hemisphere mean temperatures indicating anomalous late 20th century warmth?

Sadly, given the level of the debate, mike@RC might actually have read such an op-ed, and you might actually have seen someone making your quoted claim. But even then, if you’re going to use the quote, a citation would be a lot more constructive. Or at least non-destructive: responsible venting rather than what looks almost like an irresponsible slur. You’re putting a sufficiently bogus claim in the mouth of some opponent that even in this debased debate it’s not obvious to me that such a dumb opponent exists. (Exists in any meaningful way, anyway: anonymous trolling is hard to interpret sometimes.)

(speaking of venting: At that realclimate page, the comments make it look as though no one wants to ask mike@RC for a reference to some op-ed that actually claimed that particular odd-sounding and easy-to-refute thing. It seems hard to read that page without being awestruck by the implied editorial practices.)

Judith Curry

Some people seem to think my statements re climate model complexity is a copout, so I will try to address this. The best documented climate model is the NCAR CCSM, http://www.ccsm.ucar.edu. You can download the code and read the documentation at http://www.ccsm.ucar.edu/models/ccsm3.0/. My reasons for thinking that climate models are among the most complex, if not THE most complex models out there are:

1) In meetings between Georgia Tech and IBM for us to possibly purchase a Blue Gene (hugely big fast computer), discussions around the kinds of codes that need this kind of computing power are climate models, bioinformatics (dna type simulations), and complex fluid flow simulations. while bioinformatics and fluid flow calculations require lots of computer power, the codes aren’t nearly as complex as climate models.

2) the number of degrees of freedom in climate models is almost certainly more than any other model (after all, were doing the whole earth here).

3) the nonlinearities in the system and coupling of different systems

4) the sheer scales that are involved, and the need for subgrid parameterizations of many different processes

I would appreciate any comments from big modelers (e.g. Lucia) regarding how the NCAR climate model stacks up in terms of overall complexity (not just number of lines of code or length or run time or amount of storage).

William Newman

Judith Curry: Thank you for general helpfulness and courtesy in this trying environment, and for general on-point-ness, and specifically for the references to the Technical Reports and Technical Memos.

I’m sure I’ll learn something just from undirected stumbling around among them. But, in case you (or anyone else knowledgeable, that works too) feel like giving extra hints, I’m particularly curious about two things…

First, I’m interested in the general outline of the kinds of parameters and procedures used to tune the models. In particular, trying to understand how important the historical hockey stick fit is as constraint determining low-uncertainty high-feedback CO2 temperature response behavior. I grant in advance that it might be impossible to give a usefully honest summary in a few pages, like the classification of finite groups or something. Or, if it exists somewhere in some prominent place (TAR?) and I just haven’t read it yet, my apologies in advance for not doing a better job of getting up to speed. (This is in a sense my less-strawman version of Myth #0: I find myself wondering what all the fuss is about…)

Second, I’m interested in things quantifying the detailed predictive power the models have demonstrated. As you might guess from my earlier remarks, I’d probably be especially delighted with something with a learning-as-compression aesthetic, letting me count how many kilobytes of confusion go away when we look at the world this way, but traditional statistical significance conventions are OK too.

William Newman

Lucia: you wrote “FWIW, in some ways, I’m agnostic on this isue. I frankly don’t give a hoot whether GCM’s are verified or validated because I don’t base any of my judgment about AGW on the results of GCM’s.”

I myself do give a hoot. Lemme try to justify why.

There’s a very substantial difference in expected impact and rational policy response if CO2 forcing is only as large as you’d get from first principles “simpler energy balance models.” And unless something like the hockey stick is known with high confidence, it’s hard to know what fraction of the temperature trend assign to CO2, and then “supported by temperature trends” becomes a very iffy argument.

How very substantial is the difference in economic impact? The models seem to be saying that temperature response is amplified twofold or threefold from a trivial first-order result, right? Any detailed estimate of economic impact from climate change is a ridiculously vexed question, of course. (Imagine estimating an economic impact over the 20th century given the knowledge of someone in 1900. Good luck.) But on very general principles I’d expect that to the extent that change is a problem in and of itself, we should think of it as at least a second-order effect, with little important first-order effect. So the twofold or threefold feedback turns into half or a whole order of magnitude economic impact.

(Conversely if net economic impact had a substantial first order term, change would tend not to be a problem in and of itself, one direction of change would be net bad and the other would be net good, and we’d be highly motivated to artificially drag the earth’s temperature at least some amount in whatever the net good direction was.)

Gerald Browning

Judith Curry (#120),

And what about Dave Williamson’s tests that show that the CAM3 deviates from reality in a few days?

Jerry

Jaye

All of the models are verified against observational data,

Argh.

Gerald Browning

Jaye,

Given the inaccuracy of the large scale observational data and the sensitivity of jets to small magnitude (1 m/s) large scale perturbations (Jablonowski manuscript), what does this say about NWP model forecasts?

Jerry

Gerald Browning

Jaye,

She got off the main topic of this thread in order to avoid the real
mathematical issues with the dynamicaal cores that cannot be solved. Actually I doubt if she knows that much about numerical models. It is just a smoke screen to avoid her lack of knowledge in this area and the fact that she wrote a “review” about a manuscript that she does not fully understand.

Jerry

lucia

Dan Hughes– Can you provide us examples of full V&V software, so we can see these for other codes? I think it’s only fair to show what these are as you describe in 112?

Judy– Thanks for the specific link. It really saves me and the whole peanut gallery time.

What the V&V propoentns are howling to see is examples an of : “2) computer model verification”. done formally. What is required is dicussed in section 6 of http://www.informs-cs.org/wsc98papers/016.PDF What the also want to see is the formal, published documents that show this was done. That’s what I’m not seeing anywhere in the list of links you provided. Dan is also describing this is bullet 2 of comment 112. (FWIW, there is much, much more validation out there than verification. Sargent’s paper also discusses validation more. All scientist will say validation is more important. Scientifically it is. But, that may not be so politically.)

Obviously, Sargents document saying this should be done is not an document that contains a formal verification.( It is unlikely such a document would be presented at a conference. They are boring and exceed page restrictions. ) But at least this shows that verification is discussed in climate science.

Yes, formal verification and validation is costly. Horribly costly. Is it worth the money? Depends.

The only time I did formal verification is for *nuclear safety work* and even then only when the relationship between safety and my specific results was very direct. As in: the person who hired me to do an analysis did so specifically to decide whether or not to have a guy suit up in a rad suit, enter the hot area and do something dangerous. And dangerous means: experiments on radioactive equipment that might contain a combustible mix of hydrogen and oxygen and explode as a result of the experiment. All the formal *verification* paper work ended up in files in the archives at the Richland Field Office. If I wanted to find these for you, I’d need to remember the title of the program and then file a Freedom of Information request using very specific language.

The verification steps made an otherwise simple analysis costly. Was it worth the money? In the nuclear case, to prevent an horrible accident, it was costly. (Besides, it was a legal requirement!)

Is the cost of documenting a formal vefication worth it for GCM’s?

The reason I say some sort of document would be advisable is neither technical nor scientific. FWIW, I believe the GCM modelers do verification. They just don’t write a big costly document, or maintain archives in NASA’s equivalient of the Richland Field office Archives.

The reason documentaiton is adisable is entirely political. Documentation would permit with third parties (particulary skeptics) need to be convinced. For scientists or researchers assessing a theory, validation is what counts. For doubters both count, and verification is important.

Much of the case for AGW is presented to the public based on these models. A certain fraction of the public will feel uneasy until such time as someone can say: “Here. This document shows the verification of the GISS Model E”. Then, we need each major groups writing GCM’s to write these documents for their codes. And for the skeptics, it really, honestly has to be a formal published verification as discussed in on page 125, section 6 of http://www.informs-cs.org/wsc98papers/016.PDF . Validation won’t do.

If you read Sargent’s seciton 8, you’ll see he says just what I’m saying:

Documentation of model verification and validation is critical in convincing users of the “correctness” of a model and it’s results and should be incluced in the simulation model documentation.

It’s the lack of formal documentation that is causing a political problem.

Judith Curry

Jerry and Jae, a few points of clarification. I did not bring up V&V, Lucia did. It seems to be a topic that people are interested in. This paper was originally mentioned on the hurricane thread, i tried to make my review relevant to issues that people have been discussing on CA like climate modelling and hurricanes. My Ph.D. was in geophysical fluid dynamics at the University of Chicago (about 25 years ago). I am not currently active in fluid dynamics modelling, my most recent exercise was LES modelling of the stable boundary layer (about 7 years ago now). I do not run climate models myself, but am actively contributing to the development of parameterizations for clouds, sea ice, boundary layer, radiative transfer, and ocean surface fluxes. Am I the last word in fluid dynamics or climate modelling? not at all. But I am making an effort to communicate to this group to address some fundamental questions that people have about climate models and to what extent we can have confidence in them.

Jerry, once you write your review, I will critique the substance of it. I see several flaws already in the reasoning of your previous post advertising your forthcoming review, so this could be interesting.

Gerald Browning

Judith,

I await your comments on the NCAR Tech Note Section.

Are you claiming that one need not accurately approximate the basic dynamical system in order to obtain physically correct solutions?
Please cite a mathematical reference that contains a proof of this
assertion.

Nonlinear cascade is serious business.

Jerry

Dan Hughes

Thank you Judith for you productive additions to these discussions.

Judith Curry

Gerald, your comments are lacking in clarity and even apparent relevance to the paper. The term “nonlinear cascade” is not even mentioned in the paper, the only possible related issue seems to be the different ways in which the models cut off the high frequency waves. So unless you actually do some work, including writing something that is readable and clear, i am certainly not going to waste time responding to this. Frankly it seems to me that you have misinterpreted this paper and are reading things into it that are not there.

George Crews

RE#129: Lucia, I can give you an example of a code that has had “full V&V” for its use on the Yucca Mountain Project (high-level radioactive waste repository).

FEHM is a finite-element heat- and mass-transfer code that simulates non-isothermal, multiphase, multi-component flow and solute transport in porous media.

To look at the V&V documents for this code, go to the public Licensing Support Network site (http://www.lsnnet.gov/) and search on “FEHM and Verification and Validation”.

I got 66,485 document hits. I guess they take V&V seriously at Yucca Mountain.

Dan Hughes

re: #129: Lucia, the codes for which I was a part of the V&V and SQA are not in the public domain. Even the code manual documents are not available to the general public. And as I mentioned before, the V&V and SQA documents were never in the public domain. The owners of these codes paid for the development and especially paid for the V&Vd versions. If you want the codes you got to help the owners recover some of these costs.

Sam Urbinto

.

Population is a terrific source of heat and amateur radio signals are a powerful greenhouse electromagnetic field.

lucia

Dan Hughes– Thanks for the answer. I totally understand (and had missed your previous response). These things are generally not publicly available– and that of course is a question one might have vis. GCMs. Do they not exist? Or are they just difficult to find.

Luckily, George Crews knows where we can find a trove of examples! I went to the Yucca mountain link he provided, Search the LSN Document Collection, entered “verifiation” and found several documents, that include the utterly boring (as required of a verification) SCALE-YMP Software Verification Report for OSF1_V5, ORNL OCRW-SQA-010, Rev. 1 and SOFTWARE INDEPENDENT VERIFICATION AND VALIDATION REPORT FAR VERSION 1.1 (C)

These seem like a near top level verification document, along with cd attachments, and references to other sub verifications. I think comparing these examples to the much more intersting (to me) documents Judy linked will clarify the huge difference between validation (which can look more like what Judy linked) and verification.

@William Newman– I totally understand why you give a hoot about this. It’s an entirely rational thing to worry about. All I mean is this doesn’t happen to be the nail I tend to pound when worrying about whether something is proven to my satisfaction.

What I’m trying to communicate to both skeptics and advocates is that this step is standard, it is considered important by third parties trying to make up their minds. So, if these documents exist for GCMs, its in the interest of those who advocate for political action to dredged them up or told skeptics where to find them.

Of course, the verification documents may not exist because these GCM research programs weren’t perceived as existing to drive policy by the funding agencies. (Yet, as a political reality, they do act that way.)

Judith Curry

William, This is my take on it, note i am not a paleo-oceanographer or paleoclimate modeler. the reason the period of the last 1000 years isn’t much of a priority in terms of paleo simulations is that you need some specified change to external forcing (solar, atmospheric composition) or bottom boundary conditions (like continents moving around) to get a simulation that is different from present. We can simulate the “blade” of the stick during the last 100 years (CO2 increase), but there is no particular change to external forcing in the earlier period that we know about. The medieval warm period and little ice age (whther they are local or global phenomena) are believed to be associated with thermohaline circulations in the atlantic; correct simulation of such long term internal oscillations in an ocean basin requires long term simulations for the ocean for which we don’t really have any observational constraints. So this is why there is probably not much to be learned from such a simulation. Coupled climate model simulations for a period of about 10,000 years would be probably needed to capture these kinds of circulations ( I am just speculating here). Interesting for sure, vut the scientific payoff for using all this computer time doesn’t seem to be too high in the absence of observations of the ocean circulation during this period with which to compare the simulations.

Jaye

William,

How do you know I’m not rich?

You assume too much. There is quite a bit of difference between, process for process sake, and designing code with care and professionalism. If you were truly in the business you would know the difference…thereby sparing us from several hundred words worth of inane prattling.

To quote Hans Solo: “I must’ve hit it pretty close to the mark to get her all riled up like that, huh kid?” So I’m guess you are close to the spaghetti making processes at NASA or else where.

John Creighton

Are there any good web resources to help guide novices through the theory in global climate model. I’d expect something along the lines of a summary table giving the models, in one column, the continuous equations in another column, the discrete equations in another column and the coordinate system used.

I’d also like like links to the derivations for both the continuations equations and how the discrete equations are derived from the continonus equations. Maybe the models are too complex for such a simple summary table but it would be nice if possible.

I know it sounds rather rudimentary but not everyone that is interested in GCM have studied computational fluid mechanics.

Gerald Browning

Tony (#136)

Agreed.

Gerald Browning

Dan Hughes (#134),

I think that the wait will be worth it. 🙂

Jerry

Geoff Sherrington

An analogy.
Before they became popular, I investigated the concept of the “call centre” and recommended firmly against involvement.

There were two broad groups of people, those who designed and made the product and those who answered questions about it.

Those who answered questions were trained in people communication skills. They had a brief written manual about FAQs and a default of calling a supervisor who worked from the same manual.

Those who designed and built the product (one example was a PC) marched to a different drum. They worked with data and science and math and stats and knew things like the properties of semiconductors, heat flows, electronic circuitry etc etc. But they were never let near the telephone to answer the public’s questions.

Climate science is a bit like this. The public face knows very little about the science and the scientists know very little about how their work will be represented.

One can introduce quality control measures of various types. For example, a considerable effort went into solving the magnitude of pi to a million places. It can be stipulated that this value of pi be used as a standard and that any model which cannot beack-engineer to it fails.

There is intellectual satisfaction in the pi exercise and there is in coming up with a GCM that accounts precisely for every problem that can be thrown at it. But, for what purpose? Make the grid cells as small as computers will handle and you still have large input errors in representing surface albedo, temperature at screen height, sea surface roughness and a myriad of other variables. So it will not be possible to gain the parallel intellectual satisfaction of solving for pi to a million places. Why try?

Whatever answers come out of GCMs, no matter the standard of computing and quality control and lagrangian mathematics and progress in solutions to Navier-Stokes, we are left with fundamental fact that the wrong question was asked.

Instead of postulating “Can we model global climate” we first should have thought “How many angels can dance on the head of a pin?”

MarkW

William, #138:

Are you honestly dismissing V&V as just another means for keeping down the masses?

Tom Vonk

J.Curry wrote # 83

I suggest that the V&V discussion be framed in the context of the validation techniques listed in this document. In the case of the ECMWF model, the V&V is very clear (read the technical notes, memos).

J.Curry wrote # 105

Recommendations for verification are:
1) comparison to other models
2) degenerate tests
3) event validity
4) extreme event validity
5) extreme condition tests
6) “face” validity tests
7) fixed value tests
8.) historical data validation
9) internal validity (stochastic runs)
10) multistage validation
11) parameter variability-sensitivity analysis
12) predictive validation
13) traces
14) turing tests (i didn’t know what this is so googled ECWMF turing test, and i got 150 hits)

That is exactly what I did and I provided you in this thread with a free review of a very recent ECMWF Technical Memorandum 539 Recent Advances in Radiation Transfer Parameters .
It is true that I would need probably 1 week to audit it really but that would not be so much considering that 19 people were necessary to write this memo .
Incidentally I notice that there is no scarcity of ressources when 19 people sign a report of this quality .

I suggest that 2 of them be specifically dedicated to insure that the recommendations quoted in thread # 105 are followed by effects .
Even after only a 2-3 hours study it appears that items 3) , 4) , 8.) and 12) are severely mishandled in this ECMWF Technical Memorandum .
I suggest that the Memorandum be rejected in its present form and a serious predictive and historical validation takes place with explicit disclosure of experimental data used , its validity and representativity and its (re)treatement takes place .
Farther I suggest that while model to model comparisons may have a limited use , they can’t replace the quantitative studies concerning prediction skills . The use of such comparisons should be strictly restricted to statistics related to calculated parameters and only under the condition that the relevance of such statistics has been proven beyond any
reasonnable doubt .

Lastly I suggest that teams refrain from anecdotical evidence (like f.ex the Sahara sand plume) whose relevance to the topic is not quantified .

I have done this (small) work in my free time for free , the subject is not of extreme complexity , so I trust that there is enough time available in a team of 19 people active full time to comply with the above recommendations .

Jerry # 111

If there are basic mathematical problems with the unforced continuum PDE system, convergence tests of the numerical approximations of the unforced dynamical system will reveal those problems. Thus we need to stick to the issue of dynamical cores on this thread and not get into the more complicated questions of errors in the parameterizations (forcing).

If the dynamical cores do not provide accurate numerical solutions for
simple tests, then adding forcing is not going to help and in fact will
just be a tuning exercise and not science.

I am fully with you in that and I indeed agree that everybody would save much time and money by focusing only on the relevance , sensitivity and convergence of the dynamical core .
These issues have been largely discussed here in the “Exponential divergence thread” , Dan Hughes has an extremely interesting blog on those issues too and it appears clearly that this very basic issue is unresolved .
We may have minor divergences about the Chaos theory and its relevance to the problem but substantially we share the same concern 🙂
However I am not sure that a board discussion is a very efficient way to treat that kind of problems .

Jimmy

If so many people here know what’s wrong with the climate models, why aren’t they out there fixing them???

Wouldn’t THAT be the scientific thing to do? Jerry- why didn’t you do this while you were at NOAA?

Much of the discussions around here seem like a whole lot of finger pointing and name calling where not many people are actually stepping up to the plate. (other than SteveMc or Judy Curry…)

jae

163:

If so many people here know what’s wrong with the climate models, why aren’t they out there fixing them???

This is an audit site, not a “fix it” site. If you could provide a few $million, some of these folks might be enticed to really fix them (if possible). And I think this site is pointing out flaws that will aid in “fixing” them.

MarkW

Jimmy,

How can we fix the code, when the modelers don’t release the code?

Regardless, do you always take the position that unless someone is willing to take the time and effort to fix something, they aren’t allowed to criticize it?

Can I safely assume that you never criticize politicians, since you haven’t run for office yourself?

William Newman

MarkW wrote “Are you honestly dismissing V&V as just another means for keeping down the masses?”

I’m saying that pushing any detail of heavyweight policy as obviously *the* way to do modelling for decision making tends to sound like dogma to me. If you believe as revealed truth that V&V (e.g. vs. sleazy V or some newfangled V&V&V) is *The* Right Way to get decision-making data out of models, I’m usually skeptical.

My principled reason for skepticism is that it doesn’t seem to be utterly dominant in fields where models of their own free choice drive people to make big decisions with their own money. (for modelling: finance, and perhaps drug discovery and mineral wildcatting too. for software in general: any number of products)

My bad attitude, what you paraphrase as “way to keep down the masses,” is in part from experiences like working as a contractor at Nortel ca. 1998 (and using telecom standards, and doing some standards work at the IETF),. The old-line telecom folk (whose attitudes concerning the right way to get quality sometimes sounded like Jaye’s) ran into the IETF, Unix, and so forth; interesting times. And my “eaten alive” image could be taken to apply to that in hindsight.

My literally keep-down-the-masses analogy was partly just driven by the whimsical tie to Jaye’s image of decapitation. But there was a cliche among old cynics at Nortel that “our customers’ core competence is lobbying.” So it’s not entirely whimsy: I’m predisposed to notice when people who are self-righteous about their superior procedures happen to be protected by law from any competition by pushy upstarts with ostensibly inferior procedures.

I’m basically fine with formality and process which is common to IETF working groups, the OpenSSH project, Linux, ocaml, and any number of invisible commercial projects. Once tricky projects involve dozens of people, the successful ones tend to have some things in common. But the consensus set of good practices truly common across “engineers that have to actually make something work” does not have all the details that Jaye is calling for. And good sense and integrity and ability are even more important than procedure. And when I see someone fixating on some bureaucratic detail which is not in the intersection of sets of policies of the projects I mentioned, I take it as evidence of lack of good sense, lack of integrity, lack of ability, or all three.

I’m not dogmatically opposed to heavier process than that. I’m dimly aware that historically a surprisingly large amount of bureaucracy was needed to get interchangeable parts working, for example. But I want to see processes and bureaucracies justify themselves by eating the competition alive (not so many non-interchangeable parts used today…), not by appealing to voters’ and legislators’ intuition that ordnung muss sein.

Jaye: I am an independent software developer. I don’t have any active organizational affiliation unless you count a free software project that I started years ago and still nominally control. Not even an employer: I’m taking a flyer by working for myself on a search-related problem, which is likely too hard for me, but with a potential payoff which seems to make the expectation value worth it.

“If you were truly in the business you would know the difference.” Yeah? I’m not in the businesses I mentioned in my post. But one of the sizable bodies of university-based code that I’ve worked on is driving many dollars worth of decisions at a successful airline-ticket search/optimization website today, and I think that qualifies me for an opinion about your original rhetoric about “engineers that have to actually make something work.”

If I write off my current project and go back to contracting, I might end up in something related to finance, but I’ve never worked in anything related to finance. I chose that because there is a well-known niche there where qualitatively better modelling would tend to be all you need to make a ton of money, and I can’t think of a comparably clean example in fields closer to my work experience.

To the extent that you “hit close to the mark” (my words might be “struck a nerve”) it’s partly because of general common sense (the years have exhausted most of my patience with quality claims from process bigots, language bigots, and funding-source bigots who have the One True Way but weirdly never get around to dominating quality-sensitive fields like finance or compilers), and partly because of experiences like contracting at Nortel, above.

MarkW

William,

You believe that code shouldn’t be tested to prove that it does what’s advertised? That’s all V&V is.

Dan Hughes

re: # 115

Judith,

General
A desire for bigger, faster hardware is not necessarily a valid measure of inherent complexity. Bigger, faster hardware can be driven by the need to increase spatial and temporal resolution, for example. And as Judith notes, the scales of these for the GCMs are enormous. Higher fidelity relative to resolution of multi-scale phenomena and the interactions of these will also generally demand more compute power. Poor coding practices can also lead to the need for more power in attempts to simply power through all the operations required (give me a bigger hammer). Legacy software will generally have tons o’ poor coding and as the desire to get more info out of an application naturally evolves brute force is one way to overcome these. On a personal level, I simply like to have, and drool over, bigger hammers in my computing environment.

Complexity
I would say that a more likely measure of complexity of the software is the number of interactions/interconnections that occur between the data structures. Typical examples of this are physical situations that involve several important coupled physical phenomena and processes. Fluid flows, conduction, power production processes, the response of structures to temperature gradients and fluid forces, different kinds of engineered hardware of various degrees of geometric complexity, and others. The parameterizations, I think engineers generally call these the heat and mass transfer coefficients and friction factor correlations, might need to cover wide ranges of possible fluid and structure-interface conditions. The equations of state for the fluids in the system might also need to cover wide ranges of thermodynamic states.

Each one of these separately would not necessarily require large LOC counts (some might) and the coding should be easy to follow, if good coding practices are set and followed. But if two of these are coupled, fluid flow and conduction, for example, the data structures for each must know the data structure of the other (or carefully selected parts of each). And if three or more are coupled, fluid flow, conduction, and structure response to these, the number of interfaces of course increases and each data structure must know about at least some parts of the other data structures. Additionally, the fluid flow itself might be inherently complex. Transient compressible flows of mixtures of fluids, for example, are notoriously complex in the physical domain. Couple this flow to conduction and you’re got yourself a real hard problem. A control system, a flight control system in an airplane for example, needs to know about almost all the other systems in addition to the state of the plane relative to the fluid through which it is moving. Control systems in engineered process plants must interface with all the pieces parts as well as the working fluid. A model of such a system would have a data structure that interfaces with almost all other data structures. And so it goes.

What I’m trying to say, and maybe not too successfully, it is the coupling of the data structures as dictated by the physical situation that can lead to complex codes. The coupling in the physical domain leads naturally to coupling in the code domain. Absolute conservation of mass, momentum, and energy requires that the physical coupling at the interfaces be evaluated consistently and at the same time levels in each of the coupled systems.

For a comprehensive GCM I can count oceans, land, atmosphere, ice, biological processes, organic and inorganic chemical processes, human-made sources and other effects, radiative energy transport, conduction and convective heat transfer, phase change, clouds and aerosols, as some of the important system components, phenomena, and processes. The fluid flows are 3-dimensional and turbulent. There are probably more. This list is getting quite long, but I don’t know that any single GCM has yet included all of these.

Legacy software, which generally evolves from simple one-objective model/method/code/application is especially deficient in handling data structures and especially interfaces between data structures. Parts get tacked onto the previously existing pieces and the thing grows up to be a really big and ugly thing. Tacking on leads to really bad code primarily because the original code did not plan for coupling to other data structures. I have seen some really scary code.

Problem Size
As for the degrees of freedom, I don’t know what today’s CFD codes are running at, but I’ll bet it’s a huge number. Simulation of transient, compressible single-phase turbulent flows around the complete airframe of an Airbus 380 has got to need lots o’ nodes. Almost any problem can be made into a Grand Challenge grade problem as the need for higher temporal and spatial resolution is factored in along the software-evolution trajectory.

Nonlinearity doesn’t count. All interesting and actual real-world problems are nonlinear. The universe is nonlinear.

But none of this negates any of the requirements for V&V and SQA relative to applications that involve decisions about public policy. Equally important, as the models and codes evolve to greater degrees of complexity, the higher the payoff relative to maintaining the integrity of the entire software development cycle and the ultimate usability and maintainability of the products.

The only data point that I have is the NASA/GISS ModelE/ModelE1. These are built with about 100,000 LOC. That’s not many.

Steve Milesworthy

#173 Dan
I agree with everything you say about complexity. I am in the process of helping to build a model with all the components you describe. Common data structures can be a nightmare when bringing together components developed by different institutions (eg. the CICE model mentioned by Judith, the NEMO ocean developed in France, the chemistry and land surface schemes developed by the academic community etc.).

Looking at the paper lucia mentioned, most aspects of V&V are done here though it all has different names – eg “Event validity” is done by comparing model’s internal variability with observed variability.

One area of weakness is some of the verification tests – it is difficult to do unit tests when interfaces are complex and diverse, so we require that an updated component must include backwards compatibility such that new versions of the full model can produce bit-identical results to the old. Components also produce increment diagnostics (the per-timestep change it makes to the main model fields) that can be examined. So here, validation and verification are a little blurred.

On the other hand we’ve lived with our current model for over a decade with hundreds of developers and users poring over every result. The system we have “works” (the model is also used for short-range forecasts) even if we are looking constantly to improve it. In 98% of testing and use, the GHGs are set constant. My inability to produce V&V documentation in the standard demanded here does not reflect a lack of V&V.

I’ve had a brief look at modelE, and I have to say “my” model is significantly more readable, so I hope people don’t think it is a typical example (sorry RC folk!).

Dan Hughes

re: NCAR CCSM, ECMWF, and NASA/GISS ModelE/ModelE1

Judith has several references to codes and documentation other than NASA/GISS ModelE/ModelE1.

First I have tried to break the ModelE habit more than once. But having time invested it’s kind of hard to do. Based on the discussions here I think I can make the break now and focus on other codes.

As a rough example of what an Independent Verification exercise might look like consider the following. For what ever the reasons, Judith’s organization has given me a Grant, for sinful amounts of money, to check the SQA status of parts of the ECMWF model and code. We have decided that the momentum balance for the vertical direction is a critical aspect of the model and its applications. Possibly because we know that there is more than one approximation used for this equation, and some of these have bad properties.

Here’s an outline of some of what I need to see in order to satisfy the SQA Plan for the code.

1. The derivation down to the exact final form of the continuous equation for the vertical direction momentum balance as used in the model.

2. The exact and final form of the discrete approximation of the continuous equation. All time and spatial locations, and sizes of the discrete increments, also documented.

3. The numerical solution method used for the discrete equation. For this case the method is very likely a part of a method that solves all the momentum balance equations.

4. Demonstration that the numerical solution method used to solve the discrete approximations are consistent, stable, and convergence. Including effects of discrete approximations for the BCs.

5. Demonstration that the actual working order of the solution method(s) is in agreement with the theoretical order. Including the effects of any stopping criteria, and grid refinement effects. Ideally these demonstrations will include all the parameterizations that a part of the vertical momentum equation model.

6. Documentation of how the discrete approximations and solution method(s) fit into the actual code.

7. The routine(s) and data structure(s) that are used in the code to obtain the solution of the equation. These will be used to check that all the stuff in 1 thru 6 has actually been implemented as specified by the documentation.

For each of the above I will have objective technical evaluation criteria and associated success metrics to show that the requirements have been met, or not.

I’ve constructed this from memory as I type, so it very likely is not complete.

Here’s a test. Can anyone point me to document(s) that contain the information that is needed to complete this checklist for the ECMWF model/code.

Thank you again Judith for your constructive additions to these discussions. I’ve got some coupled radiation-convection-conduction finite-difference equations to look at.

steven mosher

Code complexity? This is a common excuse used by lazy people to argue against a variety of
things.. 1. the junk compiles so the compiler understood
it. 2. the junk executes so the NAND gate understood it.

Bottomline: It compiles. Bottom line: sheffers stroke.

If you want to measure “code complexity” there are many measures.

start here:

http://www.sei.cmu.edu/str/descriptions/halstead.html#1227444

And theoretically, this is what we mean when we talk about complexity.

http://en.wikipedia.org/wiki/Kolmogorov_complexity

Larry

This whole argument is so familiar. When I was a pup, I was arguing the “it can’t be done” side. Then my boss hit me over the head with a 2×4. Now I understand that it can be done.

Of course a half dozen calls in the middle of the night to come to the plant and reboot the system may have also had something to do with seeing the light.

Judith Curry

Dan,

Verification for 2006/2007 ECMWF forecasts is at

Click to access tm547.pdf

The model is proprietary to the EU, so the code is not publicly available (it is heavily scrutinized within the EU, and there is much documentation that is not publicly accessible).

The Technical Notes in an aggregate sense constitute the documentation that is publicly available

If I were going to audit a model, i would do the NCAR CCSM, far and away the one with best documentation and cleanest code (at least on ocean and ice part).

HMcCard

Dan,

Re: #173

I share many of your views but didn’t understand your comment near the end that “Nonlinearity doesn’t count. All …” It seems to me that nonlinearity makes “all the difference in the world.”

Gerald Browning

John (#156),

The most common coordinate system used on the sphere for hydrostatic models is the spherical coordinate system in the horizontal direction and a normalized pressure type of coordinate, e.g. p/p_s (p is pressure and p_s is surface pressure), in the vertical direction. This choice of the vertical coordinate has much to do with historical development.
There are many others, e.g. stereographic coordinates. But to understand the problems with the continuum system, Cartesian coordinates are sufficient and I wouldn’t spend tons of time on learning coordinate transformations at this point just for this purpose.

Jerry

Larry

There is another model for verification, of course. It’s called open source. It didn’t take long for the world to find the famous excel bug that Microsoft’s V&V never found. Turning the code loose to 1000 fiddlers will find the problems in short order.

However, the same people who are resisting V&V are also resisting open archiving. Funny how that works.

Gerald Browning

lucia (#190).

Well you almost got it right. I actually jumped into two different threads
with the same comment to ensure that Judith Curry and Roger Pielke Jr. saw the comment. I asked them if they were interested in reviewing the published version (QJRMS) of the Jablonowski manuscript because Judith spouted the standard line about the need for more resolution that is
complete nonsense for reasons that are becoming apparent. Judith’s review is exactly what I expected give her source of funding and ties to the meteorological community. After thoroughly reviewing the manuscript,
I noticed that the plots of the perturbation were stated not to be in the published manuscript (QJRMS), but in the NCAR Tech Note series. When I went to that Tech Note (how to access the Tech Note is discussed in comment #103 above), I found the missing plots. But even more of interest the missing section 5.3 on “Sensitivity to the diffusion coefficients”. I was
fully aware of the changes of the diffusion coefficient of the unphysical hyperviscosity as the resolution is reduced that can be seen in the published manuscript (QJRMS) in Table 1 and the vorticity cascade in Figure 9. These were exactly why I wanted the general reader to be able to see this manuscript. The serendipity of finding the missing section in the NCAR Tech Note will just make the discussion easier to understand for the general reader. I also think that the fact that the missing section was not published is very telling.

Jerry

lucia

Jerry– Will you ever get around to posting your review of J&W and explaining what you think is significant? Particularly in the materials that were omitted in the version you asked Judy, Roger Jr. and Roger Jr. to review?

It’s find to tell us you think the fact those bits were omitted is signifcant, but it might be more useful if you told people why you think it’s significant. The most common reason material in a long tech note is omitted from the journal article is that it is judged to be less important by the authors.

William– Quite honestly, I think we agree, not only on things you say, but more generally. Better regulated formal V&V practices would benefit this area and make the results of GCM’s more convincing to many. At the same time, the fact that V&V does not meet the standards of many engineers doesn’t mean the GCM’s just spew forth drivel.

Larry

Dr. Curry, just to illustrate that it isn’t just Rush Limbaugh getting into the amateur climate expert game:

http://www.businessandmedia.org/articles/2008/20080206170159.aspx

John Creighton

#166, Judtih, thanks for the link, unfortunately the equations they start with don’t look like something recognizable to people not in the field. Instead of starting with something like stokes equations they give you a reference:
http://www.ccsm.ucar.edu/models/atm-cam/docs/description/node48.html#simmons81

As to where to look up the derivation.

John Creighton

Hmmmm…..This link:
http://www.cptec.inpe.br/eta/produtos/eta/Docs/wmo2.doc

may tell me how to go from the equations mentioned by Gerald Browning (post #152)
http://www.climateaudit.org/?p=2696#comment-208514

to the continuous equations used in the climate models. It will be tomorrows reading for me.

steven mosher

RE 196. you’ll never knit in my factory.

Larry, are you an embedded type?

If so, drop Anthony Watts an email and tell him to cough up my email. reference this
post in your mail so he knows I’m cool with it.

steven mosher

RE 183. Thanks Dr. Curry. I already like what I see there.

steven mosher

RE 183. Hey Dan Hughes. Dr. Curry mentioned NCAR CCSM. I like what I see.

When you look at the working groups the ONLY POSITION without a Liason

is…………………………………..

Software engineering.

http://www.ccsm.ucar.edu/working_groups/

Now, we have been tackling this thing from the wrong perspective.

Figger a way to get on this global warming gravy train. The software I’ve
seen is retarded. We both know that better code can often beat better iron. Saving instructions
in a GCM MAY VERY WELL SAVE THE PLANET! why let the chipheads get the dollars.

I’m thinking of a consulting group. snappy climate science name will come to mind.
like “carbon neutral code”

We’ll make billions.

Judith Curry

John, I am googling “introduction to climate modeling”, not finding much. Here are a few things that should be of some use (i found these via google, didn’t look at them too closely)

http://www.climateprediction.net/science/model-intro.php
http://www.carleton.edu/departments/geol/DaveSTELLA/climate/climate_modeling_1.htm
http://www.atmos.washington.edu/honors_220/model/model_instructions.html

Some books:
A Climate Modeling Primer by Henderson-Sellers and McGuffie
Fundamentals of Atmospheric Modelling by Mark Jacobsen

Gerald Browning

All,
Here are the inviscid equations of motion so that I can discuss them by name.
Cont:
Momentum:
Pressure:
Total:

gb

# 210

Jerry, viscosity and dissipation are not the same. A different viscosity does not have to imply a different dissipation. Two models with different viscosities can have the same mean dissipation.

Gerald Browning

All,

Let me discuss a few mathematical properties of the inviscid equations.
There are five equations in the system (the continuity equation,
the three momentum equations, and the pressure equation).
Then there are 5 different motions associated with the system, e.g. two sound waves, two inertial/gravity waves, and an advective type of motion.

These motions can be seen by linearizing the system about a given state and computing the associated frequencies of motion (after Fourier transforming the system in space).

Now meteorologists like to linearize about simple states such as one that is at rest. This makes the ensuing computation of the frequencies easier, but is extremely misleading. In this case the system is hyperbolic, i.e. all of the frequencies are real (or all of the eigenvalues are purely imaginary).

However, if one linearizes about a more realistic state, one finds that the frequencies are not all real (Browning and Kreiss reference available on request). In the case of the hydrostatic system,
the complex frequencies lead to ill posedness (exponential growth that is unbounded in any small time period). This has been shown to occur in the numerical examples on the Exponential Growth in Physical Systems #1 thread (comments 165 and 166). In the nonhydrostatic system, the complex frequencies lead to bounded (but extremely fast) exponential growth. This type of growth will destroy the numerical accuracy of any numerical method in a matter of hours.

Jerry

Mike B

I have a couple of quick questions:

1) Are there four equations or five?

2) Can you give me a quick peek or preview of what you intend to show/prove?

Thanks in advance.

John Creighton

Thank you Gerald Browning,
for your posts. I look forward to further insight. About a week ago I tried to summarize what I could find on line here:

http://www.climateaudit.org/phpBB3/viewtopic.php?f=4&t=99

I’m sure you will take me while past my feeble attempts:

I think what is important is not so much that the model can predict weather. What is important is that it gets the ensemble average of heat transport correct. My biggest area of concern is that they compute temperature in a way to yield stable results. I don’t know enough about how temperature is computed in the model to know if this is the case. I do know though that purely adiabatic equations are at best metastable which means without considering the energy exchange in and out of the system the energy in the system will drift.

Gerald Browning

John (#217),

One additional point. Look carefully at the plots of the vorticity in the Jablonowski manuscipt to see how quickly the vorticity cascades down to smaller scales when the steady solution is perturbed by a very small in magnitude perturbation. And note that although the features appear to be small on the plots, they are in fact quite large, i.e. well over 100 km in lateral dimensions, i.e. the models have not even resolved any
mesoscale features let alone fronts or hurricanes.

Jerry

JamesG

Judith says: “The issue that concerns many of you is not technically V&V but whether or not the models are “perfect” enough to prescribe policy. The models are not perfect and are not being used to prescribe policy. They are being used in the assessment of risk that we are facing from global warming”

It is just basic adequacy we seek. The recent external verification tests by Douglass, Christy et al. showed that the models get the basic theory wrong and are thus inadequate. The climate community response amounted to saying “the large uncertainties in the models overlap the probable uncertainties in the observations.” In other words the models are actually worse than you thought. The only two tests that I’ve seen the climate modeling camp boast about have been the Pinatubo volcano test, and the Hadley “take away AGW and you can’t model 20th century temperature” test. The former modeled short-term, natural cooling, not AGW warming, and the latter was very obvious circular reasoning which became redundant when Hadley later admitted their natural variability assumptions were wrong anyway. One must presume that all the other internal tests are even worse than these flag-wavers. So are they good enough for risk assessment? The answer is clearly no, and nobody should pretend otherwise!

Gerald: Since Wunsch and Seager tell us the main driver for the winds and hence the ocean currents is the actual rotation of the earth then maybe we should have angular momentum in that equation list, otherwise is there much point?

Tom Vonk

JamesG #221

Gerald: Since Wunsch and Seager tell us the main driver for the winds and hence the ocean currents is the actual rotation of the earth then maybe we should have angular momentum in that equation list, otherwise is there much point?

We are dealing with fluids here .
The N-S equations are technically field equations solving for the velocity field by writing F=ma , energy conserved , mass conserved .
There is nothing that “rotates” and the field is fully defined when you know the velocity components for every point and every time .
If you look at a flowing river , you see that rotating pseudo structures are not conserved – they appear , dosappear and change all the time .
They are only transient large scale manifestations of the velocity field .
However as the local spiral like structures are common , it is interesting to consider closed field lines .
Gerry already mentioned vorticity , it will appear when he’ll talk about it .

John Creighton

#223, Don’t take my word for it but I think that that in most GCM models that the rotation of the earth is approximated by a coriolis term that is latitude independent.

Gerald Browning

John (#225),

That is correct. The f in the momentum equations is the Coriolos force.
In most models only the vertical term of f is included. On the largest scales, f plays a very important role. But as one moves to smaller scales of motion the Coriolis terms play less and less of a role.

Jerry

Gerald Browning

John (#220),

Actually the two dimensional incompressible ( viscous) Navier-Stokes equations have been solved numerically and the numerical solutions shown to converge if the minimum scale of motion is resolved. The estimates of the minimal scale for a fixed kinematic viscosity are very complicated but have been shown to be very accurate. In two dimensions, no additional assumptions are needed to obtain the minimal scale estimate. But in three dimensions one must assume that the velocity is bounded. Then the
estimates also apply there. I can supply appropriate mathematical references for the estimates. But you might want to look at the Math. Comp.
article by Heinz and me that shows that an incorrect type (hyperviscosity
or spectral chopping ) or size will lead to very different results.

Jerry

Gerald Browning

James G (#224),

There is a mathematical theorem that states that if we can solve the homogeneous (unforced) system of equations, then we can solve the forced version. At this point we are not discussing any forcing terms other than
the gravitation term. After linearization about a general state of motion,
that term will appear as a large undifferentiated term. Its importance in the frequency computation has been discussed above. Given that there are serious problems with the unforced (except for gravitation) system,
there is no reason to discuss other physical forcings – they will not
help the situation. That is why we are discussing only the inviscid system and dynamical cores at this point.

Jerry

Gerald Browning

John (#232),

Actually Heinz Kreiss proposed that we write a book containing info from many of the manuscripts I have cited. But a book is a considerable amount of work with little career gain (except for ego) when you are under pressure to produce manuscripts. 🙂

Heinz does have a book on the Navier-Stokes equations that I highly recommend, but you will have to wade thru some mathematics.

And everyone can just call me Jerry. 🙂

Jerry

Judith Curry

Gerald, I don’t have much time to write here at the moment, but you seem to be misleading the readers regarding viscosity. By comparing molasses to air in terms of molecular vlsocosity, this simply is not relevant. The molecular viscosity of air simply is not relevant in modelling of the atmosphere beyond a scale of order meters. Stable and accurate solutions to the Navier Stokes equations for large scale atmospheric flows are obtained by filtering sound waves (the hydrostatic assumption) and filtering high frequency unresolved gravity waves. These solutions converge generally to the solutions of higher resolution models (nonhydrostatic, including more of the higher frequency gravidty waves). The unresolved degrees of freedom (e.g. subgrid cascade of energy, momentum) are addressed through parameterization of the subgrid motions. Such solutions are highly robust for large-scale atmospheric circulations that are of relevance to climate modelling. The success of numerical weather prediction at predicting say the 500 hpa height field on the time scale of the lifetime of a baroclinic wave (nominally 4-7 days) is a testimony to the credibility of these solutions

You are also neglecting to consider the chaotic nature of any solutions to these equations, and the need for an ensemble of solutions for any kind of comparison, evaluation, etc.

Lets focus down in scale, say to a large eddy simulation model, with resolution of say 2 m. Even such models have parameterizations of the subgrid motions. I have addressed this issue specifically in a paper on the LES of the stable boundary layer.

Click to access Kosovic_JAS57.pdf

Are you really prepared to say that all gas phase fluid dynamics modelling (in aerospace, mechanical engineering, physics, geoscience, etc and in all their applications) is incorrect?

Judith Curry

Lucia, I agree that we accept DNS as an accurate solution. LES, with appropriate subgrid parameterization, is consistent with DNS (Kosovic published some J. Fluid Mech papers on this). Turbulent diffusivities used in atmospheric dynamical cores are parameterized based on LES simulations. The point is the parameterization for turbulent diffusivity in a large-scale (global) atmospheric model does not dominate the solution at all, as evidenced by sensitivity studies to different treatments of the diffusivity. The sensitivity of climate models to this issue wrapped up in boundary layer cloud issue, but the issue is the cloud part not the turbulent diffusivity part. I will try to get some more specifics later this weekend if i have time.

Mike B

The molecular viscosity of air simply is not relevant in modelling of the atmosphere beyond a scale of order meters. Stable and accurate solutions to the Navier Stokes equations for large scale atmospheric flows are obtained by filtering sound waves (the hydrostatic assumption) and filtering high frequency unresolved gravity waves. These solutions converge generally to the solutions of higher resolution models (nonhydrostatic, including more of the higher frequency gravidty waves).

Sorry to be such a stickler, but I want to make absolutely sure I understand what you’re saying, particularly in the highlighted portion . Are you saying that viscocity is not relevant because of the way the model accounts for it? Or are you saying that in the real world, the viscocity of air is not relevant? Or have I missed the point entirely?

Mike B

Thanks Lucia, I think I’m following…

In flows of this type, the parameterization of the mean effect of subgrid motions introduced a diffusive type term. The simplest form looks exactly like Newtonian viscosity– but contains a turbulent viscosity. The turbulent viscosity is so much larger than the viscosity of air that one can neglect the real viscosity. (It’s in the round off error of any estiamte of turbulent viscosity in many problems.)

But then to get at the crux of Judy’s complaint about Jerry’s air/molasses comment, is the turbulent viscocity independent of the fluid?

gb

The aim of the turbulent viscosity is to get the correct dissipation rate of of turbulent kiinetic energy. If the small scales are not resolved this energy will not be dissipated and accumulate. You need then to introduce a subgrid model, or turbulent viscosity.

Judith Curry

Not entirely clear i’m interpreting correctly all the questions. Yes, the Navier-Stokes equations when applied to the atmosphere are Reynolds averaged. the “turbulent viscosity” depends on the scales of motions that are not resolved by the model, this is where the molasses analogy breaks down. I’m having a hard time figuring out what Gerald’s issue is, it isn’t making much sense to me. The general modelling techniques used in the atmospheric dynamical core, and the treatment of unresolved degrees of freedom are fairly standard as a general approach across many different applications of fluid dynamics. for example, i think Gerald also complained about the treatment of the upper boundary. Well, the main point is that the upper boundary is far away from the region of interest, and the wave damping at the upper boundary is designed so as not to have any spurious boundary effects contaminate the interior of the solution that is the region of interest.

The complicated issue associated with the so called turbulent viscosity is tied up in the interplay with the convective and cloudy boundary layer parameterizations. These are a small deal for baroclinic waves, but more important in climate simulations. This is the parameterization part of the problem, that is the key issue to worry about, not the dynamical core.

Judith Curry

The NCAR model documentation gives a good description of their treatment of “turbulent diffusivity”

http://www.ccsm.ucar.edu/models/atm-cam/docs/description/
go to section 4.11

Judith Curry

Lucia, the molasses/air analogy is a bad one, we are talking about totally different reynolds numbers. Turbulence and chaos dominates in the atmosphere, while molasses is dominated by viscosity. In the atmosphere, this is determined by the flow and not the material.

John Creighton

It is my understandiIt is my understanding that the velocity due to turbulence is a consequence of the bulk flow of air while viscosity is due to the molecular flow of air. The amount of turbulence depends on the Reynolds number which depends on the viscosity. Therefore these turbulent kinematic viscosities are not independent of the fluid.

I have no problem with parameterize these turbulent viscosities if they can be shown to be realistic. It should be clear that since these viscosities depend on the length scale. Therefore in areas of higher turbulence, near the ground or near a hurricane that these viscosities should be higher then they should be in calm areas high in the atmosphere.

If these turbulent viscosities can be modeled well empirically then to me it would seem acceptable. I would however, like a more fundamental way of capturing the dependence of this viscosity on turbulence and that is why I thought fractal geometry might be relevant.

http://www.me.jhu.edu/~meneveau/pubs-fractals.html
http://www.climateaudit.org/phpBB3/viewtopic.php?f=4&t=112

Another concern is if the bulk flow of air effectively increases the turbulence does it also effectively increase the conduction? That is how much energy is exchanged by turbulent flows which are two small for the model to capture? Do the models effectively capture this energy exchange.

I have a feeling that I am going to need to read texts suggested by both Jerry and Judith to decide how significant the concerns mentioned by Jerry are.

steven mosher

Lucia,

I just picked a fight for you over at Atmoz. I promised that you would unclown the boys
who had beclowned themselves. Sorry.

Oh yes. I named Lumpy! not Sadlov. people always confuse us. I’ll forgive you.

lucia

Judy– I thought we’d agree on this. 🙂

steve moscher: So you were the one who sicked that nearly incoherent commenter on me. I think I’ve been accused of not liking approximations. Clearly, news of “Lumpy” has not gotten out! Off to unthreaded!

John M

OK. I see it now here.

As Lucia says, grist for Unthreaded.

Gerald Browning

All,

I think the correct statement is that when a numerical model cannot resolve the small scales of motion in the real solution, one must add some form
of damping to prevent those scales from forming in the model.
There have been many forms used, but little or no proof that they
will provide the real solution. For example, it has been shown mathematically (and demonstrated numerically) that the minimal scales for the regular NS second order diffusion operator are quite different than the sclaes for fourth order hyperviscosity (see Math. Comp. article).

Jerry

John Creighton

Jerry, so you are saying the the hyper viscosity is incorrectly parameterized?

Gerald Browning

John,

Do not be misled or side tracked. The issues can be understood from the simple equations I gave and an appropriate discussion of the gimmicks that are used in the dynamical core manuscript on this thread. The rest
is minor in comparison to these issues.

Jerry

Judith Curry

Gerald, you are swatting at gnats here. for any conceivable application of a global atmospheric model, say for weather forecasting or climate sensitivity experiments, the issues you raise are irrelevant. These models, even when run at quite low resolution, provide credible solutions in terms of comparing with higher resolutions simulations and verification with observations. Can more efficient, exact solutions to the dynamical core be configured? I hope so and expect so. Do these little gnats get in the way of weather prediction or climate sensitivity studies? not at all. If you want to critique these models and sow doubt about their results, you need to do much better than this.

lucia

Gerry,

As an engineer that lives off of numerical models,

My main work in fluids is as an experimentalist not a modeler. I told Judy this earlier in the thread. But, in fluid dynamics research these days, experimentalist generally need to be attentive of the state of modeling because certain types of measurements of are particular importance to modelers who need to find decent parameterizations.

Many things are done in engineering and science with no formal mathematical proof models. Every one knows that after the engineers figured out things and came up with models for boundary layers, the mathematicians howled. Meanwhile, the scientists jumped right in and used it in there studies (where it worked quite well, thank you very much.) Then 70 years or so later, some mathematician to came up with formal mathematical proof that boundary layer theory was ‘true’ in a mathematical sense.

Do you really think complaining about lack of formal mathematical proof is going to stop people from modeling?

Mike B

#245

The complicated issue associated with the so called turbulent viscosity is tied up in the interplay with the convective and cloudy boundary layer parameterizations. These are a small deal for baroclinic waves, but more important in climate simulations. This is the parameterization part of the problem, that is the key issue to worry about, not the dynamical core.

Are you conceding that there are problems with the dynamical core, but that those problems are small relative to other problems with the models?

What criteria do you use to evaluate the models? Is there a standard set of criteria?

I gots all my quatloos riding on this thing.

If I decide both Jerry and Judy are all wet do I get to keep my quatloos? 😉

Gerald Browning

Judith Curry,

Please compute the frequencies for the system I gave above linearized
about a general state or at least read the article in the book mentioned many times before on this site. (I am willing to cite it again if you want).

How about a mathematical proof (or at least convergent numerical solutions) to lend some credibility to your statements. It is very clear that the “review” you did of the Jablonowski manuscript is typical of how many articles get in print.

If you knew anything about the details of the manuscript, you would have asked the pertinent scientific questions that I have raised and you continue to skirt.

Jerry

Gerald Browning

All,

Now that it has been revealed how to carefully interpret some of the results in the Jablonowski QJRMS article, I ask that you read the missing section in the NCAR Tech Note series about the sensitivity of the solutions to different coefficients of the hyperviscosity to further support my arguments. As I have clearly stated, this problem can be seen without the missing section, but that section will help to clarify the discussion about too large of dissipation impacting the accuracy of the numerical solution
and the subsequent need for unphysical forcing to “adjust”
the spectrum.

Jerry

lucia

Jerry–
Once again, so?

You need to learn to
a) state your actual point,
b) to the support it and
c) make sure the point you are wishing to make and the argument to support that specific are contained in a single comment. (Or, should you ever write a post, in the post itself.)

The elliptic statements, though they may touch on some truths, are pointless as no one knows how to connect them to any argument or claim of interest.

Judith Curry

Gerald, please get over yourself. my credentials are nontrivial in this area, but I don’t tout them (trust me, i’m no lady), rather i try to make coherent arguments and provide references and explanations. I am pretty much through on this thread, i don’t see it going anywhere. In the unfortunate absence of bender, i leave it to people to decide what if anything they have learned here and who they learned it from.

John Creighton

I’ve learned I need to do more reading to decide the winner:

Judith mentioned:
A Climate Modeling Primer by Henderson-Sellers and McGuffie
Fundamentals of Atmospheric Modelling by Mark Jacobsen

Jerry mentioned some book by Heinz but I didn’t get a title
GB mentioned:

Vallis ‘Atmospheric and oceanic fluid dynamics’ Cambridge University Press. By Geoffrey K. Vallis
I hope that one of these texts address the concerns of Jerry and if not I hope that one of them will at least take me far enough to toughly understand the papers which address Jerry’s concerns.

steven mosher

re 281. That was evil bender trying to remove you from the fight. He’s no dummy.

Some day we’ll chat about mixing things with vastly different viscosities.

lucia

A book ever avid knitter stocks on her shelf. . .

hmmm preview doesn’t show image?

Gerald Browning

Judith Curry,

please answer the very specific mathematical questions I asked you ….

1) Compute the frequencies or read our reference on the computation of frequencies for a realistic atmospheric state.

2) What is the impact of those complex frequencies on the hydrostatic and nonhydrostatic continuum equations.

3) What is the impact of those complex frequencies on a numerical model of the atmosphere.

2) Clearly state the impact of an incorrect spatial spectrum due to incorrect dissipative operators on the tuning that is required to make the spatial spectrum appear realistic.

3) The atmosphere receives information from above any artificial lid in a numerical model. How is that info transmitted when the numerical model has an artificial boundary at a predetermined height, i.e. there is no flow of information from above the artificial lid.

4) The equations of motion change nature at higher altitudes, e.g. they change to the plasma equations. Is this change taken into account.

…
Jerry

SM- Jerry, please be more moderate in your tone.

Gerald Browning

All,

I have made my case quite clear. If one reads my comments and looks at the QJRMS manuscript as I have indicated, one can see there are serious problems with the continuum equations and their numerical approximations. (dynamical cores). No amount of wordsmithing can overcome these problems.

I will come back to this thread to answer a few more questions. But the mathematical analysis of the frequencies, the demonstrations of the
problems with the continuum systems using convergent numerical solutions
on the Exponential thread, and the dynamical core manuscript itself
should make the issues very clear.

Jerry

John Creighton

#227 Jerry, required or If you have any references I would appreciate you posting them so that in the future when hopefully I have more background on this stuff I can better investigate your concerns.