I [Dan Hughes] posted a short discussion of some software Verification and Validation issues on another thread. Here are some additional thoughts.
I have a few questions for anyone who have answers. I consider these issues to be essentially show-stoppers as far as use of the results of any of the AOLBCGCM codes and all supporting codes use in all aspects of climate-change analyses, for either (1) archival peer-reviewed publications, (2) providing true insight into the phenomena and processes that are modeled, and most importantly (3) for decision-making relative to public policies. Any and all professional software developers would absolutely require that all of the issues to be mentioned below be sufficiently addressed and documented before using any software for applications in the analyses areas for which it was designed.
In no particular order, as each of the following is very important, can anyone provide documented information about:
(1). Audited Software Quality Assurance (SQA) Plans for any of the computer software that is used in all aspects of climate-change analyses.
(2) Documentation of Maintenance under audited and approved SQA procedures of the “Åfrozen’ versions that are used for production-level applications.
(3) Documentation of the Qualifications of the users of the software to apply the software to the analyses that they perform.
(4) Documentation of independent Verification that the source coding is correct relative to the code-specification documents.
(5) Documentation of independent Verification that the equations in the code are solved correctly and the order of convergence of the solutions of the discrete equations to the continuous equations has been determined.
(6) Sufficient information from which the software and its applications and results can be independently replicated by personnel not associated with the software.
(7) It is my impression that use of ensemble averages of several computer calculations that are based on deterministic models and equations is unique to the climate-change community in all of science and engineering. I can be easily corrected on this point if anyone can provide a reference that shows that the procedure is used in any other applications. (The use of monte carlo methods to solve the model equations is not the same thing). The use of ensemble averaging and the resulting graphs of the results makes it very difficult to gain an understanding of the calculated results; rough long-term trends are about all that can be discerned from the plots.
(8) Documentation that shows that the codes always calculate physically realistic numbers. For example, the time-rate-of-change of temperature, say, is always consistent with the energy equations and is not the results of numerical instabilities or other numerical solution methods problems.
(9) Documentation in which the mathematical properties (characteristics, proper boundary condition specifications, well- (or ill-) posedness, etc.) of all the continuous equations used in a code have been determined. Do attractors exist, for example.
(10) Documentation in which it has been shown analytically that the system of continuous equations used in any AOLBCGCM model has the chaotic properties that seem to be invoked by association and not by complete analysis. Strange-looking output from computer codes does not prove that the system of continuous equations possess chaotic characteristics. Output from computer codes might very likely be results of modeling problems, mistakes, solution errors, and/or numerical instabilities.
Invoking/appealing-to an analogy to the Lorenz continuous equations is not appropriate for any other model systems. The Lorenz model equations are a severely truncated approximation of an already overly simplified model. The wide range of physical time constants and potential phase errors in the numerical solutions almost guarantees that aperiodic behavior will be calculated.
Especially true considering the next item.
(11) Documentation in which it has been determined that the discrete equations and numerical solution method are consistent and stable and thus the convergence of the solution of the discrete equations to the continuous equations is assured. Actually I understand that the large AOLBCGCM codes are known to be unable to demonstrate independence of the discrete approximations used in the numerical solution methods. The calculated results are in fact known to be functions of the spatial and temporal representations used in the numerical solutions. This characteristic proves that convergence cannot be demonstrated. Consistency and stability remain open questions.
(12) Documentation in which it is shown that the models/codes/calculations have been Validated for applications to the analyses for which it has been designed.
All software, each and every one, that is used for analyses of applications the results of which might influence decisions that affect the health and safety of the public will have addressed all these issues in detail.
If my understanding of the status of these critical issues is correct I can only conclude:
(1) The software used in the climate-change community does not meet the most fundamental requirements of software used in almost all other areas of science and engineering. Almost none of the basic elements of accepted software design and applications for production-level software are applied to climate-change software.
(2) The calculated results cannot be accepted as correct and valid additions to the peer-reviewed literature of technical journals.
(3) The software should never be used in attempts to predict the effects of changes in public policy (fuel sources for energy production, say) on the climate; neither short- or long-range.
(4) The calculated results are highly likely not correct relative to physical reality.
I will say that item (11) is in fact a totally unacceptable characteristic for any engineering and scientific software. The results from any codes that have this property would be rejected for publication by many professional engineering organizations. I can be easily corrected if anyone can point me to calculated results from any other areas of science and engineering in which the fact that the numerical methods are known to be not converged is accepted as being, well, acceptable practice. Buildings, airplanes, bridges, elevators, flight-control systems, nothing in fact, are never designed under this approach.
Actually all professional software development projects require far more than the information that I discuss above. Any textbook on software development can be consulted for a more complete listing and detailed discussions. Almost all the large complex AOLBCGCM codes have evolved over decades from software that was significantly more simple than the present versions. These codes have not been designed and built “Åfrom scratch’ on a “Åclean piece of paper’. Newly-built software, designed and constructed under SQA plans and associated procedures require very significantly more documentation and independent review, Verification, and Validation than that mentioned above.
Several have mentioned that source listings for some of the AOLBCGCM codes are available on the Web. This is very true. And it is equally true that in theory the source coding could be used to Verify the coding. However, in order for this to be a useful exercise we need some kind of specification of what was intended to be coded into the code. In the absence of this information we cannot develop objective metrics for judging that the coding is correct.
The level of detail needed for Verification of the coding is generally many times greater than that typically available for many software products. Because the objective is Verification of the coding a specification of all the equations in the code is needed. For legacy software that has evolved over decades of time, this information is usually contained in a theory and numerical methods manual in which the continuous equations and the discrete approximations to these and the numerical solution methods used to solve the discrete equations are described in detail. A computer code manual in which the structure of the code is describe in sufficient detail that independent outside interests can understand the source code would also be helpful in any attempts to Verify the coding. I have not been successful in finding such manuals for AOLBCGCM codes.
As someone mentioned, as taxpayer-funded software, this documentation should in fact be readily available. It is not.
The level of documentation detail required for an independent V&V and SQA effort is enormous. Chip H above has mentioned documentation required for development and Verification of the coding. In order to get a even better handle on the exact nature of the models, methods and codes and their applications other documentation should be available. This documentation includes:
Volume 1: Model Theory and Solution Methods Manual. A theory manual in which the derivations of each and every equation, continuous and discrete, are given in sufficient detail that the final equations for the models can be obtained by independent interests.
Volume 2: Computer Code Manual. A computer code manual in which the code is described in sufficient detail that independent outside interests can understand the source code.
Volume 3: User’s Manual. A user’s manual that describes how to develop the input for the code, perform the calculations and understand the results form the calculations.
Volume 4: Verification and Validation Manual. A manual or reports in which the verification and validation of the basic functions of the software and example applications are given.
Volume 5: Qualification Manual. Additional manuals or reports in which the models and methods, software and user are demonstrated to be qualified for application to analyses of the intended application areas.
Other reports and papers can be used to supplement the above documentation. These then become a part of the official record of the software relative to independent V&V and SQA efforts.
The coding cannot be Verified given the level of documentation that I have found so far. This was only a first attempt, and by someone not familiar with the codes. However I think it does give a first glimpse into the lack of sufficient documentation for verifying the coding.