ASTM D6589-00

NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
Designation: D 6589 – 00
Standard Guide for
Statistical Evaluation of Atmospheric Dispersion Model
Performance
This standard is issued under the fixed designation D 6589; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (e) indicates an editorial change since the last revision or reapproval.
1. Scope 1.4 This guide offers an organized collection of information
oraseriesofoptionsanddoesnotrecommendaspecificcourse
1.1 This guide provides techniques that are useful for the
of action. This guide cannot replace education or experience
comparison of modeled air concentrations with observed field
and should be used in conjunction with professional judgment.
data. Such comparisons provide a means for assessing a
Not all aspects of this guide may be applicable in all circum-
model’s performance, for example, bias and precision or
stances. This guide is not intended to represent or replace the
uncertainty, relative to other candidate models. Methodologies
standard of care by which the adequacy of a given professional
for such comparisons are yet evolving; hence, modifications
service must be judged, nor should it be applied without
will occur in the statistical tests and procedures and data
consideration of a project’s many unique aspects. The word
analysis as work progresses in this area. Until the interested
“Standard” in the title of this guide means only that the
parties agree upon standard testing protocols, differences in
document has been approved through the ASTM consensus
approach will occur. This guide describes a framework, or
process.
philosophical context, within which one determines whether a
1.5 This standard does not purport to address all of the
model’s performance is significantly different from other
safety concerns, if any, associated with its use. It is the
candidate models. It is suggested that the first step should be to
responsibility of the user of this standard to establish appro-
determine which model’s estimates are closest on average to
priate safety and health practices and to determine the
the observations, and the second step would then test whether
applicability of regulatory limitations prior to use.
the differences seen in the performance of the other models are
significantly different from the model chosen in the first step.
2. Referenced Documents
An example procedure is provided inAppendix X1 to illustrate
2.1 ASTM Standards:
an existing approach for a particular evaluation goal. This
D 1356 Terminology Relating to Sampling and Analysis of
example is not intended to inhibit alternative approaches or
Atmospheres
techniques that will produce equivalent or superior results. As
discussed in Section 6, statistical evaluation of model perfor-
3. Terminology
mance is viewed as part of a larger process that collectively is
3.1 Definitions—For definitions of terms used in this guide,
referred to as model evaluation.
refer to Terminology D 1356.
1.2 This guide has been designed with flexibility to allow
3.2 Definitions of Terms Specific to This Standard:
expansion to address various characterizations of atmospheric
3.2.1 atmospheric dispersion model, n—an idealization of
dispersion, which might involve dose or concentration fluctua-
atmospheric physics and processes to calculate the magnitude
tions, to allow development of application-specific evaluation
and location of pollutant concentrations based on fate, trans-
schemes, and to allow use of various statistical comparison
port, and dispersion in the atmosphere. This may take the form
metrics. No assumptions are made regarding the manner in
of an equation, algorithm, or series of equations/algorithms
which the models characterize the dispersion.
used to calculate average or time-varying concentration. The
1.3 The focus of this guide is on end results, that is, the
model may involve numerical methods for solution.
accuracy of model predictions and the discernment of whether
3.2.2 dispersion, absolute, n—the characterization of the
differences seen between models are significant, rather than
spreading of material released into the atmosphere based on a
operationaldetailssuchastheeaseofmodelimplementationor
coordinate system fixed in space.
the time required for model calculations to be performed.
3.2.3 dispersion, relative, n—the characterization of the
spreading of material released into the atmosphere based on a
This guide is under the jurisdiction ofASTM Committee D22 on Sampling and
AnalysisofAtmospheresandisthedirectresponsibilityofSubcommitteeD22.11on
Meteorology.
Current edition approved Sept. 10, 2000. Published November 2000. Annual Book of ASTM Standards, Vol 11.03.
Copyright © ASTM International, 100 Barr Harbor Drive, PO Box C700, West Conshohocken, PA 19428-2959, United States.
D 6589
coordinate system that is relative to the local median position 4.3 In assessing the performance of air quality models to
of the dispersing material. characterize a particular evaluation objective, one should
3.2.4 evaluation objective, n—a feature or characteristic, consider what the models are capable of providing. As dis-
which can be defined through an analysis of the observed cussed in Section 7, most models attempt to characterize the
concentration pattern, for example, maximum centerline con- ensemble average concentration pattern. If such models should
centration or lateral extent of the average concentration pattern provide favorable comparisons with observed concentration
as a function of downwind distance, which one desires to maxima,thisisresultingfromhappenstance,ratherthanskillin
assess the skill of the models to reproduce. the model; therefore, in this discussion, it is suggested a model
3.2.5 evaluation procedure, n—the analysis steps to be be assessed on its ability to reproduce what it was designed to
takentocomputethevalueoftheevaluationobjectivefromthe produce, for at least in these comparisons, one can be assured
observed and modeled patterns of concentration values. that zero bias with the least amount of scatter is by definition
3.2.6 fate, n—the destiny of a chemical or biological pol- good model performance.
lutant after release into the environment. 4.4 As an illustration of the principles espoused in this
3.2.7 model input value, n—characterizations that must be guide, a procedure is provided inAppendix X1 for comparison
estimated or provided by the model developer or user before of observed and modeled near-centerline concentration values,
model calculations can be performed. which accommodates the fact that observed concentration
3.2.8 regime, n—a repeatable narrow range of conditions, values include a large component of stochastic, and possibly
defined in terms of model input values, which may or may not deterministic, variability unaccounted for by current models.
be explicitly employed by all models being tested, needed for The procedure provides an objective statistical test of whether
dispersion model calculations. It is envisioned that the disper- differences seen in model performance are significant.
sion observed should be similar for all cases having similar
5. Significance and Use
model input values.
5.1 Guidance is provided on designing model evaluation
3.2.9 uncertainty, n—refers to a lack of knowledge about
performance precedures and on the difficulties that arise in
specific factors or parameters. This includes measurement
statistical evaluation of model performance caused by the
errors, sampling errors, systematic errors, and differences
stochastic nature of dispersion in the atmosphere. It is recog-
arising from simplification of real-world processes. In prin-
nized there are examples in the literature where, knowingly or
ciple, uncertainty can be reduced with further information or
unknowingly, models were evaluated on their ability to de-
knowledge (1) .
scribe something which they were never intended to charac-
3.2.10 variability, n—refers to differences attributable to
terize. This guide is attempting to heighten awareness, and
true heterogeneity or diversity in atmospheric processes that
thereby, to reduce the number of “unknowing” comparisons.A
result in part from natural random processes. Variability
goal of this guide is to stimulate development and testing of
usually is not reducible by further increases in knowledge, but
evaluation procedures that accommodate the effects of natural
it can in principle be better characterized (1).
variability. A technique is illustrated to provide information
4. Summary of Guide
from which subsequent evaluation and standardization can be
derived.
4.1 Statistical evaluation of dispersion model performance
with field data is viewed as part of a larger process that
6. Model Evaluation
collectively is called model evaluation. Section 6 discusses the
6.1 Background—Air quality simulation models have been
components of model evaluation.
used for many decades to characterize the transport and
4.2 To statistically assess model performance, one must
dispersion of material in the atmosphere (2-4). Early evalua-
define an overall evaluation goal or purpose. This will suggest
tions of model performance usually relied on linear least-
features (evaluation objectives) within the observed and mod-
squares analyses of observed versus modeled values, using
eled concentration patterns to be compared, for example,
traditional scatter plots of the values, (5-7). During the 1980s,
maximum surface concentrations, lateral extent of a dispersing
attempts have been made to encourage the standardization of
plume. The selection and definition of evaluation objectives
methods used to judge air quality model performance (8-11).
typically are tailored to the model’s capabilities and intended
Further development of these proposed statistical evaluation
uses. The very nature of the problem of characterizing air
procedures was needed, as it was found that the rote applica-
quality and the way models are applied make one single or
tion of statistical metrics, such as those listed in (8), was
absolute evaluation objective impossible to define that is
incapable of discerning differences in model performance (12),
suitable for all purposes. The definition of the evaluation
whereas if the evaluation results were sorted by stability and
objectives will be restricted by the limited range conditions
distancedownwind,thendifferencesinmodelingskillcouldbe
experienced in the available comparison data suitable for use.
discerned (13). It was becoming increasingly evident that the
For each evaluation objective, a procedure will need to be
models were characterizing only a small portion of the ob-
defined that allows definition of the evaluation objective from
served variations in the concentration values (14). To better
the available observations of concentration values.
deduce the statistical significance of differences seen in model
performance in the face of large unaccounted for uncertainties
and variations, investigators began to explore the use of
The boldface numbers in parentheses refer to the list of references at the end of
this standard. bootstraptechniques(15).Bythelate1980s,mostofthemodel
D 6589
performance evaluations involved the use of bootstrap tech- algorithms. And as models become more complex, discerning
niques in the comparison of maximum values of modeled and the sensitivity of the modeling results to input parameter
observed cumulative frequency distributions of the concentra- variations becomes less clear; hence, two important tasks that
tionsvalues(16).Eventhoughtheproceduresandmetricstobe support model evaluation efforts are verification of software
employed in describing the performance of air quality simula- and sensitivity and Monte Carlo analyses.
tion models are still evolving (17-19), there has been a general
6.5.1 Verification of Software—Often a set of modeling
acceptance that defining performance of air quality models
algorithms will require numerical solution. An important task
needs to address the large uncertainties inherent in attempting
supportive to a model evaluation is a review in which the
to characterize atmospheric fate, transport and dispersion
mathematics described in the technical description of the
processes. There also has been a consensus reached on the
model are compared with the numerical coding, to insure that
philosophical reasons that models of earth science processes
the code faithfully implements the physics and mathematics.
can never be validated, in the sense of claiming that a model is
6.5.2 Sensitivity and Monte Carlo Analyses—Sensitivity
truthfully representing natural processes. No general empirical
and Monte Carlo analyses provide insight into the response of
proposition about the natural world can be certain, since there
a model to input variation. An example of this technique is to
will always remain the prospect that future observations may
systematically vary one or more of the model inputs to
callthetheoryinquestion(20).Itisseenthatnumericalmodels
determine the effect on the modeling results (22). Each input
of air pollution are a form of a highly complex scientific
should be varied over a reasonable range likely to be encoun-
hypothesis concerning natural processes, that can be confirmed
tered.Thetraditionalsensitivitystudies(22)weredevelopedto
through comparison with observations, but never validated.
better understand the performance of plume dispersion models
6.2 Components of Model Evaluation—Amodel evaluation
simulating the transport and dispersion of inert pollutants. For
includes science peer reviews and statistical evaluations with
characterization of the effects of input uncertainties on model-
field data. The completion of each of these components
ing results, Monte Carlo studies with simple random sampling
assumes specific model goals and evaluation objectives (see
are recommended (23), especially for models simulating
Section 10) have been defined.
chemically reactive species where there are strong nonlinear
6.3 Science Peer Reviews—Given the complexity of char-
couplings between the model input and output (24). Results
acterizing atmospheric processes, and the inevitable necessity
from sensitivity and Monte Carlo analyses provide useful
of limiting model algorithms to a resolvable set, one compo-
guidance on which inputs should be most carefully prescribed
nent of a model evaluation is to review the model’s science to
because they account for the greatest sensitivity in the model-
confirm that the construct is reasonable and defensible for the
ing output. These analyses also provide a view of what t
...