ASTM D6589-05
(Guide)Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
Standard Guide for Statistical Evaluation of Atmospheric Dispersion Model Performance
SIGNIFICANCE AND USE
Guidance is provided on designing model evaluation performance procedures and on the difficulties that arise in statistical evaluation of model performance caused by the stochastic nature of dispersion in the atmosphere. It is recognized there are examples in the literature where, knowingly or unknowingly, models were evaluated on their ability to describe something which they were never intended to characterize. This guide is attempting to heighten awareness, and thereby, to reduce the number of “unknowing” comparisons. A goal of this guide is to stimulate development and testing of evaluation procedures that accommodate the effects of natural variability. A technique is illustrated to provide information from which subsequent evaluation and standardization can be derived.
SCOPE
1.1 This guide provides techniques that are useful for the comparison of modeled air concentrations with observed field data. Such comparisons provide a means for assessing a model's performance, for example, bias and precision or uncertainty, relative to other candidate models. Methodologies for such comparisons are yet evolving; hence, modifications will occur in the statistical tests and procedures and data analysis as work progresses in this area. Until the interested parties agree upon standard testing protocols, differences in approach will occur. This guide describes a framework, or philosophical context, within which one determines whether a model's performance is significantly different from other candidate models. It is suggested that the first step should be to determine which model's estimates are closest on average to the observations, and the second step would then test whether the differences seen in the performance of the other models are significantly different from the model chosen in the first step. An example procedure is provided in to illustrate an existing approach for a particular evaluation goal. This example is not intended to inhibit alternative approaches or techniques that will produce equivalent or superior results. As discussed in Section , statistical evaluation of model performance is viewed as part of a larger process that collectively is referred to as model evaluation.
1.2 This guide has been designed with flexibility to allow expansion to address various characterizations of atmospheric dispersion, which might involve dose or concentration fluctuations, to allow development of application-specific evaluation schemes, and to allow use of various statistical comparison metrics. No assumptions are made regarding the manner in which the models characterize the dispersion.
1.3 The focus of this guide is on end results, that is, the accuracy of model predictions and the discernment of whether differences seen between models are significant, rather than operational details such as the ease of model implementation or the time required for model calculations to be performed.
1.4 This guide offers an organized collection of information or a series of options and does not recommend a specific course of action. This guide cannot replace education or experience and should be used in conjunction with professional judgment. Not all aspects of this guide may be applicable in all circumstances. This guide is not intended to represent or replace the standard of care by which the adequacy of a given professional service must be judged, nor should it be applied without consideration of a project's many unique aspects. The word "Standard" in the title of this guide means only that the document has been approved through the ASTM consensus process.
This standard does not purport to address all of the safety concerns, if any, associated with its use. It is the responsibility of the user of this standard to establish appropriate safety and health practices and to determine the applicability of regulatory limitations prior to use.
General Information
Relations
Standards Content (Sample)
NOTICE: This standard has either been superseded and replaced by a new version or withdrawn.
Contact ASTM International (www.astm.org) for the latest information
Designation: D6589 – 05
Standard Guide for
Statistical Evaluation of Atmospheric Dispersion Model
1
Performance
This standard is issued under the fixed designation D6589; the number immediately following the designation indicates the year of
original adoption or, in the case of revision, the year of last revision. A number in parentheses indicates the year of last reapproval. A
superscript epsilon (´) indicates an editorial change since the last revision or reapproval.
1. Scope 1.4 This guide offers an organized collection of information
oraseriesofoptionsanddoesnotrecommendaspecificcourse
1.1 This guide provides techniques that are useful for the
of action. This guide cannot replace education or experience
comparison of modeled air concentrations with observed field
and should be used in conjunction with professional judgment.
data. Such comparisons provide a means for assessing a
Not all aspects of this guide may be applicable in all circum-
model’s performance, for example, bias and precision or
stances. This guide is not intended to represent or replace the
uncertainty, relative to other candidate models. Methodologies
standard of care by which the adequacy of a given professional
for such comparisons are yet evolving; hence, modifications
service must be judged, nor should it be applied without
will occur in the statistical tests and procedures and data
consideration of a project’s many unique aspects. The word
analysis as work progresses in this area. Until the interested
“Standard” in the title of this guide means only that the
parties agree upon standard testing protocols, differences in
document has been approved through the ASTM consensus
approach will occur. This guide describes a framework, or
process.
philosophical context, within which one determines whether a
1.5 This standard does not purport to address all of the
model’s performance is significantly different from other
safety concerns, if any, associated with its use. It is the
candidate models. It is suggested that the first step should be to
responsibility of the user of this standard to establish appro-
determine which model’s estimates are closest on average to
priate safety and health practices and to determine the
the observations, and the second step would then test whether
applicability of regulatory limitations prior to use.
the differences seen in the performance of the other models are
significantly different from the model chosen in the first step.
2. Referenced Documents
An example procedure is provided inAppendix X1 to illustrate
2
2.1 ASTM Standards:
an existing approach for a particular evaluation goal. This
D1356 Terminology Relating to Sampling and Analysis of
example is not intended to inhibit alternative approaches or
Atmospheres
techniques that will produce equivalent or superior results. As
discussed in Section 6, statistical evaluation of model perfor-
3. Terminology
mance is viewed as part of a larger process that collectively is
3.1 Definitions—For definitions of terms used in this guide,
referred to as model evaluation.
refer to Terminology D1356.
1.2 This guide has been designed with flexibility to allow
3.2 Definitions of Terms Specific to This Standard:
expansion to address various characterizations of atmospheric
3.2.1 atmospheric dispersion model, n—an idealization of
dispersion, which might involve dose or concentration fluctua-
atmospheric physics and processes to calculate the magnitude
tions, to allow development of application-specific evaluation
and location of pollutant concentrations based on fate, trans-
schemes, and to allow use of various statistical comparison
port, and dispersion in the atmosphere. This may take the form
metrics. No assumptions are made regarding the manner in
of an equation, algorithm, or series of equations/algorithms
which the models characterize the dispersion.
used to calculate average or time-varying concentration. The
1.3 The focus of this guide is on end results, that is, the
model may involve numerical methods for solution.
accuracy of model predictions and the discernment of whether
3.2.2 dispersion, absolute, n—the characterization of the
differences seen between models are significant, rather than
spreading of material released into the atmosphere based on a
operationaldetailssuchastheeaseofmodelimplementationor
coordinate system fixed in space.
the time required for model calculations to be performed.
3.2.3 dispersion, relative, n—the characterization of the
spreading of material released into the atmosphere based on a
1
This guide is under the jurisdiction of ASTM Committee D22 on Air Quality
2
and is the direct responsibility of Subcommittee D22.11 on Meteorology. For referenced ASTM standards, visit the ASTM website, www.astm.org, or
Current edition approved October 1, 2005. Published October 2005. Originally contact AST
...
Questions, Comments and Discussion
Ask us and Technical Secretary will try to provide an answer. You can facilitate discussion about the standard in here.