ISO/TR 27877:2021

TECHNICAL ISO/TR
REPORT 27877
First edition
2021-10
Statistical analysis for evaluating the
precision of binary measurement
methods and their results
Analyse statistique pour l'évaluation de la fidélité des méthodes de
mesure binaire et de leurs résultats
Reference number
© ISO 2021
All rights reserved. Unless otherwise specified, or required in the context of its implementation, no part of this publication may
be reproduced or utilized otherwise in any form or by any means, electronic or mechanical, including photocopying, or posting on
the internet or an intranet, without prior written permission. Permission can be requested from either ISO at the address below
or ISO’s member body in the country of the requester.
ISO copyright office
CP 401 • Ch. de Blandonnet 8
CH-1214 Vernier, Geneva
Phone: +41 22 749 01 11
Email: copyright@iso.org
Website: www.iso.org
Published in Switzerland
ii
Contents Page
Foreword .v
Introduction . vi
1 Scope . 1
2 Normative references . 1
3 Terms and definitions, and symbols . .1
3.1 Terms and definitions . 1
3.2 Symbols . 3
4 Overview . 4
5 Examples used in this document .6
5.1 Case 1: Listeria monocytogenes . 6
5.2 Case 2: Human cell line activation test (h-CLAT)-1 . 6
5.3 Case 3: Intraracheal administration testing . 7
5.4 Case 4: Histopathological classification of lung carcinoma . 8
5.5 Case 5: Human cell line activation test (h-CLAT)-2 . 8
5.6 Case 6: Statistical model for predicting chemical toxicity . 8
6 Statistical analysis for evaluating the precision of binary measurement methods
and their results .9
6.1 ISO 5725-based method . 9
6.1.1 Overview . 9
6.1.2 Case 1 .12
6.1.3 Case 2(a) .12
6.1.4 Case 2(b) .13
6.1.5 Case 3(a) . 13
6.1.6 Case 3(b) .13
6.2 Accordance and concordance . 13
6.2.1 Overview . 13
6.2.2 Case 1 . 15
6.2.3 Case 2(a) . 15
6.2.4 Case 2(b) . 16
6.2.5 Case 3(a) . 16
6.2.6 Case 3(b) . 16
6.3 ORDANOVA . . 16
6.3.1 Overview . 16
6.3.2 Case 1 . 18
6.3.3 Case 2(a) . 18
6.3.4 Case 2(b) . 19
6.3.5 Case 3(a) . 19
6.3.6 Case 3(b) . 19
6.4 CM-accuracy, sensitivity and specificity . 19
6.4.1 Overview . 19
6.4.2 Case 4 . 20
6.4.3 Case 5 . 20
6.4.4 Case 6 . 20
6.5 Kappa coefficient . 21
6.5.1 Overview . 21
6.5.2 Case 4 . 21
6.5.3 Case 5 . 21
6.5.4 Case 6 . 22
7 Remarks on the methods introduced in this document .22
7.1 Comparison between the mathematical expressions of the precision estimates .22
7.2 Comparison between the numerical examples of the precision estimates .23
iii
Bibliography .25
iv
Foreword
ISO (the International Organization for Standardization) is a worldwide federation of national standards
bodies (ISO member bodies). The work of preparing International Standards is normally carried out
through ISO technical committees. Each member body interested in a subject for which a technical
committee has been established has the right to be represented on that committee. International
organizations, governmental and non-governmental, in liaison with ISO, also take part in the work.
ISO collaborates closely with the International Electrotechnical Commission (IEC) on all matters of
electrotechnical standardization.
The procedures used to develop this document and those intended for its further maintenance are
described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria needed for the
different types of ISO documents should be noted. This document was drafted in accordance with the
editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives).
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. ISO shall not be held responsible for identifying any or all such patent rights. Details of
any patent rights identified during the development of the document will be in the Introduction and/or
on the ISO list of patent declarations received (see www.iso.org/patents).
Any trade name used in this document is information given for the convenience of users and does not
constitute an endorsement.
For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and
expressions related to conformity assessment, as well as information about ISO's adherence to
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT), see
www.iso.org/iso/foreword.html.
This document was prepared by Technical Committee ISO/TC 69, Application of statistical methods,
Subcommittee SC 6, Measurement methods and results.
Any feedback or questions on this document should be directed to the user’s national standards body. A
complete listing of these bodies can be found at www.iso.org/members.html.
v
Introduction
The documents in the ISO 5725 series define the precision of quantitative measurement methods
and their results, and assume that the errors follow normal distributions in their basic models. Also,
they provide how to run experiments to evaluate precision measures, such as repeatability and
reproducibility. Nowadays, there is also a demand for dealing with qualitative measurement methods
and their results, which output binary data, categorical data, etc. However, the ISO 5725 series is not
suitable mathematically for analyzing such data.
Several existing studies propose statistical methods for dealing with binary and/or categorical data,
but no guidance documents are available so far. Hence, this document summaries various methods to
evaluate the precision of binary measurement methods and their results, which are the most essential
and frequently used methods for qualitative data.
vi
TECHNICAL REPORT ISO/TR 27877:2021(E)
Statistical analysis for evaluating the precision of binary
measurement methods and their results
1 Scope
This document introduces five statistical methods for evaluating the precision of binary measurement
methods and their results. The five methods can be divided into two types. Both types are based on
measured values provided by each laboratory participa
...