SIST-TS CEN/TS 18212-3:2026

SLOVENSKI STANDARD
01-julij-2026
Osebna identifikacija - Zahteve za biometrične izdelke - 3. del: Metodologija
ocenjevanja funkcionalnosti
Personal identification - Requirements for biometric products - Part 3: Functionality
evaluation methodology
Persönliche Identifikation - Anforderungen an biometrische Produkte - Teil 3: Methodik
zur Beurteilung der Funktionalität
Identification personnelle - Exigences relatives aux produits biométriques - Partie 3 :
Méthodologie d'évaluation de la fonctionnalité
Ta slovenski standard je istoveten z: CEN/TS 18212-3:2026
ICS:
35.240.15 Identifikacijske kartice. Čipne Identification cards. Chip
kartice. Biometrija cards. Biometrics
2003-01.Slovenski inštitut za standardizacijo. Razmnoževanje celote ali delov tega standarda ni dovoljeno.

CEN/TS 18212-3
TECHNICAL SPECIFICATION
SPÉCIFICATION TECHNIQUE
May 2026
TECHNISCHE SPEZIFIKATION
ICS 35.240.15
English Version
Personal identification - Requirements for biometric
products - Part 3: Functionality evaluation methodology
Identification personnelle - Exigences relatives aux Persönliche Identifikation - Anforderungen an
produits biométriques - Partie 3 : Méthodologie biometrische Produkte - Teil 3: Methodik zur
d'évaluation de la fonctionnalité Beurteilung der Funktionalität
This Technical Specification (CEN/TS) was approved by CEN on 13 April 2026 for provisional application.

The period of validity of this CEN/TS is limited initially to three years. After two years the members of CEN will be requested to
submit their comments, particularly on the question whether the CEN/TS can be converted into a European Standard.

CEN members are required to announce the existence of this CEN/TS in the same way as for an EN and to make the CEN/TS
available promptly at national level in an appropriate form. It is permissible to keep conflicting national standards in force (in
parallel to the CEN/TS) until the final decision about the possible conversion of the CEN/TS into an EN is reached.

CEN members are the national standards bodies of Austria, Belgium, Bulgaria, Croatia, Cyprus, Czech Republic, Denmark, Estonia,
Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway,
Poland, Portugal, Republic of North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and
United Kingdom.
EUROPEAN COMMITTEE FOR STANDARDIZATION
COMITÉ EUROPÉEN DE NORMALISATION

EUROPÄISCHES KOMITEE FÜR NORMUNG

CEN-CENELEC Management Centre: Rue de la Science 23, B-1040 Brussels
© 2026 CEN All rights of exploitation in any form and by any means reserved Ref. No. CEN/TS 18212-3:2026 E
worldwide for CEN national Members.

Contents Page
European foreword . 4
Introduction . 5
1 Scope . 7
2 Normative references . 7
3 Terms and definitions . 8
3.1 General terms . 8
3.2 Evaluation elements and parameters . 8
4 Symbols and abbreviated terms . 10
4.1 General. 10
4.2 Symbols related to the evaluation workflow . 11
4.2.1 Common symbols . 11
4.2.2 Phase 2 symbols . 11
4.2.3 Phase 3 symbols . 11
5 General concepts . 12
5.1 General. 12
5.2 Functional evaluation phases . 12
5.3 Compliance with ISO/IEC 19795 series . 13
5.4 Compliance with ISO/IEC 30107 series . 14
5.5 Terms and parameters used during the evaluation . 14
6 Test data . 15
6.1 General considerations . 15
6.2 Stored databases . 16
6.2.1 Recorded databases . 16
6.2.2 Use of synthetic databases . 16
6.3 Test crews in scenario evaluations . 17
7 Evaluation process for Phase 2 . 17
7.1 Overall view of the scenario evaluation . 17
7.2 TEST-level process . 18
7.3 SUBJECT-level process . 19
7.4 TRIAL-level process . 20
7.5 Families of tests in Phase 2. 22
7.6 Families of extended tests in Phase 2 . 22
8 Evaluation process for Phase 3 . 22
8.1 Overall view of the scenario evaluation . 22
8.2 TEST-level process . 23
8.3 SUBJECT-level process . 23
8.4 TRIAL-level process . 24
8.5 Families of tests in Phase 3. 26
9 Additional methodology when evaluating machine-learning-based (ML-based)
biometric products . 27
9.1 General requirements . 27
9.2 Continual improvement . 27
9.3 Continuous learning. 28
9.3.1 Introduction . 28
9.3.2 Evaluation time lapse and infrastructure . 28
9.3.3 Period between evaluations . 29
9.3.4 Evaluation procedure . 29
Bibliography . 30
European foreword
This document (CEN/TS 18212-3:2026) has been prepared by Technical Committee CEN/TC 224
“Personal identification and related personal devices with secure element, systems, operations and
privacy in a multi sectorial environment”, the secretariat of which is held by AFNOR.
Attention is drawn to the possibility that some of the elements of this document may be the subject of
patent rights. CEN shall not be held responsible for identifying any or all such patent rights.
Any feedback and questions on this document should be directed to the users’ national standards body.
A complete listing of these bodies can be found on the CEN website.
According to the CEN/CENELEC Internal Regulations, the national standards organisations of the
following countries are bound to announce this Technical Specification: Austria, Belgium, Bulgaria,
Croatia, Cyprus, Czech Republic, Denmark, Estonia, Finland, France, Germany, Greece, Hungary, Iceland,
Ireland, Italy, Latvia, Lithuania, Luxembourg, Malta, Netherlands, Norway, Poland, Portugal, Republic of
North Macedonia, Romania, Serbia, Slovakia, Slovenia, Spain, Sweden, Switzerland, Türkiye and the
United Kingdom.
Introduction
The use of remote services has increased significantly. This was boosted during 2020-2021, when many
service providers and administrations migrated most of their processes to online handling. Many online
services can now be found, such as opening of a bank account, claiming expenses, paying taxes, starting
legal actions, etc.
For all these services there is the need of identifying the persons claiming for that service, and doing it
in a comfortable, universal, reliable and auditable way. Even though some of those services, in some
countries, were deployed using public key infrastructures (PKIs), as recommended by eIDAS [9], this
approach was far away from being used by a significant part of the population.
Biometric recognition has been considered as a technology to solve the binding between the system and
the consumer. Adding biometric recognition to all kind of systems is a common practice nowadays.
In this context, service providers and administrations define their own requirements, select the
products and deploy the solution. On the other hand, manufacturers implement different solutions to
different customers, in order to fulfil each of those requirement sets. Both sides would benefit from
standards and regulations, on which to rely for the product definition.
Everybody benefits from having a common way of defining those requirements, and a detailed
evaluation methodology. These two items can be used by conformity assessment bodies or by business
owners, to create their own certification schemes for this kind of technology/products, by following
applicable standards.
NOTE ISO/IEC 17000 and related standards are examples of applicable conformity assessment standards.
This document is addressing this need for the case of biometric products, analysing and merging all
current works, and defining a detailed set of requirements, a biometric-mode-specific evaluation
methodology, and the passing criteria for different application profiles. This document has been
developed with consideration for GDPR [1] principles.
Application profiles (APs) are targeting the evaluation of a specific range of products using biometric
recognition. APs are the baseline for checking conformity with the CEN/TS 18212 series. Indeed, a
product manufacturer (PM), product vendor (PV) or sponsor can ask a conformity assessment body
(CAB) for the evaluation of a specific product to check its conformity according to the CEN/TS 18212
series and a specific AP at a certain level of assurance (basic, substantial or high).
The specifications given in this document are based on EN ISO/IEC 15408-1, ISO/IEC 19989-3 and
the ISO/IEC 17000 family of standards, including ISO/IEC 17007, EN ISO/IEC 17025
and EN ISO/IEC 17065. These standards specify processes dealing with evaluation and certification of
products and services, either related to their performance or to their security.
These objectives are reached by the development of a multipart Technical Specification (i.e. the
CEN/TS 18212 series) with the following structure:
— Parts 1-3: Defining the generic principles and methodologies, not requiring a biometric mode
specific approach.
In particular, these parts are:
— Part 1: General requirements and application profile definition;
— Part 2: Interoperability tests;
— Part 3: Functionality evaluation methodology.
— Parts 4-n: Planned future parts of the CEN/TS 18212 series, defining the particularities of each
biometric mode (e.g. specific tests, specific requirements) and containing a set of APs that establish
the test and requirements applicable for a specific application and context. Those APs will be
addressed in individual annexes, following the structure provided in CEN/TS 18212-1.
For example, these parts can be:
— Part 4: Fingerprint biometrics;
— Part 5: Face biometrics.
CEN/TS 18212-3 is focused on the definition of the biometric evaluation methodology, both for Phase 2
and Phase 3, in a biometric-mode-independent way.
1 Scope
The CEN/TS 18212 series specifies a generic framework for the establishment of requirements and
their evaluation methodology for biometric products. The requirements depend on the biometric mode
considered and are adapted to each scenario, through the definition of a variety of application profiles.
The CEN/TS 18212 series specifies the evaluation methodology, the individual tests and the application
profiles (with their particular requirements).
This document specifies:
— The different kind of evaluations to be performed.
— The terms used during the description of the tests to be applied.
— The parameters used, whose values are defined by each application profile, for each of the
individual tests.
— Test data used, and considerations dealing with personal data protection.
— How to perform technology evaluations.
— Execution flow for functionality scenario evaluations.
— Execution flow for attack resistance evaluations.
NOTE 1 Future parts of the CEN/TS series are planned to address the specifics of each biometric mode.
For each of these modalities, this document specifies application-independent tests, as well as a set of
application profiles, that detail the applicable tests, the evaluation parameters and the passing criteria.
The CEN/TS 18212 series can be taken by any certification body and/or sector, to define and evaluate
the requirements for their biometric products within their selected applications.
NOTE 2 National regulations and requirements can apply.
NOTE 3 Regarding biometrics for public sector applications, see also BSI TR-03121 [8] which can apply.
NOTE 4 For an overview of sectors addressed in the Cybersecurity Act, see Regulation (EU) 2019/881 [2].
2 Normative references
The following documents are referred to in the text in such a way that some or all of their content
constitutes requirements of this document. For dated references, only the edition cited applies. For
undated references, the latest edition of the referenced document (including any amendments) applies.
CEN/TS 18212-1, Personal identification — Requirements for biometric products — Part 1: General
requirements and application profile definition
ISO/IEC 2382-37, Information technology — Vocabulary — Part 37: Biometrics
ISO/IEC 19795 (all parts), Information technology — Biometric performance testing and reporting
ISO/IEC 30107 (all parts), Information technology — Biometric presentation attack detection
ISO/IEC 30108 (all parts), Biometrics — Identity attributes verification services
3 Terms and definitions
For the purposes of this document, the terms and definitions given in CEN/TS 18212-1, the
ISO/IEC 19795 series, ISO/IEC 30107 series, ISO/IEC 2382-37 and the following apply.
ISO and IEC maintain terminology databases for use in standardization at the following addresses:
— ISO Online browsing platform: available at https://www.iso.org/obp/
— IEC Electropedia: available at https://www.electropedia.org/
NOTE Certain terms, being common-use words, are used in capitals throughout the text to make it clear for
the reader that they are evaluation parameters, not regular terms.
3.1 General terms
3.1.1
biometric subsystem
set of modules that perform the biometric functions within the biometric product
Note 1 to entry: Examples of biometric functions are quality checking, pre-processing, comparison,
presentation attack detection methods.
3.2 Evaluation elements and parameters
3.2.1
ARTEFACT
artificial object or representation, that present a copy of the biometric characteristics of a SUBJECT
3.2.2
ATTACKER
person that attacks the system
EXAMPLE An imposter using an ARTEFACT for attempting a presentation attack
3.2.3
ATTEMPT
each of the individual interactions between the SUBJECT and the TOE within a TRIAL
3.2.4
ERROR
situation in which the TOE is not able to operate correctly, and therefore, is not able to accomplish a
result of the biometric comparison
EXAMPLE The TOE is not able to acquire a biometric sample from a bona-fide SUBJECT due to low quality
samples captured.
Note 1 to entry: In the case of a PAD TEST, an ERROR (once the maximum number of allowed ATTEMPTS has
been reached) can be considered as a NON-MATCH, as the ARTEFACT was not able to be properly captured.
3.2.5
FAIL
final result for TESTs which, within Phase 2, indicates that the TOE behaviour is not appropriate or,
within Phase 3, indicates that the attack has not been successful and, therefore, the TOE behaviour is
the correct one
3.2.6
MATCH
positive result of a biometric comparison during a TRIAL
EXAMPLE A bona-fide SUBJECT acceptance in a functional TEST.
Note 1 to entry: In the case of a PAD TEST, a MATCH is the non-desired result, as it indicates that the ARTEFACT
used was able to achieve a successful comparison.
3.2.7
NON-MATCH
negative result of a biometric comparison during a TRIAL
EXAMPLE A bona-fide SUBJECT rejection in a functional TEST.
3.2.8
OPERATOR
human being that, based on the TOE acquired data and result, takes the decision on whether the
transaction is valid or not
3.2.9
PASS
final result for TESTs which, within Phase 2, indicates that the TOE is presenting an appropriate
behaviour or, within Phase 3, indicates that the attack has been successful and, therefore, the TOE is
vulnerable
3.2.10
SERVER
computer-based equipment in which the TOE stores the acquired data during the biometric recognition
process
Note 1 to entry: Such data can be analysed later by an OPERATOR.
3.2.11
SETTING
execution context for a TRIAL within a TEST
Note 1 to entry: The SETTING can be the description of equipment to use, the way the SUBJECT has to interact
with the TOE, ambient conditions, ARTEFACTs to be used, etc. For each TEST, one or several SETTINGs have to be
specified.
3.2.12
SUBJECT
individual whose biometric data is intended to be enrolled or compared as part of the evaluation
Note 1 to entry: Traditionally, the SUBJECT is a USER, but in certain evaluations the SUBJECT would be a
combination of a USER and some additional property or element.
EXAMPLE 1 In the case of a videoconference system, where the TOE is being used with a USER and potentially
a variety of documents, the SUBJECT would be the combination of USER plus document.
EXAMPLE 2 In the case Phase 3 test, the SUBJECT is the combination of USER, ARTEFACT, and any other
relevant property.
3.2.13
TEST
action to evaluate the behaviour of the TOE for certain features
Note 1 to entry: One TEST is composed of several TRIALS, which involve several SUBJECTS and, probably,
several SETTINGs.
3.2.14
TEST_ERROR
situation in which, within a TEST, the TRIALs corresponding to a certain SUBJECT get over the limit
MAX_SUBJECT_ERRORS
3.2.15
TRIAL
each of the interactions between the SUBJECT and the TOE, during the TEST
Note 1 to entry: Depending on the TOE, each TRIAL may allow several ATTEMPTS.
EXAMPLE The TOE may ask the SUBJECT to repeat the biometric presentation due to acquisition errors. In
such a case, the new presentation would be considered as a new ATTEMPT within the same TRIAL.
3.2.16
USER
human being that takes part in a TRIAL
Note 1 to entry: Depending on the TEST, the USER could be a bona-fide SUBJECT or an ATTACKER, or it can
behave in one TRIAL as a bona-fide SUBJECT, and in another TRIAL as an ATTACKER.
4 Symbols and abbreviated terms
4.1 General
For the purposes of this document, the following symbols and abbreviations apply:
AP Application profile
CSA Cybersecurity Act [2]
eIDAS Electronic identification, authentication and trust services (see the eIDAS Regulation [9])
ETR Evaluation technical report
EU European Union / European
GDPR General Data Protection Regulation [1]
ID Identity
LoA Level of assurance
PAD Presentation attack detection (as described in ISO/IEC 30107-1)
TL Testing laboratory
TOE Target of evaluation
4.2 Symbols related to the evaluation workflow
4.2.1 Common symbols
The symbols listed here are constants to be used while explaining the evaluation process in both
Phase 2 (see Clause 7) and Phase 3 (see Clause 8). These constants shall be the limiting values for the
workflow of the evaluation.
— MAX_ATTEMPTS: Maximum number of ATTEMPTS allowed for a TRIAL, before resulting in an
ERROR for that TRIAL.
— MIN_SETTINGS: Minimum number of SETTINGS defined.
— MIN_SUBJECTS: Minimum number of SUBJECTS defined.
— MIN_TRIALS: Minimum number of TRIALS defined.
4.2.2 Phase 2 symbols
The symbols listed here are constants to be used while explaining the evaluation process only
in Phase 2 (see Clause 7). These constants shall be the limiting values for the workflow of the
evaluation.
— MAX_SETTING_NON_MATCHES: Maximum number of TRIALS, among all required for a SETTING
during a TEST, that provide a NON-MATCH result. When this number is reached, the TEST is
considered as FAIL for that SETTING. This is only applicable to Phase 2.
— MAX_SUBJECT_ERRORS: Maximum number of ERRORs allowed for the sum of all TRIALS for a
single SUBJECT, within a particular SETTING and TEST. This is only applicable to Phase 2.
— MAX_SUBJECTS_FAIL: Maximum number of SUBJECTs, for which TRIALS within a SETTING and
TEST have reached the limit of MAX_SUBJECT_NON_MATCHES. This is only applicable to Phase 2.
— MAX_SUBJECT_NON_MATCHES: Maximum number of TRIALS with a NON-MATCH result, allowed
for a single SUBJECT within one SETTING. This is only applicable to Phase 2.
— MAX_TEST_ERRORS: Maximum number of SUBJECTS, within a TEST, for which its TRIALS have
reached the limit given by MAX_SUBJECT_ERRORS. This is only applicable to Phases 2.
— MAX_TEST_NON_MATCHES: Maximum number of TRIALS, among all included in a TEST, with a
NON-MATCH result. If such number is reached, the TEST is considered as FAIL. This is only
applicable to Phase 2.
4.2.3 Phase 3 symbols
The symbols listed here are constants to be used while explaining the evaluation process only in
Phase 3 (Clause 8). These constants shall be the limiting values for the workflow of the evaluation. In a
Phase 3 TEST, a PASS result means that the TOE is vulnerable for that attack.
— MAX_SETTING_MATCHES: Maximum number of TRIALS, among all required for a SETTING during
a TEST, that provide a MATCH result. When this number is reached, the TEST is considered as a
PASS for that SETTING. This is only applicable to Phase 3.
— MAX_SUBJECT_MATCHES: Maximum number of TRIALS with a MATCH result, allowed for a single
SUBJECT within one SETTING. This is only applicable to Phase 3.
— MAX_SUBJECTS_PASS: Maximum number of SUBJECTs, for which TRIALS within a SETTING and
TEST have reached the limit of MAX_SUBJECT_MATCHES. This is only applicable to Phase 3.
— MAX_TEST_MATCHES: Maximum number of TRIALS, among all included in a TEST, with a MATCH
result. If such number is reached, the TEST is considered as a PASS. This is only applicable
to Phase 3.
5 General concepts
5.1 General
The evaluation of a biometric product is done through 3 phases, where Phase 1, detailed in
CEN/TS 18212-2, is focused on the interoperability aspects relevant to the TOE and the AP. But
Phases 2 and 3 are focused on evaluating the biometric functionality of the TOE, regarding performance
and suitability to the AP (Phase 2), and robustness against presentation attacks (Phase 3).
This document defines the basis for all the functional evaluation, i.e. the tasks to execute Phases 2 and 3.
This functional evaluation is based on the specifications provided by the ISO/IEC 19795 series and the
ISO/IEC 30107 series.
Future parts of the CEN/TS 18212 series are planned to specify the biometric mode-specific tests to be
executed, as well as a set of APs. Each of those APs determines the main characteristics of the TOE for
which the AP is applicable, as well as which are the applicable tests, and the acceptance criteria for each
of the tests, as well as for the overall functional evaluation.
In order to better understand the general testing methodology, Clause 5 revisits the evaluation phases
introduced in CEN/TS 18212-1, as well as the relationship with the ISO/IEC 19795 series and the
ISO/IEC 30107 series.
Test data shall be handled in accordance with Clause 6. The methodology for Phase 2 shall be according
to Clause 7, and the methodology for Phase 3 shall be according to Clause 8.
The additional methodology that shall be applied for those cases where the biometric subsystem of the
TOE has been developed using machine learning tools shall be according to Clause 9.
5.2 Functional evaluation phases
Within this conformity assessment methodology, the evaluation of the TOE shall be in accordance with
the phases defined in CEN/TS 18212-1. This document is focussed on the definition of Phases 2 and 3,
which are expected to be executed in a sequential manner:
— Phase 2: TOE performance evaluation
— The main target of these TESTs is to verify the TOE behaviour according to what has been
declared by the product supplier. This is to be checked using the relevant SETTINGs for the AP
selected.
— In addition, this phase also defines extended tests which
— The main target of these TESTs is to learn about the TOE, as to be able to locate the
operating boundaries in using the TOE with bona-fide SUBJECTs.
— This knowledge can help evaluators to discover strategies to attack the TOE during Phase 3
tests.
— Results obtained shall be checked with the TOE documentation, as to check if the FAILed
tests are clearly excluded from the TOE usage.
— Phase 3: Vulnerability assessment
— The main target of these tests is to determine if the TOE is vulnerable to presentation attacks,
either Type 1 or Type 2 attacks (as defined in ISO/IEC 30107-1 and CEN/TS 18099).
— According to the AP, the evaluated attacks can be impostor attacks, concealer attacks or both.
NOTE The EU Cybersecurity Act (EUCSA, Regulation 2019/881 [2]) defines 3 levels of assurance (LoA),
named as Basic, Substantial and High.
— Under an LoA of “High”, any Phase 3 ATTEMPT resulting in a PASS shall declare a FAIL for the
biometric product to achieve an LoA of “High”. This shall be determined by analysing that the
attack is not exceeding the maximum attack potential for the TOE evaluation.
5.3 Compliance with ISO/IEC 19795 series
Phase 2 evaluates the performance and suitability of the TOE for the AP defined. The evaluation shall be
according to the ISO/IEC 19795 series.
The ISO/IEC 19795 series, under the general title “Information technology — Biometric performance
testing and reporting”, contains the following four more relevant parts:
— Part 1: Principles and framework;
— Part 2: Testing methodologies for technology and scenario evaluation;
— Part 3: Modality-specific testing;
— Part 9: Testing on mobile devices.
ISO/IEC 19795-9 shall be considered when the TOE is a mobile device. Relevant clauses from
ISO/IEC 19795-3 shall be considered in addition to the planned biometric-mode-specific parts of the
CEN/TS 18212 series. The evaluation principles and the basic testing methodology are specified in
ISO/IEC 19795-1 and ISO/IEC 19795-2.
Within these principles, three kinds of evaluations are specified:
— Technology evaluations: Where testing is carried out on a standardized corpus, ideally collected by
a “universal” sensor. This kind of evaluation is thought to be applied directly to the biometric
algorithm, and using a previously collected database.
— Scenario evaluations: Where testing is carried out on a complete system in an environment that
models a real-world target application of interest. The evaluation is performed using real subjects
(i.e. not a database), where the context in which the TOE is expected to be used is simulated at
the TL.
— Operational evaluations: Where testing is carried out when the TOE is deployed in the real
application, and the evaluation is being performed under its current operation.
Within the ISO/IEC 19795 series, operational evaluations are not considered. Most of the tests defined
are scenario-based tests, but some others use databases, approaching the concept of a technology
evaluation.
5.4 Compliance with ISO/IEC 30107 series
Phase 3 is focused on evaluating the robustness of the TOE under those relevant attacks. Most of those
attacks are presentation attacks, as defined in ISO/IEC 30107-1. For the evaluation of the capability of
presentation attack detection (PAD), ISO/IEC 30107-3 specifies the general methodology in a biometric
mode agnostic manner, specifying the basis for a more detailed and applicable methodology.
Therefore, PAD tests in Phase 3 shall use ISO/IEC 30107-3 as the initial specification of the evaluation.
Also, when reporting the results, ISO/IEC 30107-3 shall be followed. ISO/IEC 30107-3 define two main
philosophies for carrying out PAD evaluation. When the relevant AP requires a LoA “High” or
“Substantial”, the Common Criteria approach shall be used, which is detailed in ISO/IEC 30107-3.
5.5 Terms and parameters used during the evaluation
Most of biometric TESTs follow a very similar execution sequence, which is described in Clauses 7
and 8. Such clauses are written in a generic way, so as to allow an easier description of each of the
TESTs. Future parts of the CEN/TS 18212 series are planned to specify each of the specific TESTs based
on that sequence.
For a better understanding of this methodology, the following terms are needed (defined in Clause 3):
— ARTEFACT
— ATTACKER
— ATTEMPT
— ERROR
— FAIL
— MATCH
— NON-MATCH
— OPERATOR
— PASS
— SERVER
— SETTING
— SUBJECT
— TEST
— TEST_ERROR
— TRIAL
— USER
It is also important to consider the following parameters for use in this evaluation methodology
(defined in Clause 4):
— MAX_ATTEMPTS
— MAX_SETTING_MATCHES
— MAX_SETTING_NON_MATCHES
— MAX_SUBJECT_ERRORS
— MAX_SUBJECTS_FAIL
— MAX_SUBJECT_MATCHES
— MAX_SUBJECT_NON_MATCHES
— MAX_SUBJECTS_PASS
— MAX_TEST_ERRORS
— MAX_TEST_MATCHES
— MAX_TEST_NON_MATCHES
— MIN_SETTINGS
— MIN_SUBJECTS
— MIN_TRIALS
MIN_TRIALS, MIN_SETTINGS and MIN_SUBJECTS define the minimum number specified for each TEST.
These are the numbers to be used by the TL. If during an evaluation the TL detects too many ERRORs
during the TRIALS, the TL may increase those numbers until it can obtain a number of conclusive
(i.e. NON-ERROR) results to satisfy the following formula:
Minimum conclusive results = MIN_TRIALS × MIN_SETTINGS × MIN_SUBJECTS (1)
This deviation shall be fully justified and included in the ETR.
6 Test data
6.1 General considerations
Data are needed for performing biometric evaluations.
In the case of technology evaluations, when the biometric capture subsystem can be detached from
the TOE, previously recorded databases can be used to speed up the evaluation, increasing also the
significance of the results obtained.
In those cases where the biometric capture subsystem cannot be detached from the rest of the TOE, test
data can only be obtained by calling test crews, i.e. performing a scenario evaluation.
NOTE The GDPR [1] contains requirements regarding privacy protection.
6.2 Stored databases
6.2.1 Recorded databases
Most of the tests to be defined under this evaluation methodology are going to be scenario-based tests,
which means using real users as input to the TOE. But there are some tests that can be considered as
technology evaluations and, therefore, use databases.
Technology evaluations which use databases shall be according to ISO/IEC 19795-1 and
ISO/IEC 19795-2. The databases shall be representative of the target population where the TOE is going
to be applied, and be varied enough as to be able to cover most of the diversity of such population.
Databases can be previously recorded and used in several evaluation of TOEs, as long as the
representativeness of the database is guaranteed.
The records within a database shall be anonymized whenever possible.
NOTE The GDPR [1] contains requirements regarding privacy protection.
The above-mentioned technology tests are typical from either interoperability testing
(see CEN/TS 18212-2), or for some of the tests in Phase 2 (see Clause 7).
6.2.2 Use of synthetic databases
Due to the difficulty of creating large databases, the use of synthetic databases can be considered and
can remove the challenge of achieving a large number of biometric samples.
NOTE The GDPR [1] contains requirements regarding privacy protection.
But in order to use such synthetic database, it shall be proven that:
— The database is representative of the target population indicated by the AP. In order to reach this
objective, the distribution of the database shall be representative in terms of gender, age, ethnicity,
and/or any other relevant parameter important for the target population.
— The database samples shall be realistic enough, so that the behaviour of state-of-the-art algorithms
may be considered equivalent with the performance achieved using real sample databases.
In order to achieve this second requirement, synthetic biometric samples shall be extensive in
parameters which influence the quality of the biometric characteristic (e.g. for fingerprints scars, dry
finger, partial fingerprints), the technical image quality (e.g. noise, sharpness) and combination of both.
This is needed as to allow the algorithms to achieve, not only an equivalent value of FNMR@FMR, but
also an equivalent FTA rate.
EXAMPLE 1 In the case of fingerprint mode, there is a well-known application called SFinGe [3], which can
create completely clean fingerprints. SFinGe Version 2 added image distortion methods, to keep into account skin
plasticity. Version 2.5 added the generation of realistic backgrounds and different fingerprint sizes. Version 3.0
added improved noising algorithms and parameters. Version 5 added a parameter to control the probability of
generating very-low quality fingerprints.
In order to determine that the synthetic database can be used for the evaluations defined by this
document, 5 state-of-the-art biometric algorithms shall be used. The decision on the criteria to choose
which 5 algorithms to use shall be given by the relevant AP.
The testing set shall be dimensioned and composed in such a manner that it can provide a statistical
relevant representation to the performance on the societal clusters composed on age, gender, ethnicity,
labour status, and/or any further criteria relevant to the AP.

This information is given for the convenience of users of this document and does not constitute an endorsement
by CEN of the product named.
Each of those algorithms shall be executed against a real data set and the synthetic data set. The real
data set shall be also equivalent to the synthetic database, i.e. representative of the target population.
NOTE Depending on the relevant biometric mode, there could be repositories (or listings) of state-of-the-art
algorithms. The repository of algorithms to be used for this task are planned to be addressed in a future part of the
CEN/TS 18212 series.
For the AP requested FMR, both FNMR and FTA rate shall provide numbers within half of the same
order of magnitude between the execution with the real data set and the synthetic data set.
EXAMPLE 2 If, for the requested FMR, the FNMR is of 1 % on the real data set, then it is accepted to have FNMR
for the synthetic data set of between 0,5 % and 5 %.
If the results show that for at least 4 out of the 5 algorithms, both error rates are within the same order
of magnitude, the synthetic database shall be considered valid for being used in the evaluation defined
in the applicable part of the CEN/TS 18212 series.
6.3 Test crews in scenario evaluations
When a scenario-based test is required, then databases are not used, but real users. The use of human
beings as test crew members at the moment of the evaluation, drives important challenges to the TL,
especially if that same test subject is expected to participate in the evaluation several times.
The AP may consider limiting the test crew size, as to allow a higher viability of the evaluation, in
particular when the evaluation faces some time and/or cost limitations.
Some tests might require particular features for the test crew members, such as diversity in the
biometric characteristics, similarity among them, possibility of acting in different manners when
interacting with the TOE, etc. Tests should be careful in following all indications given by the TL
members.
In Phase 2, test crew members shall not be involved in the evaluation of the TOE so as to keep a
behaviour not biased by an excessive knowledge of the TOE. In Phase 4, this is also recommended.
At all moments, the privacy of each of the test crew members shall be respected. When a sample from a
test crew member is needed to explain the results in the ETR, that sample shall be anonymized as much
as possible by, for example, segmenting all non-significant information within the sample.
NOTE The GDPR [1] contains requirements regarding privacy protection.
7 Evaluation process for Phase 2
7.1 Overall view of the scenario evaluation
At Phase 2, several scenario evaluations are executed. For this methodology, each of these evaluations is
called a TEST. Each TEST shall consider a number of SETTINGS and a set of SUBJECTS (i.e. a test crew).
For each combination of SETTINGS and SUBJECTS, a number of TRIALS are performed, being possible
that each TRIAL allows a maximum number of ATTEMPTS.
The following figure represents the hierarchical relationship among these elements.
Figure 1 — Hierarchical relationship among evaluation elements
This is a hierarchical relationship that is used during the whole description of this evaluation
methodology.
But, depending on the evaluation, it could be interesting to exchange the order among TESTS, SETTINGS
and SUBJECTS. For example, the TL might consider more practical to execute all TESTs relevant to the
same SETTING to all SUBJECTS, before changing the SETTING. Or it could be more practical to execute
all TEST with all SETTINGS for each of the SUBJECTS. This decision is up to the TL. If the relationship
given in Figure 1 is modified in any manner, this shall be justified and detailed in the ETR. Figure 2
shows some alternatives.
Figure 2 — Alternatives to the relationship among evaluation elements
In the description of each of the processes in the following subclauses, it shall be noted that:
— s is the variable that holds the SUBJECT being used
— k is the variable that holds the SETTING being used.
7.2 TEST-level process
Each TEST is composed by the execution of a series of SETTINGS, up to reaching the limit given by
MIN_SETTINGS. The TL may increment this number if the number of conclusive results is below the one
demanded by Formula (1).
Once the execution of all SETTINGS, for all SUBJECTS and TRIALS, is finished, the final results are
analysed to determine if the TEST is a PASS or FAIL, according to the criteria provided by the
relevant AP.
The flowchart for the TEST-level process is given in Figure 3, which includ
...