ISO/IEC 25022:2016

DRAFT INTERNATIONAL STANDARD
ISO/IEC DIS 25022.2
ISO/IEC JTC 1/SC 7 Secretariat: SCC
Voting begins on: Voting terminates on:
2015-08-03 2015-10-03
Systems and software engineering — Systems and
software quality requirements and evaluation (SQuaRE) —
Measurement of quality in use
Titre manque
ICS: 35.080
THIS DOCUMENT IS A DRAFT CIRCULATED
FOR COMMENT AND APPROVAL. IT IS
THEREFORE SUBJECT TO CHANGE AND MAY
NOT BE REFERRED TO AS AN INTERNATIONAL
STANDARD UNTIL PUBLISHED AS SUCH.
IN ADDITION TO THEIR EVALUATION AS
BEING ACCEPTABLE FOR INDUSTRIAL,
TECHNOLOGICAL, COMMERCIAL AND
USER PURPOSES, DRAFT INTERNATIONAL
STANDARDS MAY ON OCCASION HAVE TO
BE CONSIDERED IN THE LIGHT OF THEIR
POTENTIAL TO BECOME STANDARDS TO
WHICH REFERENCE MAY BE MADE IN
Reference number
NATIONAL REGULATIONS.
ISO/IEC DIS 25022.2:2015(E)
RECIPIENTS OF THIS DRAFT ARE INVITED
TO SUBMIT, WITH THEIR COMMENTS,
NOTIFICATION OF ANY RELEVANT PATENT
RIGHTS OF WHICH THEY ARE AWARE AND TO
©
PROVIDE SUPPORTING DOCUMENTATION. ISO/IEC 2015

---------------------- Page: 1 ----------------------
ISO/IEC DIS 25022.2:2015(E)

COPYRIGHT PROTECTED DOCUMENT
© ISO/IEC 2015, Published in Switzerland
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized otherwise in any form
or by any means, electronic or mechanical, including photocopying, or posting on the internet or an intranet, without prior
written permission. Permission can be requested from either ISO at the address below or ISO’s member body in the country of
the requester.
ISO copyright office
Ch. de Blandonnet 8 • CP 401
CH-1214 Vernier, Geneva, Switzerland
Tel. +41 22 749 01 11
Fax +41 22 749 09 47
copyright@iso.org
www.iso.org
ii © ISO/IEC 2015 – All rights reserved

---------------------- Page: 2 ----------------------
Contents

1 Scope . 8
2 Conformance . 9
3 Normative references . 9
4 Terms and definitions . 9
5 Abbreviated terms .13
6 Use of quality in use measures .13
6.1 Applications of quality in use .13
6.2 Measurement of quality in use .13
6.3 Normalising measures .14
6.4 Selecting appropriate quality in use measures .14
6.5 Aspects of quality in use described in other International Standards .15
7 Format used for documenting the quality measures.15
8 Quality in use measures .16
8.1 General .16
8.2 Effectiveness measures.16
8.3 Efficiency measures .17
8.4 Satisfaction measures .18
8.4.1 General .18
8.4.2 Usefulness measures .19
8.4.3 Trust measures .20
8.4.4 Pleasure measures .20
8.4.5 Comfort measures .20
8.5 Freedom from risk measures .20
8.5.1 General .20
8.5.2 Economic risk mitigation measures .21
8.5.3 Health and safety risk mitigation measures .22
8.5.4 Environmental risk mitigation measures .23
8.6 Context coverage measures .23
8.6.1 General .23
8.6.2 Context completeness measures .23
8.6.3 Flexibility measures .24
Annex A (Informative) Examples of how to measure context coverage .25
Annex B (Informative) Normalisation of quality in use measures .27
Annex C (Informative) Use of ISO/IEC 25022 for measuring usability in ISO 9241-11 .31
Annex D (Informative) Quality in use evaluation process .32
D.1 Establish evaluation requirements .32
D.2 Specify the evaluation .32
D.3 Design the evaluation .35
D.4 Execute the evaluation .35
Annex E (Informative) Relationship between different quality models .36
Annex F (Informative) Quality measurement concepts .37
Annex G (Informative) QMEs used to define quality measures .38
Bibliography .39
© ISO/IEC 2015 – All rights reserved
4

---------------------- Page: 3 ----------------------
ISO/IEC DIS 25022.2(E)

Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work. In the field of information technology, ISO and IEC have
established a joint technical committee, ISO/IEC JTC 1.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 2.
The main task of the joint technical committee is to prepare International Standards. Draft International Standards
adopted by the joint technical committee are circulated to national bodies for voting. Publication as an International
Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this document may be the subject of patent rights.
ISO and IEC shall not be held responsible for identifying any or all such patent rights.
ISO/IEC 25022, which replaces ISO/IEC 9126-4, is a part of the SQuaRE series of standards and was prepared by
Joint Technical Committee ISO/IEC JTC 1, information technology, Subcommittee SC 7, Software and Systems
Engineering
The SQuaRE series of standards consists of the following divisions under the general title Systems and software
Quality Requirements and Evaluation:
 ISO/IEC 2500n - Quality Management Division,
 ISO/IEC 2501n - Quality Model Division,
 ISO/IEC 2502n - Quality Measurement Division,
 ISO/IEC 2503n - Quality Requirements Division,
 ISO/IEC 2504n - Quality Evaluation Division,
 ISO/IEC 25050 - 25099 SQuaRE Extension Division.
Annexes A, B, C, D, E, F and G are for information only.
5
© ISO/IEC 2015 – All rights reserved 5

---------------------- Page: 4 ----------------------
Introduction
This International Standard is a part of the SQuaRE series of international standards. It provides a set of measures
for the characteristics of quality in use (defined in ISO/IEC 25010) that can be used for specifying quality in use
requirements (in conjunction with ISO/IEC 25030) and measuring and evaluating quality in use (in conjunction with
ISO/IEC 25040 and ISO/IEC 25041).
The quality measures included in this International Standard were selected based on their practical value. They are
based on established practice (including for example [4] in the Bibliography), They are not intended to be
exhaustive, and users of this International Standard are encouraged to refine them if necessary.
This International Standard replaces ISO/IEC 9126-4, and has the following changes:
 Measures are given for the revised quality model for quality in use in ISO/IEC 25010.
 Measures are categorised as generally applicable, could be used in a wide range of situations or
specialised for specific needs.
 Annexes that were common to ISO/IEC 9126-2, -3 and -4 have been removed (and might be included in a
future revision of ISO/IEC 25020).
Quality Measurement Division
This International Standard is a part of ISO/IEC 2502n Quality Measurement Division of SQuaRE series that
currently consists of the following International Standards:
 ISO/IEC 25020 – Measurement reference model and guide: provides a reference model and guide for
measuring the quality characteristics defined in ISO/IEC 2501n Quality Model Division.
 ISO/IEC 25021 – Quality measure elements: provides a format for specifying Quality Measure Elements
and some examples of QMEs that can be used to construct software quality measures.
 ISO/IEC 25022 – Measurement of quality in use: provides measures, including associated measurement
functions for the quality characteristics in the quality in use model.
 ISO/IEC 25023 – Measurement of system and software product quality: provides measures, including
associated measurement functions and QMEs for the quality characteristics in the product quality model.
 ISO/IEC 25024 – Measurement of data quality: provides measures, including associated measurement
functions and QMEs for the quality characteristics in the data quality model.
Figure 1 depicts the relationship between this International Standard and the other standards in ISO/IEC 2502n
division.
© ISO/IEC 2015 – All rights reserved
6

---------------------- Page: 5 ----------------------
ISO/IEC DIS 25022.2(E)


Figure 1 - Structure of the Quality Measurement division

Outline and Organization of SQuaRE Series
The SQuaRE series consists of five main divisions and extension division. Outline of each divisions within SQuaRE
series are as follows.
 ISO/IEC 2500n - Quality Management Division. The standards that form this division define all common
models, terms and definitions referred further by all other standards from SQuaRE series. The division also
provides requirements and guidance for the planning and management of a project.
 ISO/IEC 2501n - Quality Model Division. The standards that forms this division provides quality models for
system/software products, quality in use and data. A service quality model is under development. Practical
guidance on the use of the quality model is also provided.
 ISO/IEC 2502n - Quality Measurement Division. The standards that form this division include a
system/software product quality measurement reference model, definitions of quality measures, and
practical guidance for their application. This division presents internal measures of software quality,
external measures of software quality and quality in use measures. Quality measure elements forming
foundations for the quality measures are defined and presented.
 ISO/IEC 2503n - Quality Requirements Division. The standard that forms this division helps specifying
quality requirements. These quality requirements can be used in the process of quality requirements
elicitation for a system/software product to be developed, designing a process for achieving necessary
quality, or as inputs for an evaluation process.
 ISO/IEC 2504n - Quality Evaluation Division. The standards that form this division provide requirements,
recommendations and guidelines for system/software product evaluation, whether performed by
independent evaluators, acquirers or developers. The support for documenting a measure as an Evaluation
Module is also presented.
ISO/IEC 25050 to ISO/IEC 25099 are reserved for SQuaRE extension International Standards, which currently
include 25051 Requirements for quality of Ready to Use Software Products (RUSP) and instructions for testing,
and the ISO/IEC 25060 – 25069 Common industry format for usability series of standards.
7
© ISO/IEC 2015 – All rights reserved 7

---------------------- Page: 6 ----------------------
Systems and software engineering – Systems and software Quality
Requirements and Evaluation (SQuaRE) – Measurement of quality in use
1 Scope
This International Standard defines quality in use measures for the characteristics defined in ISO/IEC 25010, and is
intended to be used together with ISO/IEC 25010. It can be used in conjunction with the ISO/IEC 2503n – Quality
requirements division and the ISO/IEC 2504n – Quality evaluation division standards or to more generally meet
user needs with regard to product or system quality.
This International Standard contains:
 a basic set of measures for each quality in use characteristic;
 an explanation of how quality in use is measured.
This International Standard provides a suggested set of quality in use measures to be used with the quality in use
model in ISO/IEC 25010. They are not intended to be an exhaustive set.
It includes as informative annexes examples of how to measure context coverage (Annex A), options for
normalising quality in use measures (Annex B), Use of ISO/IEC 25022 for measuring usability in ISO 9241-11,
(Annex C), a quality in use evaluation process (Annex D), the relationship between different quality models (Annex
E) and quality measurement concepts (Annex F).
The measures are applicable to the use of any human-computer system, including both computer systems in use
and software products that form part of the system.
This International Standard does not assign ranges of values of the measures to rated levels or to grades of
compliance, because these values are defined for each system, product depending on the context of use and
users' needs.
Some attributes could have a desirable range of values, which does not depend on specific user needs but
depends on generic factors; for example, human cognitive factors.
The proposed quality in use measures are primarily intended to be used for quality assurance and management of
systems and software products based on their effects when actually used. The main users of the measurement
results are people managing development, acquisition, evaluation or maintenance of software and systems.
The main users of this International Standard are people carrying out specification and evaluation activities as part
of:
 Development: including requirements analysis, design, and testing through acceptance during the life cycle
process.
 Quality management: systematic examination of the product or computer system, for example when
evaluating quality in use as part of quality assurance and quality control.
 Supply: a contract with the acquirer for the supply of a system, software product or software service under
the terms of a contract, for example when validating quality at qualification test.
 Acquisition: including product selection and acceptance testing, when acquiring or procuring a system,
software product or software service from a supplier.
 Maintenance: improvement of the product based on quality in use measures.
© ISO/IEC 2015 – All rights reserved
8

---------------------- Page: 7 ----------------------
ISO/IEC DIS 25022.2(E)

2 Conformance
Any quality requirement specification or quality evaluation that conforms to this International Standard shall:
a) Select the quality in use characteristics and/or subcharacteristics to be specified or evaluated as defined in
ISO/IEC 25010.
b) For each selected characteristic or subcharacteristic, select all the General (G) quality measures defined in
clause 8 or provide a rationale for any that are excluded.
c) Optionally select any other quality measures that are relevant.
d) If any quality measure is modified, provide the rationale for any changes.
e) Define precisely how each quality measure is operationalised (for example details of the measurement
method or questionnaire used).
f) Define any additional quality measures used that are not included in this International Standard.
3 Normative references
The following documents, in whole or in part, are normatively referenced in this document and are indispensable
for its application. For dated references, only the edition cited applies. For undated references, the latest edition of
the referenced document (including any amendments) applies.
ISO/IEC 25010:2011, Systems and software engineering - Systems and software Quality Requirements and
Evaluation (SQuaRE) - System and software quality models
4 Terms and definitions
For the purposes of this International Standard, the following terms and definitions and those given in ISO/IEC
25000 and in ISO/IEC 25010 apply.
NOTE: The essential definitions from ISO/IEC 25000 SQuaRE series and the other ISO standards are reproduced here.
4.1
context completeness
degree to which a product or system can be used with the required levels of effectiveness, efficiency, satisfaction
and freedom from risk in each of the specified contexts of use
[SOURCE: ISO/IEC 25010:2011 modified – added "the required levels of" and changed "all" to "each of" for
clarification]
Note 1 to entry: Context completeness is a subcharacteristic of context coverage.
4.2
context coverage
degree to which a product or system can be used with effectiveness, efficiency, satisfaction and freedom from risk
in both specified contexts of use and in contexts beyond those initially explicitly identified
Note 1 to entry: Context of use is relevant to both quality in use and some product quality (sub)characteristics (where it is
referred to as “specified conditions”).
[SOURCE: ISO/IEC 25010:2011 4.1.5]
4.3
effectiveness
accuracy and completeness with which users achieve specified goals
[SOURCE: ISO 9241-11:1998]
4.4
efficiency
resources expended in relation to the accuracy and completeness with which users achieve goals
[SOURCE: ISO 9241-11:1998]
9
© ISO/IEC 2015 – All rights reserved 9

---------------------- Page: 8 ----------------------
Note 1 to entry: Relevant resources can include time to complete the task (human resources), materials, or the financial cost of
usage.
4.5
context of use
users, tasks, equipment (hardware, software and materials), and the physical and social environments in which a
system, product or service is used
[SOURCE: ISO 9241-11:1998, 3.5, with "product" replaced by "system, product or service".]
4.6
flexibility
degree to which a product or system can be used with acceptable levels of effectiveness, efficiency, freedom from
risk in contexts beyond those initially specified in the requirements satisfaction and
[SOURCE: ISO/IEC 25010:2011, modified – added "acceptable levels of" for clarification.]
Note 1 to entry: Flexibility is a subcharacteristic of context coverage.
4.7
formative evaluation
evaluation designed and used to improve the object of evaluation, especially when it is still being developed
[SOURCE: ISO/TS 18152, 2010, 4.6]
4.8
freedom from risk
degree to which the quality of a product or system mitigates or avoids potential risks to economic status, human life,
health, or the environment
Note 1 to entry: Risk is a function of the probability of occurrence of a given threat and the potential adverse consequences of
that threat's occurrence.
[SOURCE: ISO/IEC 25010:2011, modified – added "quality of" and "or avoids" for clarification]
Note 2 to entry: The risks considered by the SQuaRE series are those arising from insufficient product quality.
Note 3 to entry: Freedom from risk includes reduction of potential risks to the user, organisation or project.
4.9
goal
intended outcome
[SOURCE: ISO 9241-11:1998]
4.10
measure (noun)
variable to which a value is assigned as the result of measurement
Note 1 to entry: The term “measures” is used to refer collectively to base measures, derived measures, and indicators.
[SOURCE: ISO/IEC 15939:2007]
Note 2 to entry: In this document when the word "measure" is used qualified by a characteristic or subcharacteristic it refers to
a quality measure.
4.11
measurement
set of operations having the object of determining a value of a measure
[SOURCE: ISO/IEC 15939:2007]
Note 1 to entry: Measurement can include assigning a qualitative category such as the language of a source program (ADA, C,
COBOL, etc.).
© ISO/IEC 2015 – All rights reserved
10

---------------------- Page: 9 ----------------------
ISO/IEC DIS 25022.2(E)

4.12
measurement function
algorithm or calculation performed to combine two or more quality measurement elements
[SOURCE: ISO/IEC 25021:2012]
4.13
psychometrics
field of study concerned with the theory and technique for developing valid and reliable psychological measures
4.14
quality in use
degree to which a product or system can be used by specific users to meet their needs to achieve specific goals
with effectiveness, efficiency, satisfaction and freedom from risk in specific contexts of use
[SOURCE: ISO/IEC 25010: 2011]
Note 1 to entry: The quality in use of a software product or system can be measured and evaluated by the effect of the target
system or software products when used by users of the implemented system, or during field testing or prototype testing.
Note 2 to entry: When quality in use is specified it relates to specified users meeting their needs to achieve specified goals with
effectiveness, efficiency, satisfaction and freedom from risk in specified contexts of use.
4.15
quality measure
measure that is defined as a measurement function of two or more values of quality measure elements
[SOURCE: ISO/IEC 25021:2012]
4.16
quality measure element
measure defined in terms of a property and the measurement method for quantifying it, including optionally the
transformation by a mathematical function
[SOURCE: ISO/IEC 25021:2012]
4.17
quality model
defined set of characteristics, and of relationships between them, which provides a framework for specifying quality
requirements and evaluating quality
[SOURCE: ISO/IEC 25000:2014]
4.18
satisfaction
degree to which user needs are satisfied when a product or system is used in a specified context of use
Note 1 to entry: For a user who does not directly interact with the product or system, only purpose accomplishment and trust
are relevant.
Note 2 to entry: Satisfaction is the user’s response to interaction with the product or system, and includes attitudes towards use
of the product.
[SOURCE: ISO/IEC 25010:2011]
Note 3 to entry: Users include: primary users who interacts with the system to achieve the primary goals, secondary users who
provide support and indirect users who receives output, but does not interact with the system.
Note 4 to entry: In this International Standard, user's needs include their desires and expectations associated with use of a
product, system or service. Exceeding desires and expectations is a means of significantly increasing satisfaction and improving
the user experience.
11
© ISO/IEC 2015 – All rights reserved 11

---------------------- Page: 10 ----------------------
4.19
stakeholder satisfaction
degree to which stakeholder needs are satisfied when a product or system is used in a specified context of use
[SOURCE: ISO/IEC 25010:2011 definition for the term "satisfaction" modified to refer to stakeholders]
Note 1 to entry: Users of a product or system are one type of stakeholder, so user satisfaction is one type of stakeholder
satisfaction.
4.20
summative evaluation
evaluation designed to present conclusions about the merit or worth of the object of evaluation
Note 1 to entry: The results can be used to produce recommendations about whether it should be retained, altered or
eliminated.
Note 2 to entry: It is possible to design a method to provide a combined formative and summative evaluation.
Note 3 to entry: A summative test method is used to perform a summative evaluation.
[SOURCE: ISO TS 20282-2:2012, 4.16]
4.21
system
a combination of interacting elements organised to achieve one or more stated purposes
Note 1 to entry: A system may be considered as a product or as the services it provides.
Note 2 to entry: In practice, the interpretation of its meaning is frequently clarified by the use of an associative noun, e.g.
aircraft system. Alternatively the word system may be substituted simply by a context dependent synonym, e.g. aircraft, though
this may then obscure a system principles perspective.
[SOURCE: ISO/IEC 15288:2008]
4.22
task
activities required to achieve a goal
Note 1 to entry: These activities can be physical or cognitive.
Note 2 to entry: Job responsibilities can determine goals and tasks.
[SOURCE: ISO 9241-11:1998]
4.23
usability
degree to which a product or system can be used by specified users to achieve specified goals with effectiveness,
efficiency and satisfaction in a specified context of use
Note 1 to entry: Based on ISO 9241-210.
Note 2 to entry: Usability can either be specified or measured as a product quality characteristic in terms of its
subcharacteristics, or specified or measured directly by measures that are a subset of quality in use.
[SOURCE: ISO/IEC 25010:2011]
4.24
use error
act or omission of an act that results in a different system response than intended by the manufacturer or expected
by the user
[SOURCE: IEC 62366:2007 with "medical device" replaced by "system"]
© ISO/IEC 2015 – All rights reserved
12

---------------------- Page: 11 ----------------------
ISO/IEC DIS 25022.2(E)

4.25
user
individual or group that benefits from a system during its utilization
[SOURCE: ISO/IEC 15939:2007]
5 Abbreviated terms
The following abbreviation is used in this International Standard.
 QME – Quality Measure Element
6 Use of quality in use measures
6.1 Applications of quality in use
Quality in use is the degree to which a product or system can be used by specific users to meet their needs to
achieve specific goals with effectiveness, efficiency and satisfaction and reduced risk in specific contexts of use.
NOTE 1 Quality in use is thus a combination of usability (when specified or measured directly by measures of effectiveness,
efficiency and satisfaction) and freedom from risk.
Quality in use measures the outcomes of interaction between a user and a system. The final quality in use can be
measured when an implemented system is used in the user’s environment for its intended purpose. Quality in use
measures can also be used at conceptual design and development stages.
 Quality in use requirements can be specified early in the design and development process, to give a high
level goal for the intended quality to be experienced by users and other stakeholders.
 Specific problems with quality in use can be identified by testing prototypes with small numbers of
representative users during development using unnormalised measures in order to improve the product
(formative evaluation).
 Estimates of the final quality in use of different design solutions can be obtained by testing prototypes with
larger numbers of representative users during development (summative evaluation using normalised
measures), and estimating how the resulting
...