ISO/IEC 14598-6:2001

INTERNATIONAL ISO/IEC
STANDARD 14598-6
First edition
2001-06-01
Software engineering — Product
evaluation —
Part 6:
Documentation of evaluation modules
Ingénierie du logiciel — Évaluation du produit —
Partie 6: Documentation des modules d'évaluation
Reference number
ISO/IEC 14598-6:2001(E)
©
ISO/IEC 2001

---------------------- Page: 1 ----------------------
ISO/IEC 14598-6:2001(E)
PDF disclaimer
This PDF file may contain embedded typefaces. In accordance with Adobe's licensing policy, this file may be printed or viewed but shall not
be edited unless the typefaces which are embedded are licensed to and installed on the computer performing the editing. In downloading this
file, parties accept therein the responsibility of not infringing Adobe's licensing policy. The ISO Central Secretariat accepts no liability in this
area.
Adobe is a trademark of Adobe Systems Incorporated.
Details of the software products used to create this PDF file can be found in the General Info relative to the file; the PDF-creation parameters
were optimized for printing. Every care has been taken to ensure that the file is suitable for use by ISO member bodies. In the unlikely event
that a problem relating to it is found, please inform the Central Secretariat at the address given below.
© ISO/IEC 2001
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or utilized in any form or by any means, electronic
or mechanical, including photocopying and microfilm, without permission in writing from either ISO at the address below or ISO's member body
in the country of the requester.
ISO copyright office
Case postale 56 � CH-1211 Geneva 20
Tel. + 41 22 749 01 11
Fax + 41 22 749 09 47
E-mail copyright@iso.ch
Web www.iso.ch
Printed in Switzerland
ii © ISO/IEC 2001 – All rights reserved

---------------------- Page: 2 ----------------------
ISO/IEC 14598-6:2001(E)
Contents Page
1 Scope .1
2 Conformance.1
3 Normative references .1
4 Terms and definitions .2
5 The evaluation module concept.2
6 Format for documentation of an evaluation module.3
6.1 EM0 : Foreword and introduction .3
6.1.1 Foreword.3
6.1.2 Introduction.3
6.2 EM1 : Scope.3
6.2.1 Characteristics.3
6.2.2 Level of evaluation .3
6.2.3 Techniques.4
6.2.4 Applicability.4
6.3 EM2 : References.4
6.4 EM3 : Terms and definitions.4
6.5 EM4 : Inputs and metrics .4
6.5.1 Input for the evaluation .4
6.5.2 Data elements.5
6.5.3 Metrics and measures .5
6.6 EM5 : Interpretation of results.5
6.6.1 Mapping of measures.5
6.6.2 Reporting.5
6.7 EMA : Application procedure .5
6.7.1 Definition of Technical Terms Used.5
6.7.2 Resources Required.5
6.7.3 Evaluation instructions .6
6.7.4 Documentation.6
Annex A (informative) Development of evaluation modules .7
Annex B (informative) Example of an evaluation module – Fault density.8
Annex C (informative) Example of an evaluation module – Functionality .12
Annex D (informative) Example of an evaluation module – Usability and Quality in Use .25
Bibliography.31
© ISO/IEC 2001 – All rights reserved iii

---------------------- Page: 3 ----------------------
ISO/IEC 14598-6:2001(E)
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical Commission)
form the specialized system for worldwide standardization. National bodies that are members of ISO or IEC
participate in the development of International Standards through technical committees established by the
respective organization to deal with particular fields of technical activity. ISO and IEC technical committees
collaborate in fields of mutual interest. Other international organizations, governmental and non-governmental, in
liaison with ISO and IEC, also take part in the work.
International Standards are drafted in accordance with the rules given in the ISO/IEC Directives, Part 3.
In the field of information technology, ISO and IEC have established a joint technical committee, ISO/IEC JTC 1.
Draft International Standards adopted by the joint technical committee are circulated to national bodies for voting.
Publication as an International Standard requires approval by at least 75 % of the national bodies casting a vote.
Attention is drawn to the possibility that some of the elements of this part of ISO/IEC 14598 may be the subject of
patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent rights.
International Standard ISO/IEC 14598-6 was prepared by Joint Technical Committee ISO/IEC JTC 1, Information
technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598-6 is intended for use in conjunction with ISO/IEC 9126-1 (in preparation) which will replace
ISO/IEC 9126:1991.
ISO/IEC 14598 consists of the following parts, under the general title Software engineering — Product evaluation:
� Part 1: General overview
� Part 2: Planning and management
� Part 3: Process for developers
� Part 4: Process for acquirers
� Part 5: Process for evaluators
� Part 6: Documentation of evaluation modules
Annexes A, B, C and D of this part of ISO/IEC 14598 are for information only.
iv © ISO/IEC 2001 – All rights reserved

---------------------- Page: 4 ----------------------
ISO/IEC 14598-6:2001(E)
Introduction
Software product evaluation depends on a set of evaluation techniques and metrics that provide information about
the quality characteristics of the software. Many metrics and associated methods for using the measurement
results can be used for specific software product evaluation. ISO/IEC 9126-2 and ISO/IEC 9126-3 provide example
metrics that correspond to one sub-characteristic. It is difficult to use these metrics consistently in an organisation.
It may be necessary to develop new metrics for specific use. Therefore, it is necessary that a supporting function
(see 14598-2) in the organisation specifies each metric for correct and consistent use within the organisation. The
format for documenting a metric and associated methods, as well as guides for their use, should be standardised.
The concept of an evaluation module provides a solution to this need.
An evaluation module specifies the evaluation methods applicable for evaluating a quality characteristic and
identifies the evidence it needs. It also defines the elementary evaluation procedure and the format for reporting the
measurements resulting from the application of the techniques.
A consistent way of documenting evaluation modules has a number of advantages:
� It provides a common reference in the description of the theoretical basis of evaluation modules.
� It identifies a minimal set of requirements for documenting and developing evaluation modules.
� It provides a necessary tool in collecting and cataloguing the large number of evaluation modules anticipated.
Evaluation modules provide a flexible and structured approach to making metrics applicable for evaluating
intermediate and completed products. The use of evaluation modules produced according to this part of ISO/IEC
14598 helps to ensure that software product evaluations can be repeatable, reproducible and objective.
The format for documenting an evaluation module takes into account the following:
� It will be applied within the context of the evaluation of software products.
� The format supports the need for developing new metrics with respect to state of the art.
� The format provides a precise definition of metrics and their application.
� It provides the information needed for those who will use them for an evaluation.
Annex A provides guidance for the development process for new evaluation modules.
Annexes B, C and D are examples of evaluation modules.
© ISO/IEC 2001 – All rights reserved v

---------------------- Page: 5 ----------------------
INTERNATIONAL STANDARD ISO/IEC 14598-6:2001(E)
Software engineering — Product evaluation —
Part 6:
Documentation of evaluation modules
1 Scope
This part of ISO/IEC 14598 defines the structure and content of the documentation to be used to describe an
Evaluation Module. Evaluation modules are intended to be used within the context of the ISO/IEC 9126 and the
ISO/IEC 14598 multipart standards.
This part of ISO/IEC 14598 is intended to be used by experts in evaluation technology such as testing laboratories,
research institutes and others when producing new evaluation modules.
2 Conformance
The documentation of an evaluation module conforms to this part of ISO/IEC 14598 if it meets the requirements of
clause 6 (format for documentation of an evaluation module).
3 Normative references
The following normative documents contain provisions which, through reference in this text, constitute provisions of
this part of ISO/IEC 14598. For dated references, subsequent amendments to, or revisions of, any of these
publications do not apply. However, parties to agreements based on this part of ISO/IEC 14598 are encouraged to
investigate the possibility of applying the most recent editions of the normative documents indicated below. For
undated references, the latest edition of the normative document referred to applies. Members of ISO and IEC
maintain registers of currently valid International Standards.
ISO/IEC 9126-1, Software engineering — Product quality — Part 1: Quality model.
ISO/IEC 12207, Information technology — Software life cycle processes.
ISO/IEC 14598-1, Information technology — Software product evaluation — Part 1: General overview.
ISO/IEC 14598-2, Product evaluation — Part 2: Planning and management.
ISO/IEC 14598-3, Product evaluation — Part 3: Process for developers.
ISO/IEC 14598-4, Software engineering — Product evaluation — Part 4: Process for acquirers.
ISO/IEC 14598-5, Information technology — Software product evaluation — Part 5: Process for evaluators.
© ISO/IEC 2001 – All rights reserved 1

---------------------- Page: 6 ----------------------
ISO/IEC 14598-6:2001(E)
4 Terms and definitions
For the purposes of this part of ISO/IEC 14598, the following terms and definitions apply.
4.1
evaluation module
a package of evaluation technology for measuring software quality characteristics, sub-characteristics or attributes
NOTE The package includes.
� evaluation methods and techniques
� inputs to the evaluation
� data to be measured and collected
� supporting procedures and tools
4.2
evaluation technology (technology used for evaluation)
the techniques, tools, metrics, measures and other technical information used for evaluation [ISO/IEC 14598-2]
5 The evaluation module concept
Evaluation of software products can be a comprehensive task. Different aspects of quality characteristics and sub-
characteristics may require different evaluation techniques to be applied and different data to be collected. In order
to manage this complexity, an evaluation should be structured into manageable units. Each of these units can
cover one or more quality aspects. However, each unit should be focused on the evaluation of one specific quality
aspect applying a specific evaluation technique. The information necessary for conducting one of these evaluations
should be collected and packaged for future use. Such a package is called an evaluation module.
The benefit of using a standardized format of an evaluation module is that
� it supports the development of evaluation modules since it provides a table of contents which makes visible
what information is necessary for an evaluation and how this information is handled (principles, metrics, tools,
...)
� it supports the use of evaluation modules since information will be available in a homogenous way
� it supports the reuse of evaluation modules since it facilitates the establishment and maintenance of libraries of
evaluation modules
� it supports standardization of evaluation modules since the format is compliant with standards requirements
An evaluation module collects all information necessary to perform an evaluation of a specific aspect of a quality
characteristic applying a specific evaluation technique. It clarifies which specific aspect of a software quality
characteristic is being measured. The procedure for making the measurement is defined as well as the
preconditions and accuracy of the measurement.
Evaluation modules provide the link between evaluation techniques, metrics and measures. When parts 3, 4 or 5 of
ISO/IEC 14598 recommend the application of an evaluation technique, an appropriate evaluation module can be
selected from an evaluation module library (see ISO/IEC 14598-2).
The documentation of an evaluation module has six parts EM0 - EM5 serving different purposes, and an optional
annex EMA.
2 © ISO/IEC 2001 – All rights reserved

---------------------- Page: 7 ----------------------
ISO/IEC 14598-6:2001(E)
EM0 provides formal information about the evaluation module and gives an introduction to the evaluation technique
described in the evaluation module.
EM1 defines the scope of applicability of the evaluation module.
EM2 provides relevant references.
EM3 includes the definitions needed for the evaluation module.
EM4 specifies the input products required for the evaluation and defines the data to be collected and measures to
be calculated.
EM5 contains information about how to interpret measurement results.
The optional annex EMA includes the detailed procedure for applying the evaluation module. Although EMA is
optional it is suggested that it be included.
NOTE The format for the documentation of an evaluation module complies with the formal requirements of a standard as
described in ISO Directives Part 3. This facilitates standardization of evaluation modules.
6 Format for documentation of an evaluation module
The documentation of an evaluation module shall be formatted according to 6.1, 6.2, 6.3, 6.4, 6.5, 6.6 and 6.7.
6.1 EM0 : Foreword and introduction
6.1.1 Foreword
This clause shall provide information about
� preparation, approval, contributions and changes,
� relationship to other standards or other documents.
6.1.2 Introduction
This clause should introduce the principles, background and technical rationale underlying the evaluation module.
NOTE A formal description of evaluation method is provided in 6.2.3.
6.2 EM1 : Scope
6.2.1 Characteristics
This clause shall identify the characteristics, sub-characteristics or attributes that the evaluation module can
evaluate.
NOTE The evaluation module may contribute to one or more characteristics or sub-characteristics.
The quality model describing the characteristics, sub-characteristics or attributes shall be identified. The model in
ISO/IEC 9126-1 should be used, unless there is a particular reason to use another model.
6.2.2 Level of evaluation
This clause shall describe the level of the evaluation defined by the evaluation module. Evaluation levels are
related to the importance of the characteristic, sub-characteristics or attribute evaluated. The level should be
© ISO/IEC 2001 – All rights reserved 3

---------------------- Page: 8 ----------------------
ISO/IEC 14598-6:2001(E)
described taking into account the assumed usage of the software and the environment of the software product (for
example, safety conditions, security constraints, economic risks, and application constraints).
The level defines the depth or thoroughness of the evaluation in terms of the evaluation techniques to be applied
and evaluation results to be achieved. Different evaluation levels give different levels of confidence in the quality of
the software product.
NOTE Levels can be formulated as A, B, C, or D as described in ISO/IEC 14598-5. Software integrity levels are described
in ISO/IEC 15026.
6.2.3 Techniques
This clause shall describe the evaluation techniques applied by the evaluation module. The relevant theory, models
or heuristics underlying the evaluation shall be included or adequately referenced.
NOTE Examples of evaluation techniques are reliability growth models, benchmark testing, static analysis of code.
6.2.4 Applicability
This clause shall identify the scope of application of the evaluation module.
NOTE 1 For example, the evaluation module may be applicable to a particular programming language, or to the class of all
imperative languages.
It shall be described where in the software life cycle the evaluation module can be used. If it is intended to be used
in a specific software life cycle process, this should be identified.
NOTE 2 Software life cycle processes are described in ISO/IEC 12207.
6.3 EM2 : References
This clause shall provide references to normative and technical documents. If the evaluation module depends on
the result of other evaluation modules, then these shall be mentioned here.
6.4 EM3 : Terms and definitions
This clause shall define technical terms used in the evaluation module. Alternatively, references to sources in which
definitions can be found shall be made.
6.5 EM4 : Inputs and metrics
6.5.1 Input for the evaluation
This clause shall identify the inputs required for the evaluation. These shall be classified as product component,
product information, supporting information and product in use information.
NOTE 1 Information classified as product component includes software requirements specification, software design
description, program description, source code, executable code, and user documentation.
NOTE 2 Information classified as product information includes software requirements review report, software design review
report, program review report, unit test report, and user documentation review report.
NOTE 3 Information classified as supporting information includes quality assurance plan, configuration management plan,
program test plan, and description of programming language and compiler. Supporting information is not evaluated but only
used as background information necessary for conducting the evaluation.
NOTE 4 Information classified as product in use information includes testing report and operation report describing the
behavior of the system. System includes any associated hardware, software and users.
4 © ISO/IEC 2001 – All rights reserved

---------------------- Page: 9 ----------------------
ISO/IEC 14598-6:2001(E)
6.5.2 Data elements
This clause shall specify which data elements are to be extracted from the inputs.
NOTE 1 Examples of data elements are: Number of lines of code containing comments; frequency distribution of sentence
length in the user-manual; number of words in each help message; number of failures observed per hour of operation; number
of tokens of specified types found in each module by a specified lexical scan.
NOTE 2 In general, the data elements are the material from which measures are computed. But in some cases, the raw data
may itself constitute metrics.
6.5.3 Metrics and measures
This clause shall describe how the measures are calculated from the data elements using metrics.
If the metrics are to be combined to obtain ‘higher-order’ metrics, then the dependency shall be made explicit.
All assumptions and preconditions to be satisfied before measuring shall be stated.
6.6 EM5 : Interpretation of results
6.6.1 Mapping of measures
This clause shall specify the meaning of the measures, that is, the interpretation of the results of the measurement.
This includes the evaluation scale onto which the values obtained by the defined metrics are to be mapped. If the
mapping is non-trivial then the details of the algorithms (functions) needed to make the mappings shall be defined
or references shall be made to their sources. If several measures are obtained for a single characteristic, sub-
characteristics or attribute, then the clause shall describe how they can be combined into a rating for the
characteristic, sub-characteristics or attribute.
The accuracy of the measure shall be specified.
6.6.2 Reporting
This clause shall describe the contents of the report providing the result of applying the evaluation module. In some
cases, visualization of obtained values is important and should be encouraged.
6.7 EMA : Application procedure
NOTE The inclusion of EMA is optional but, if included, it shall have the following contents.
6.7.1 Definition of Technical Terms Used
This clause shall define technical terms that are not defined in 6.4, but used in part EMA of the evaluation module,
or referenced sources.
6.7.2 Resources Required
This clause shall specify the resources required when applying the evaluation module. This should include:
Software tools required (any software tools required should be identified and both generic tool types and
proprietary tools should be referenced); hardware/software needed; test-harness or other equipment; skills and
qualifications (any special skills and qualifications (for example, certification) required by the evaluator or the
evaluating organization should be identified); effort of application (the effort required for typical application of the
evaluation module should be estimated and, if this effort depends on attributes of the product (for example, number
of lines of code), an estimating algorithm should be given); and any other resources required.
© ISO/IEC 2001 – All rights reserved 5

---------------------- Page: 10 ----------------------
ISO/IEC 14598-6:2001(E)
6.7.3 Evaluation instructions
This clause shall describe full details of the procedure to be followed. This should include the selection of evidence
(for example, sampling of code), the generation and recording of raw data, counting rules, algorithms for computing
metrics from raw data, the recording of results, and requirements for the retention of working and final
documentation. In particular, the steps to be taken to ensure traceability and repeatability of results should be
highlighted. The described procedure should be compliant with ISO Guide 25.
6.7.4 Documentation
This clause shall outline the internal documentation resulting from the usage of the evaluation module. The outlined
report should be compliant with ISO Guide 25.
6 © ISO/IEC 2001 – All rights reserved

---------------------- Page: 11 ----------------------
ISO/IEC 14598-6:2001(E)
Annex A
(informative)
Development of evaluation modules
This informative annex provides guidance for the process of developing new evaluation modules. The new ISO/IEC
9126 part 2, part 3 and part 4 may serve as input for this process. The evaluation module development process
should include five steps:
A.1 Identification of the evaluation module requirements
When the need for a new evaluation module is identified and a decision is made to develop the evaluation module,
the first step should be to identify the requirements for the new evaluation module. This includes identification of a
quality model and quality characteristic or sub-characteristic. Also, the stringency of the evaluation should be
decided.
A.2 Specification of the evaluation module
Based on the identified requirements for the evaluation module the next step is to specify the evaluation techniques
and inputs to the evaluation (for example, source code) as well as the set of metrics and the fundamental set of
data elements. Here ISO/IEC 9126 part 2, part 3 and part 4 may be useful.
A.3 Development of the evaluation procedure
This step takes the formal specification from the previous step and adds some procedural aspects. The
interpretation of metrics and data elements should be explained in the context of the evaluation. The resources
required should be estimated and detailed evaluation procedures should be developed. Trial application of the
evaluation procedure may be necessary.
A.4 Description of the evaluation procedure
In this step the evaluation procedure developed in the previous step should be described according to the
evaluation module format, that is, the description should conform to this part of ISO/IEC 14598.
A.5 Verification and validation of the evaluation module
The evaluation module should be reviewed (verified) against its specification. Experts in the area covered by the
evaluation technique described should do this. The validation should ensure that the evaluation module represents
state of the art technology and existing technology that is unfamiliar to the organization.
The evaluation module should be tested (validated) by different groups of people in different environments. The
experience obtained should be fed back to the evaluation module development team as input for an update.
© ISO/IEC 2001 – All rights reserved 7

---------------------- Page: 12 ----------------------
ISO/IEC 14598-6:2001(E)
Annex B
(informative)
Example of an evaluation module – Fault density
Information technology - Software product evaluation - Evaluation Module: Fault density
B.0 Introduction
This evaluation module is used to determine the fault density of a program. Detecting a high number of faults
during design and testing phase reduces potential faults in the software, which will cause failures in the operation
phase. A high number of faults at the beginning of the operation phases cause frequent failures and decreases
product reliability. Therefore it should be ensured that the fault density is below a specified threshold value before
the program is put into operation.
In general, it is impossible to count the number of faults remaining in a software product. But by using the model
and the historical data of fault detection, the number can be estimated. The fault density is calculated using this
estimate. The general estimation procedure is as below:
1) Choose an appropriate Reliability Growth Model RGM to adopt, for example, an exponential RGM or S-shaped
RGM.
2) Record the cumulative number of detected faults with time at a particular point in the test period.
3) Decide on the number of parameters of the RGM equation that are required to fit the curve to the set of data
recorded.
4) As time (t) of the RGM equation tends to infinity, the estimated number of potential faults can be calculated.
B.1 Scope
B.1.1 Characteristics
Reliab
...