ISO/IEC 14598-5:1998

INTERNATIONAL ISO/IEC
STANDARD 14598-5
First edition
1998-07-01
Information technology — Software product
evaluation —
Part 5:
Process for evaluators
Technologies de l’information — Évaluation du produit logiciel —
Partie 5: Procédés pour les évaluateurs
Reference number
B C
ISO/IEC 14598-5:1998(E)

---------------------- Page: 1 ----------------------
ISO/IEC 14598-5:1998(E)
Contents
Foreword.iv
Introduction.v
1 Scope.1
2 Conformance .1
3 Normative references.2
4 Definitions.2
5 Evaluation Concepts.3
5.1 General aspects . 3
5.2 Evaluation starting point. 3
5.2.1 Initial agreement.3
5.2.2 Parties involved in the evaluation .4
5.3 Characteristics of the evaluation process . 4
5.4 Evaluation process. 4
5.4.1 Evaluation activities.5
5.4.2 Input to the evaluation process.5
5.4.3 Output of the evaluation process.6
5.5 Relations between evaluation and life-cycle. 7
6 Evaluation process requirements.8
6.1 General requirements. 8
6.1.1 Organisation and quality system .8
6.1.2 Requester’s responsibilities.8
6.1.3 Evaluator’s responsibilities .9
6.2 Establishment of evaluation requirements . 9
6.2.1 Purpose of the establishment of evaluation requirements.9
6.2.2 Elaboration of the evaluation requirements .9
6.2.3 Contents of the evaluation requirements.10
6.2.4 Approbation and reporting .10
©  ISO/IEC 1998
All rights reserved. Unless otherwise specified, no part of this publication may be reproduced or
utilized in any form or by any means, electronic or mechanical, including photocopying and micro-
film, without permission in writing from the publisher.
ISO/IEC Copyright Office • Case postale 56 • CH-1211 Genève 20 • Switzerland
Printed in Switzerland
ii

---------------------- Page: 2 ----------------------
©
ISO/IEC ISO/IEC 14598-5:1998(E)
6.3 Specification of the evaluation.11
6.3.1 Purpose of the specification of the evaluation.11
6.3.2 Elaboration of the evaluation specification .11
6.3.3 Contents of the evaluation specification .13
6.3.4 Approbation and reporting .14
6.4 Design of the evaluation.14
6.4.1 Purpose of designing the evaluation.14
6.4.2 Elaboration of the evaluation plan.14
6.4.3 Contents of the evaluation plan.16
6.4.4 Approbation and reporting .16
6.5 Execution of the evaluation.16
6.5.1 Purpose of the evaluation execution.16
6.5.2 Performing the evaluator actions.16
6.5.3 Reviewing and reporting .18
6.6 Conclusion of the evaluation.19
6.6.1 Purpose of the evaluation conclusion.19
6.6.2 Joint review of the evaluation report.19
6.6.3 Disposition of evaluation data and documents.19
Annex A (normative) Template evaluation report.20
Annex B (informative) Levels of evaluation .22
Annex C (informative) Software product components.26
Annex D (informative) Interactions between requester and evaluator .30
Annex E (informative) Evaluation contract .32
Annex F (informative) Bibliography .35
iii

---------------------- Page: 3 ----------------------
©
ISO/IEC 14598-5:1998(E) ISO/IEC
Foreword
ISO (the International Organization for Standardization) and IEC (the International Electrotechnical
Commission) form the specialised system for worldwide standardisation. National bodies that are
members of ISO or IEC participate in the development of International Standards through technical
committees established by the respective organisation to deal with particular fields of mutual
interest. Other international organisations, governmental and non-governmental, in liaison with
ISO and IEC, also take part in the work.
In the field of information technology, ISO and IEC have established a joint technical committee
ISO/IEC JTC 1. Draft International Standards adopted by the joint technical committee are
circulated to national bodies for voting. Publication as an International Standard requires approval
by at least 75% of the national bodies casting a vote.
International Standard ISO/IEC 14598-5 was prepared by Joint Technical Committee ISO/IEC
JTC1, Information technology, Subcommittee SC 7, Software engineering.
ISO/IEC 14598 consists of the following parts, under the general title Information Technology -
Software product evaluation :
- Part 1: General overview
- Part 2: Planning and management
- Part 3: Process for developers
- Part 4: Process for acquirers
- Part 5: Process for evaluators
- Part 6: Evaluation modules
Annex A forms an integral part of this part of ISO/IEC 14598. Annexes B, C, D, E and F are for
information only.
iv

---------------------- Page: 4 ----------------------
©
ISO/IEC ISO/IEC 14598-5:1998(E)
Introduction
Software products are becoming more and more important in all domains of industry and services.
It is therefore necessary to be able to evaluate the quality of these software products.
Software products are extremely varied. They are produced to fulfil very diverse requirements in
terms, for example, of functionality. Their context for use can also be very varied such as in the case
of application software in a management information system, of software embedded in other
products or of game software, to cite a few examples.
Potential benefits from evaluation are:
- the developer can use the results of the evaluation of its product to identify corrective actions, in
order to improve the product or to make decisions about the evolution strategy for the product;
- for the supplier of a product the benefit from an evaluation can be to get confidence in the value
of the product; in addition the evaluation report can be used for commercial purposes;
- for software product acquirers, evaluation results may be used as objective data on which to base
acquiring decisions;
- for the industry at large, the spread of software product evaluation will help the use of quality as
a marketing argument.
The primary purpose of software product evaluation is to provide quantitative results concerning
software product quality that are comprehensible, acceptable to and can be depended on by any
interested party.
The evaluation process is described as a step-wise procedure that allows expression of evaluation
requirements in terms of quality characteristics as defined in ISO/IEC 9126. The evaluation takes
into account various documents that can be considered as part of the software product, e.g. design
documentation, test or validation reports, source code or user documentation. It is recommended
that the evaluator uses a library of evaluation modules that define evaluation methods. These
evaluation modules could be standardised, although no provision for that is proposed in this
standard. The evaluation leads to the production by the evaluator of an evaluation report.
This evaluation process is a generic abstract process that follows the model defined in ISO/IEC
9126. Therefore, this process is applicable within all primary life-cycle processes defined in
ISO/IEC 12207. Specific supporting life-cycle processes defined in ISO/IEC 12207 are directly
related to the evaluation process. They are quality assurance, verification, validation, joint review
and audit.
The tailoring process defined in ISO/IEC 12207 is built in the evaluation process defined in this
standard by allowing the user to specify and design the evaluation activities.
The evaluation process described here may be used to test the conformity to standards such as
ISO/IEC 12119.
v

---------------------- Page: 5 ----------------------
©
INTERNATIONAL STANDARD  ISO/IEC ISO/IEC 14598-5:1998(E)
Information technology —
Software product evaluation —
Part 5: Process for evaluators
1 Scope
This part of ISO/IEC 14598 provides requirements and recommendations for the practical
implementation of software product evaluation when several parties need to understand, accept and
trust evaluation results. In particular, it may be used to apply the concepts described in
ISO/IEC 9126.
The process described in this part of ISO/IEC 14598 defines the activities needed to analyse
evaluation requirements, to specify, design and perform evaluation actions and to conclude the
evaluation of any kind of software product.
The evaluation process may be used to evaluate already existing products, provided the needed
product components are available, or to evaluate products in development.
NOTE  For the evaluation of a product in development, the evaluation process needs to be
synchronized with the software development process and product components are evaluated as they
are delivered.
This part of ISO/IEC 14598 may be used by
- testing laboratory evaluators, when providing software product evaluation services,
- software suppliers, when planning evaluation of their products, including evaluation to be
carried out by independent testing services,
- software acquirers, when requesting evaluation information from a supplier or testing service,
- software users when evaluating products or when using evaluation reports provided by testing
laboratories,
- certification bodies in defining new certification schemes for software products.
2 Conformance
Because of the freedom of choice afforded to the user by the general nature of its recommendations,
a simple claim of compliance with this part of ISO/IEC 14598 is not valid. Any organization
imposing this part of ISO/IEC 14598 as a condition of trade is responsible for specifying and
making public a set of requirements which constitute the terms for compliance for a given
application of this part of ISO/IEC 14598. All requirements of clause 6 should be considered for
applicability.
1

---------------------- Page: 6 ----------------------
©
ISO/IEC 14598-5:1998(E) ISO/IEC
3 Normative references
The following standards contain provisions, which through reference in this text, constitute
provisions of this part of ISO/IEC 14598. At the time of publication, the editions indicated were
valid. All standards are subject to revision, and parties to agreements based on this part of
ISO/IEC 14598 are encouraged to investigate the possibility of applying the most recent editions of
the standards indicated below. Members of IEC and ISO maintain registers of currently valid
International Standards.
ISO/IEC 9126:1991, Information technology — Software product evaluation — Quality
characteristics and guidelines for their use.
1)
ISO/IEC 14598-1:— ,
Information technology — Software product evaluation — Part 1: General
overview.
1)
ISO/IEC 14598-6:— , Information technology — Software product evaluation — Part 6:
Evaluation modules.
4 Definitions
For the purposes of this part of ISO/IEC 14598, the following definitions apply.
4.1 evaluation method: a procedure describing the action to be performed by the evaluator in
order to obtain the result for the specified measurement or verification applied on the specified
product components or on the product as a whole.
4.2 evaluation report: the document that presents evaluation results and other information
relevant to an evaluation.
4.3 evaluation records: documented objective evidence of all activities performed and of all
results achieved within the evaluation process.
: the person or organisation that requests an evaluation.
4.4 evaluation requester
4.5 evaluation tool: an instrument that can be used during evaluation to collect data, to perform
interpretation of data or to automate part of the evaluation.
NOTE  Examples of such tools are source code analysers to compute code metrics, CASE tools to
produce formalised models, test environments to run the executable programs, checklists to collect
inspection data or spreadsheets to produce syntheses of measures.
4.6 evaluator: the organisation that performs an evaluation.
NOTE  An evaluator may, for example, be a testing laboratory, the quality department of a software
development organisation, a government organisation or a user.
4.7 software product developer: the person or organisation that manufactures a software
product.
___________
1)  To be published.
2

---------------------- Page: 7 ----------------------
©
ISO/IEC ISO/IEC 14598-5:1998(E)
4.8 software product evaluation: technical operation that consists of producing an assessment of
one or more characteristics of a software product according to a specified procedure.
NOTE 1  This definition can be compared to that of testing in ISO/IEC Guide 2. However, in this part
of ISO/IEC 14598, the term evaluation is preferred in order to avoid confusion with the notion of
testing widely accepted in the field of software engineering.
NOTE 2  Software product evaluation is not necessarily conformity testing (as defined in ISO/IEC
Guide 2, 13.3.2) in the context of a certification scheme. However, conformity testing can be part of
an evaluation.
5 Evaluation Concepts
5.1 General aspects
The quality of software products can be described in terms of quality characteristics as defined in
ISO/IEC 9126. However, the state of the art in software measurement is such that, in general, the
direct measurement of these characteristics is not practical. What is possible is to assess these
characteristics based on the measurement of lower abstraction attributes of the product.
In this context, the evaluator can use his or her experience in software engineering to make the
assessment. This might reduce the objectivity of the evaluation. Another aspect to be considered is
the possibility of using non-deterministic evaluation methods; although precisely defined, such a
method can require the evaluator to make choices which cannot be pre-defined.
NOTE  An example of a non-deterministic evaluation method is the one that consists of translating a
specification component of the product into a formal model and of performing performance or
reliability evaluation of this model; the translation phase involves many choices to be made by the
evaluator.
Therefore, provisions in this part of ISO/IEC 14598 are provided to maintain the level of objectivity
of evaluation as high as possible in all circumstances. These provisions bear on the organisation of
reviews of intermediate and final evaluation results and the keeping of records of the evaluation
process.
5.2 Evaluation starting point
5.2.1 Initial agreement
The evaluation of a software product occurs when the requester of the evaluation requests the
evaluator to perform an evaluation of this software product.
NOTE  When requesting the evaluation, the requester expresses evaluation requirements which are
analysed by the evaluator. The requester and the evaluator subsequently agree on the evaluation
specification.
3

---------------------- Page: 8 ----------------------
©
ISO/IEC 14598-5:1998(E) ISO/IEC
5.2.2 Parties involved in the evaluation
Potential requesters of evaluations are, for example,
- software developers,
- software suppliers,
- software acquirers,
- software users,
- system integrators in their role of software acquirers.
Potential evaluators are, for example,
- third party testing laboratories,
- testing entities within software producing or distributing organisations,
- testing entities within software buying or using organisations,
- testing entities within system integration organisations,
- organisations making comparisons between products.
In some cases, the developer of the software product is involved in the evaluation even if the
developer is not the requester of the evaluation.
5.3 Characteristics of the evaluation process
A principal objective of the evaluation process described in this part of ISO/IEC 14598 is to
promote the following desirable evaluation process characteristics:
- repeatability: repeated evaluation of the same product to the same evaluation specification by the
same evaluator should produce results that can be accepted as being identical,
- reproducibility: evaluation of the same product to the same evaluation specification by a
different evaluator should produce results that can be accepted as being identical,
- impartiality: the evaluation should not be biased towards any particular result,
- objectivity: the evaluation results should be factual, i.e. not coloured by the feelings or the
opinions of the evaluator.
NOTE  Evaluations of the same product can be conducted with different evaluation specifications.
They are therefore not comparable and may lead to different results.
5.4 Evaluation process
The evaluation process (see clause 6) consists of a set of activities which are conducted in co-
operation with the requester and the evaluator. These activities are performed on the basis of data
provided by the requester and the evaluator or produced by other activities. They produce data
which is used by other activities or which is the result of the evaluation process.
4

---------------------- Page: 9 ----------------------
©
ISO/IEC ISO/IEC 14598-5:1998(E)
The activities are designed to take into account the following issues:
- objectives vary from one evaluation case to another since software products are developed to
fulfil varied requirements and an evaluation requester may agree particular evaluation
requirements (see 6.2.1),
- software products are composed of components, the form and nature of which depend on
development methods which can be very different,
- possible evaluation techniques are numerous and need to be selected taking into account the
objectives of the evaluation and the composition of the product.
All these considerations impose a high flexibility for the process.
5.4.1 Evaluation activities
The evaluation process (see clause 6) comprises the five activities listed below:
- establishment of evaluation requirements (see 6.2.1);
- specification of the evaluation based on the evaluation requirements and on the description of the
product provided by the requester (see 6.3.1);
- design of the evaluation which produces an evaluation plan on the basis of the evaluation
specification; this activity takes into account the components of the software product to be
evaluated and the evaluation methods proposed by the evaluator;
- execution of the evaluation plan which consists of inspecting, modelling, measuring and testing
the products and its components according to the evaluation plan; these actions can be performed
using software tools (which are usually provided by the evaluator); the actions performed by the
evaluator are recorded and the results obtained are put in a draft evaluation report;
- conclusion of the evaluation, which consists of the delivery of the evaluation report and the
disposal by the evaluator of the product evaluated as well as its components when they have
been transmitted independently.
5.4.2 Input to the evaluation process
The requester provides the requester’s requirements which are an initial version of the evaluation
requirements.
The requester provides, during the evaluation, the following input to the evaluation process:
- the product description,
- the product components.
The product description identifies the software product as well as its components submitted for
evaluation.
5

---------------------- Page: 10 ----------------------
©
ISO/IEC 14598-5:1998(E) ISO/IEC
NOTE 1  The product may include documents related to planning, process or development methods
used for its production. A planning document may include schedule, organisation structure or
estimated costs.
NOTE 2  If the requester is a user, he or she should agree with the developer to support the evaluator
and may require the developer to deliver to the evaluator the description of the software component
and software product to be evaluated.
The evaluator provides the following input to the evaluation process:
- pre-defined evaluation specifications,
- evaluation methods and
- evaluation tools.
5.4.3 Output of the evaluation process
During the evaluation process, the evaluator provides the following output products:
- evaluation records, including evaluation plan and records of evaluation actions,
- the draft evaluation report, including evaluation requirements, evaluation specification and
synthetised evaluation results,
- the reviewed evaluation report.
The evaluation requirements, specification and plan are the intermediate products of the evaluation
process. The evaluation records and evaluation report are the final products of the evaluation
process.
The evaluation requirements describe the objectives of the evaluation; in particular, quality
requirements for the product are described.
The evaluation specification defines all analyses and measurements to be performed on the product
and on its components. The components of the product that will be analysed and measured are
identified.
The evaluation plan describes operational procedures needed to implement the evaluation
specification; in particular all the methods and tools to be used in the evaluation are described.
The evaluation records consist of the evaluation plan and a detailed account of actions performed by
the evaluator while executing the evaluation plan; these records are kept by the evaluator.
NOTE 1  The evaluation records are kept in order to allow re-processing of the evaluation results.
The evaluation report contains evaluation requirements, the evaluation specification, results from
the measurements and analyses performed and any other information necessary to be able to repeat
or reproduce the evaluation. The evaluation report is first issued as a draft for review. When in final
form, it is delivered to the requester.
6

---------------------- Page: 11 ----------------------
©
ISO/IEC ISO/IEC 14598-5:1998(E)
NOTE 2  The figure below gives an overview of the process described above. The information flow
between activities is identified.
Requester's
Evaluation
Requirements
Establishment Requirements
of Evaluation
Requirements
Evaluation
Specification
Specification
of the
Evaluation
Pre-defined
Product
Evaluation
Evaluation
Evaluation
Description
Design of
Specifications
Records
Plan
the
Record of
Evaluation
Evaluation
Evaluation
Actions
Evaluator's
Methods
Evaluation Tools
Input
Execution
of the
Evaluation
Draft Evaluation
Report
Reviewed
Synthetised
Product
Evaluation
Evaluation
Conclusion
Components
Report
Results
of the
Evaluation
Requester's
Input
Figure 1 — The evaluation process
5.5 Relations between evaluation and life-cycle
Evaluation of a software product can be performed within the context of any life-cycle process as
defined in ISO/IEC 12207. In particular, evaluation can occur within one of the acquisition, supply,
development, operating or maintenance processes.
The decision as to whether a software product evaluation is to be performed may be taken as early
as possible in the product development process. If this is done right at the beginning of the
development process, it is possible to build into the software development process the
measurements and tests to be performed for the evaluation. This would ensure the maximum
likelihood for the product to satisfy all requirements concerning the evaluation results, as well as
minimising the risk of extra, unexpected costs being incurred.
When the requester is the product developer, early contact with the evaluator to discuss the
intention of submitting a product for evaluation would also help the developer to anticipate any
special needs (such as particular documents or evidence which might be required) which the
evaluators could have.
It is possible that some (or even all) of the evaluation actions will have to be done on site rather than
at the evaluator’s. In this case, the actions will still be controlled by the evaluator to ensure that the
results are impartial.
For very large, complex software projects it would be beneficial for the developer to have
continuous, detailed co-operation with the evaluator during the whole development of the product to
7

---------------------- Page: 12 ----------------------
©
ISO/IEC 14598-5:1998(E) ISO/IEC
minimise the duration and cost of the evaluation process. This co-operation should be such that it
does not reduce the impartiality of the evaluator.
6 Evaluation process requirements
6.1 General requirements
6.1.1 Organisation and quality system
In order to satisfy the characteristics expressed in 5.3, i.e. repeatability, reproducibility, impartiality
and objectivity of the evaluation results, the evaluator shall act in an organisational context that
provides all necessary assurance to obtain sufficient quality for its activities. In order to satisfy this
requirement, the evaluator’s organisation may comply with the requirements prescribed in ISO/IEC
Guide 25.
6.1.2 Requester’s responsibilities
The responsibilities of the requester of the evaluation shall be
- to establish necessary legal rights in the software product for the purpose of the evaluation,
- to provide information necessary for identification and description of the product,
- to state initial evaluation requirements and to negotiate with the evaluator to determine the actual
evaluation requirements; these requirements for the evaluation should comply with relevant
regulations and standards,
- to state confidentiality requirements concerning the information submitted to the evaluation,
- to act, whenever necessary, as an intermediary between the developer and the evaluator,
- to provide the evaluator, whenever necessary, with suitable access to computers and other
equipment used for development and for operational use of the software product,
- to provide, whenever necessary, support to the evaluator, including training and access to
suitable staff,
- to ensure the timely supply, whenever necessary, of the software product, its description and
components, including documentation and other material,
- to inform, whenever necessary, the evaluator of any factor that might invalidate the evaluation
results.
8

---------------------- Page: 13 --
...