EuroSPI99 
Learn from the Past - Experience the Future
European Software Process Improvement
SPI and Assessments
Category Index
Rated Newspaper Supported by EU Project 

The General Effect of an Integrated Software Product Evaluation

Jozsef Gyorkos , Ivan Rozman, Robert T. Leskovar
1 University of Maribor, Faculty of Electrical Engineering and Computer Science,
Smetanova 17, SI-2000 Maribor, Slovenia; e-mail: gyorkos@uni-mb.si
 

Abstract

Well organized software product evaluation has a general effect to the discipline of the software processes. In our paper a case of the governmental institution where step-by-step introduction of formal evaluation methodologies has been forced to the outsourced projects is presented. The strategic influence of outsourcing is stressed throughout and the methods of risk prevention with improved communication of the outsourced client and vendor described. Risk prevention is based upon the formal controllability of outsourced projects. Formality is achieved by using, firstly, project management environments, which allow communication between parties, whereby it is not only a planning/tracing tool. Secondly, there are process and product assessment mechanisms.

Introduction

Outsourcing is an important method of managing information systems (IS). In relation to government projects, it is now a common practice, primarily when there is a requirement for the development of different applications, integrated into a comprehensive system.

Changes in the labour market have brought even more intensive outsourcing. Recently companies have moved away from traditional long term employment arrangements (insourcing) to relatively short-term market mediated arrangements (outsourcing) [Slaughter, 1996]. This source states (Derived from research in U.S.), that firms in the public sector are more likely to outsource IS employment, than firms in the private sector.

Our paper will use as a basis for the discussion the Government Information Centre of the Republic Slovenia, the CVI.

Outsourcing has brought about dynamic boundary changes, i.e. (distributed application development, non transparent organisation structure of vendors and heterogeneous methodologies). However, the continuos operation of an integrated information system, must be assured at a conceptual level, being in relation to the data and common functional structures. The structure is based on the rules and processes of public administration, but at the same time it is highly dependent on actual decisions, rules and policies of the State and (actual) Government. Outsourcing also circulates the resources (money) received from the tax payer, back to its productive source - the tax payer.

CVI has a very broad aim, that is the introduction and maintenance of information technology in all government institutions, and to lesser degree public institutions.

Slovenia is small country in relation to many other European states, and like others born or 'reborn' in the late 1980s and early 1990s, it has had to develop a whole new IS supported administrative apparatus, which is a requirement for a modern functional state. In the world of computer technology, there is a similar situation, whereby, there are not so many clients, but the functionality of the server is the same as in larger states.

Types and Extent of Outsourcing

Outsourcing is the contracting of various systems to outside information systems vendors [Nam, 1996]. For most managers – IS outsourcing clients - outsourcing sounds like 'entropy' – the possibility for disorder an uncertainty in a system. Consequently, outsourced projects can be endangered by 'entropy', if they are not managed in a proper fashion. The minimal attributes of 'properly' managed IS are those well known from software process models - best illustrated in the key process areas of Capability Maturity Model [Paulk, 1995]. For example, at least the second level key process areas must be covered: requirements management, project planning, configuration management, project tracing and oversight, quality assurance, subcontract management.

Nam mentions four types of outsourcing when he talks of the relationship between 'Extent of substitution by vendor' and 'Strategic impact of applications', these are: support, reliance, alignment and alliance. Using the CVI as a model we can see the distribution of outsourcing contracts among different types of outsourcing in Table I below.

In non-information technology industries, 'alignment' and 'alliance' are the usual types of outsourcing. In the forthcoming information society, the business process in a given public administration will be under pressure to re-engineer, due to the process automation and the demand for information dissemination, with or without administrative assistance.

The low level of alignment and almost non-alliance in outsourcing explains the strategic position of CVI. Whereby, we have not only a reliable information technology (IT) service provider for public administration but also the added responsibility of having a forum for making long-term, highly influential decisions in IT.

Table I

Types of Outsourcing and

their Approximate Extent in Case of CVI

Type of Outsourcing Tasks (e.g.) Distribution of Outsourcing Contracts*
1. Support
(non-core IS activities, small contracts)		 contract programming,
hardware maintenance,
minor technical services,
installations etc.
50%
2. Reliance
(large extent of substitution by vendors in non-core IS activities)		 same activities and tasks as above, longer length of contracts
30%
3. Alignment
(low extent of substitution and high strategic impact)		 consulting,

technical supervision for IS planning and design,

small contracts with strategic influence
20%
4. Alliance
(high extent of substitution by vendor and high strategic impact)		 substitution of in-house IS operations and vendors' responsibility for highly strategic IS activities,

based on mutual relationships,

highest commitments from vendors and clients
almost 0%

* Derived out of values of contracts.


Supervision Threshold

Preparation and control of the Outsourced projects performed by client, are both highlighted in this paper.

The scope of preparation is the requirements definition and selection of the vendor (usually directed by the legislative council). The outsourcing vendor is the organisation which delivers the required product and/or service. In this paper we will focus on outsourced projects where software develops within information technology projects and all IT related services.

Control of the Outsourced project by client consist of project tracing, quality assurance and of change management (technical or contextual – business process - changes).

When an outsourced project is running, threshold must be drawn on the scale, representing the degree of supervision of the vendor by the client. Table II highlights the achievements and drawbacks for a client regarding the different degrees of supervision.

Sources of Risk

Different classifications of risk can be found in the literature [Boehm, 1989; PRINCE, 1995] but the aim of all methodologies is the prevention of project being endangered. Risk management is an built-in part of practically all software development methodologies and quality assurance techniques (at least indirectly). The different phases of the software process, or software project development, have the effect, whereby, risk management appears in many forms and with different scopes of influence. Table III highlights this.
 
 

Table Ii

Degree of Client Supervision in outsourced Projects

Type of supervision Supervised products Achievements 
for Client		 Drawbacks 
for Client
minor general specification,

contract,

integration test results of the final product

 minimal in-house resources needed unexpected results
managed as in minor,

project tracing results
(schedule, resources)

 dependable resource management technical imperfection 
can turn up
technical as in minor,

technical specification,

change management records,

test coverage plans and test results

 technical overview, reliable products highly-specialised technical staff required
optimal minor, managed and the technical joined early uncertainty detection,

reliability

 resource pretentious,

asking about worth of outsourcing

 

Evolving Practice

Three years ago within our environment of organised activities of risk prevention in outsourced projects, there was the introduction of an overall quality system based on ISO 9001, with the addition of a Capability Maturity Model. We gave a special attention to those processes that contributed to the overall control of (1) the outsourcing vendors and their processes, (2) the products delivered by vendors and (3) the activities of the client.

It is important to stress, that these processes not only control the vendor but also contribute to a flexible and transparent project management in relation to the client (in our case the CVI). The intensity of project management activities, carried out by a client, depends on the type of supervision, formulated in Table II. Table IV, enumerates the existing practices within CVI in relation to risk prevention.
 

A. Assessment of the process

The most important aspect of the assessment of the process is, when a new outsourcer is chosen i.e. - no previous experience with him or her. This assessment can now be carried out using a different criteria, usually CMM or ISO and this is done before a formal agreement between vendor and client is agreed. PROCESSUS methodology is highlighted in [Györkös, 1996; Rozman, 1997]. This approach is also mentioned in Table III, which joins both approaches in an comprehensive tool supported questionnaire. Figure 1 shows the logical object model of the tool. However assessment of any process as a means risk prevention, must be done also when the project is in progress, with checking of existing practice and of an active quality assurance model established.

Table Iii

Means of risk management in the software process.

Phase/Part
of the Process		 Critical Success 
Factors		 Means of Risk Management
Requirement
definition		 formality of requirements specification (RS)
compliance of RS with
- long term strategy
- actual needs		 Audit and 

Review of specification
Procurement/
acquisition		 credibility of outsourcer in public competition Audit, reference and
previous experience

(PROCESSUS Tool, chapter 5.1)
phase of
development		 methodological approach enabling traceability and technical quality Technical review

Project tracing
Release product quality (using the parameters derived from ISO 9126) Product assessment (PRO+ Tool, chapter 5.2)

 

When consistent project management is used only the ‘products’ of the project management process can be checked. In our case the consistency of project management, ensures the adopted PRINCE methodology [PRINCE, 1996]. This PRINCE methodology is applied in the form of ProjectOffice, a distributed environment with functionalities, i.e. project group co-ordination, process control, exception handling and quality assurance.
 

B. Assessment of the product

Review is an activity that checks the correctness of particular software development phase at technical level. In the context of an ‘outsourcing review’, this is rarely done by the outsorcing client, but by the vendor himself. The vendor is obliged to perform the quality assurance of his products at a technical level. The client checks periodically whether the partial results are correct, this results are then published within progress reports by the ProjectOffice.

Before a release, the customer (CVI) performs one final assessment of the product not only from general view. This assessment can take up to five days, depending on the size of the product and the level of its integration with other (existing or in-development) products. These results are then collated in a tool called PRO+ which is planned to be a consistent part of the ProjectOffice application.

PRO+ is based on quality attributes described in standards; ISO 9126 (IT – Software product evaluation – Quality characteristics and guidelines for their use), and partialy on ISO 9127 (Information processing systems – User documentation and cover information for consumer software packages) and ISO 12119 (IT – Software packages – Quality requirements and testing). In the background of the tool a formal decision support is defined, this enables calibration of parameters and in the process will give reports of the different views (client, developer, user). The calibration is fully transparent and enables multiple-level assessment of the same product.

Table IV

Existing risk prevention practices in CVI

Activity/task, 
subject of the outsourcing contract		 Type of outsourcing (according to Table I) Method Tool Practice
application development
(full life cycle, technical level)		 reliance weak, TBD yes good
programming/prototyping support
reliance		 strong yes good
procurement/acquisition - strong, BOR yes good
strategic plans preparation alignment
(alliance)		 weak none medium
IS planning and design alignment medium, TBD weak
(different tools)		  
Maintenance reliance none medium
(interactive database apllication)		 improving
project management alignment strong yes (ProjectOffice) good
quality assurance
(technical supervision including acceptance testing and validation)		 support,
alignment		 strong PROCESSUS
PRO+		 improving from medium

TBD – to be declared (method preparation/selection in progress).

BOR – based on legislative regulation.
 
 

Fig. 1. High-level report from PRO+ Tool.

Fig. 2. The logical object model of the PROCESSUS Tool.

The results of the final decision (assessment of the product), are given in the form of a vector, containing multiple parameters, with normalised values (from 0 to 1). Each parameters (functionality, reliability, usability, efficiency, maintainability, portability and completeness of the documentation) is kept on its own ordinary axis, in the form of a Kiviat diagram (Figure 1).

Conclusions

This paper has described the various practices and problems involved in a large governmental institution, and how to solve them. Client-vendor relationships in outsourced projects, are often limited to a project initiating and a product delivery.

Initially we listed the various types of outsourcing, and declared the degree of client supervision in various projects. A practice based on mutual project management and methodology-approach to the quality assurance. With three years of experience, practicing this approach, we can show an essential progress in the controllability of outsourced projects. This has resulted in a more reliable risk prevention, better product quality, resource optimisation and not least of all, in better communication and co-operation with vendors.

References

[Boehm, 1989] B.W. Boehm, Software Risk Management, IEEE Computer Society, 1989.

[Györkös, 1996] J. Györkös, I. Rozman, R. Vajde-Horvat, M. Hericko, "Quality Management in Software Development Process: An Empirical Model", Proceedings of IEEE International Engineering Management Conference, IEEE Press, 1996.

[Paulk, 1995] M.C. Paulk, B. Curtis, M.B. Chrissis, and C.V. Weber, Capability Maturity Model for Software, Version 1.1, Software Engineering Institute, CMU/SEI-93-TR-24, February 1995.

[Slaughter, 1996] S. Slaughter, S. Ang, Employment Outsourcing in Information Systems, Communications of ACM, Vol. 39, No. 7, 1996.

[Rozman, 1997] I. Rozman, R. Vajde-Horvat, J. Györkös, M. Hericko, "PROCESSUS - Integration of SEI CMM and ISO quality models", Software Quality Journal, Volume 6 Number 1 March 1997, ISSN 0963-9314, Chapman & Hall, 1997.

[Nam, 1996] K. Nam, S. Rajagopalan, R. Rao, A Chaudhury, "A Two-Level Investigation of Information Systems Outsourcing", Communications of the ACM, Vol. 39, No. 7, 1996.

[Paquin, 1996] J.P. Paquin, J. Coulliard, P. Paquin, D. Godcharles, "Earned Quality: Improving Project Control", Proceedings of IEMC'96, IEEE Press, 1996.

[PRINCE, 1995] Management of Programme Risk, CCTA The UK Central Computer and Telecommunications Agency, London, 1995.

[PRINCE, 1996] PRINCE 2, Project Management for Business, The Stationery Office Publication Centre, London, 1996.

Partners in EuroSPI

Editors
ISCN LTD, ISCN GesmbH, Schieszstattgasse 4/24, 8010 Graz, and Coordination Office, Florence House, 1 Florence Villas, Bray, Ireland, office@iscn.at, office@iscn.com, office@iscn.ie, Editing Done: 19.7.2002