EuroSPI 2000

Practical and innovation based software process improvement to prepare for the new millenium.

European Software Process Improvement SPI and Small Businesses

Category Index

Rated Newspaper Supported by EU Project

EVALUATION OF SMALL SOFTWARE COMPANIES FOR LARGE CONTRACTS

Elif Demirors
The Bilgi Group Software Research, Education and Consultancy
Ankara Turkey
demirors@bg.com.tr,

Onur Demirors
Informatics Institute, Middle East Technical University
Ankara Turkey
demirors@metu.edu.tr

Introduction

The global software industry includes a considerable number of small companies. In Turkey, for example, 69% of software companies have 20 or less employees [1]. Similar observations are published for other European countries. Somewhat contrary to this, in large government contracts it is usually the tradition to call in large, mature software developing organizations in order to ensure the completion of the contract and a high quality product. However in some cases large organizations may not be interested in the domain of contract, or the government agency would like to involve small organizations to increase competition. Although generally away from standards, and improvement frameworks, small companies have a potential to develop quality software, especially in their specific domain of expertise. Also, if such a company has a group of very talented, open-minded developers that are ready to change given the chance, then it will have a chance to succeed in contracts much higher than their revenues.

When an agency is faced to select the best-fit software developer among a number of small-sized companies, it fails to find an applicable evaluation strategy. The widely used assessment methodologies, SPICE and CMM, work well in large organizations. However little data has been gathered on the quality of their feedback when applied to small organizations with practices far away from what CMM or SPICE expect to find.

We have been involved in a project that was initiated by the Ministry of National Education (MONE) to define a set of pre-qualification criteria that will evaluate the capability of software development companies. Specific domain of application was instructional software which eliminated most of the large software development companies operating in defence, and finance markets. The experts in instructional software was, and still are, small businesses which involve both software developers and instructional domain experts (mostly teachers). Although the agency was willing to work with the small companies in contracts approximately 1.2M USD each, they wanted to eliminate the businesses with higher chances of failure.

The project took four weeks, and involved software quality as well as instructional technology experts. A set of criteria was formed to be used by the Agency to establish a short list of potential software contractors. Several issues were addressed for defining the criteria: regulations and early experiences of the Agency; regulations and recommendations of the World Bank (the project was financed by the World Bank); the capability of developing quality instructional software; and related best practices and standards of global software industry. The process followed to define the criteria is in accordance with the ISO/IEC 14598-1:1999 Information Technology-Software Product Evaluation.

Methodology

Several issues are addressed for defining a set of pre-qualification criteria that will evaluate the capability of software development companies:

The regulations, recommendations and early experiences of the MONE- Since the MONE has been involved in acquisition of instructional software packages for more than a year, and has already completed a bidding cycle for off-the-shelf packages, a large experience base has been formed regarding to the company profiles, acquisition process, and evaluation process. This experience base has been utilized in defining the pre-qualification criteria.
The regulations and recommendations of the World Bank- The procurement of customized instructional software development is required to follow the World Bank’s acquisition policies. The strategies of the World Bank for pre-qualification of bidders and the regulations on the bidding process have been considered. The pre-qualification criteria support the strategies and policies of the World Bank.
The capability of developing quality instructional software- Instructional software development requires software engineering expertise, subject matter expertise, and instructional design expertise. The capability of a company to operate in this domain depends on how well these different expertise areas are represented, and how well individuals from different backgrounds are harmonized. Suggestions of the industry standards as well as the World Bank policies on the infrastructure required for a successful instructional software development project are taken into account in defining the pre-qualificaiton criteria.
The related best practices of the global software industry- Software engineering discipline has been working on standardization of related practices independent of the application domain. Both the International Organization for Standardization (ISO) and the Institute of Electrical and Electronics Engineers (IEEE) published a range of standards addressing acquisition and evaluation issues in software development. Additionally, various recommended practices as well as interpretations have been published by these organizations and by other agencies. The pre-qualificaiton criteria defined are based on the requirements and suggestions of such publications.

The process followed to define the pre-qualification criteria is in accordance with ISO/IEC 14598-1:1999 Information Technology- Software Product Evaluation. First, the purpose of evaluation was clarified, and then, the types of products that need to be evaluated and the quality model that defines the relevant quality characteristics are identified. The process then focused on selecting the metrics for evaluation and establishing the rating levels for these metrics. The criteria for assessment were also defined. The remaining phases of ISO 14598 evaluation process are out of the scope of this study.

As mentioned above several international standards and practices have been utilized in this study. These are:

(a) ISO/IEC 12207:1995 : Standard for Information Technology – Software Life Cycle Processes.
(b) ISO/IEC 14598-1:1999 Information technology - Software product evaluation -- Part 1: General overview.
(c) ISO/IEC DIS 14598-4 Information technology -- Software product evaluation -- Part 4: Process for acquirers.
(d) IEEE Std 1062:1998 : IEEE Recommended Practice for Software Acquisition.
(e) ISO/IEC TR 15504: 1998 Information Technology - Software Process Assessment.
(f) IEEE Std 1012:1998 : IEEE Standard for Software Verification and Validation.

Organization

The measures established for pre-qualification of companies are classified under three equally important criteria: organizational capability criteria, product capability criteria, and process capability criteria. For each one of these groups, a set of quality characteristics are defined. The quality characteristics are composed of a set of attributes that are measured in terms of the indicators listed.

A company’s total score is calculated by summing up the scores that the company earns from each of the three capability criteria. The score of each criteria is obtained by totaling the scores gained from each attribute as defined by the attribute’s measure in terms of the indicator values. In other words, the evaluation will determine whether a specific indicator is valid for that company and if so, the corresponding point value will be added to the company’s score according to the measurement policy defined. A company is also required to earn a score above the threshold defined for each criteria for pre-qualification.

The hierarchical organization of the pre-qualification criteria is depicted in Figure 1.

Capability Criteria

Organizational Capability Criteria

The organizational capability criteria characterizes the adequacy of organizational infrastructure in supporting the development team during a customized instructional software development contract. For a software development project to be successful a solid infrastructure should exist that includes experienced personnel, mature business practices, adequate financial resources and appropriate development environment. The organizational criteria is divided into three characteristics as defined below:

Company Profile
Staffing
Development Environment

The weight of the organizational criteria is one third of the overall score, that is 300 out of 900. There is a threshold of 150 points over 300 for pre-qualification.

Product Capability Criteria

The product capability criteria evaluate the quality of an existing product to determine a company’s past performance in software development which is an important indicator of its future performance. For evaluation, the bidder is expected to submit one of its products that best satisfies the requirements listed. If there is only a prototype of a future product, then the bidder may also submit this prototype for evaluation. The size of a product or prototype is required to be no greater than the size of a typical instructional software targeting a specific course for a specific grade (i.e. Mathematics for the 6th grade).

The evaluation is performed on the product itself and on the intermediate documents produced during the development process, namely software requirements specification, design documents, and test documentation. The success of the product in the market will also be evaluated if the company will be able to submit appropriate documents.

Product capability criteria is divided into three characteristics as defined below:

Product/Prototype Quality
Quality of Intermediate Products
Customer Satisfaction

The weight of the product-in-use evaluation is one third of the overall, having a score of 300. There is a threshold for this part of evaluation, that is 150 over 300, for pre-qualification.

Process Capability Criteria

A company’s software process capability determines its ability of producing quality products since product quality depends on the quality of the processes enacted during development. The process capability criteria is generally consistent with the Software Engineering Institute's (SEI) Software Capability Evaluation methodology which evaluates process maturity with respect to the framework defined by the Capability Maturity Model (CMM).

The software process capability is evaluated with a Maturity Questionnaire. The Maturity Questionnaire is based on SEI’s maturity questionnair and is being used as the formal data collection device within the Software Capability Evaluation methodology. Instead of requesting a full Level 2 or Level 3 compliance from the bidders, a number of the key process areas are selected based on the requirements of the ministry and on the nature of instructional software development. The key process areas grouped under Basic Management Processes include most of the Level 2 process areas except "Requirements Management" (classified as Extended Process) and " Subcontract Management" (eliminated completely from the evaluation). On the other hand, Extended Processes include the key process areas from Level 3 as well as "Requirements Management". These processes are not considered mandatory like Basic Management Processes but rather taken as indicators of more mature software development process. The ministry reserves the right to perform and/or request on-site independent assessment. If assessment is to be performed during pre-qualification process than all acceptable bidders will be evaluated. However the ministry may choose to perform and/or request an independent assessment within the post-qualification framework in which only prequalified bidders will be evaluated. Any inconsistency with the answers provided here will result the disqualification of the bidder from submitting any bid to this and any related calls.

The process capability criteria is divided into two quality characteristics:

Conformity to Basic Management Processes

Conformity to Extended Processes

The weight of the process capability criteria is one third of the overall score, that is 300 out of 900. At least 75 points need to be earned from the key process areas of the "Basic Management Processes" quality characteristic for pre-qualification. There is no threshold for the second characteristic, however any points earned from extended management processes will be totaled to the score of process capability criteria.

Lessons Learned

The project helped the whole team to merge two different worlds of instructional domain and software engineering for the benefit of small organizations which work in between these domains. The criteria defined provides a general guidance on the expectations of the MONE from the suppliers, as a result will also be utilized by the organizations as a guidance for improvement. Our experience and issues that we have learned from the project are summarized below:

Supplier processes should be the focus but product, people and organizational aspects should also be considered. The organization and product criteria enabled evaluation of the companies previous assets in terms of people, customers and development environment.
Mature supplier processes best work with mature acquisition processes but acquirers might not be ready for a complete process improvement initiative. As a result suppliers need to adapt their process for immature acquirers. Using objective pre-qualification criteria enables acquirers to minimize risks at early phases without substantial investments.
International standards should be adopted whenever possible but enforced standards should not prevent organizations to apply different methodologies and techniques. The utilization of international standards enables wide range of organizations to compete. It also enables faster dissemination of knowledge and higher possibility of creating a nationwide acceptance. However many application domains are distinct. For example, instructional software development is generally considered as an art, and engineering of products is often neglected. Work product and process standards should be flexible enough to cover such domains.
Utilization of suitable pre-qualification criteria is most beneficial if it is combined with an explicitly defined development process and product expectations as well as a well defined feedback mechanism throughout the contract work.

References

[1] TURSOFT, The Turkish Software Industry (in Turkish), 1996.

Fig. 1. Pre-Qualification Criteria

Partners in EuroSPI

Editors

ISCN LTD, ISCN GesmbH, Schieszstattgasse 4/24, 8010 Graz, and Coordination Office, Florence House, 1 Florence Villas, Bray, Ireland, office@iscn.at, office@iscn.com, office@iscn.ie, Editing Done: 19.7.2002