Software process improvement using CASE: lessons from the front-line
David Wastell University of Manchester, UK
Chris Williams & Mike Willetts Salford City Council
Karlheinz Kautz Norwegian Computing Centre, Oslo, Norway.
Peter Kawalek Warwick Business School, UK.
Tom McMaster University of Salford, UK.
At the outset of the project, the software development group in ITSD consisted of around 28 software professionals organized in three teams under a single manager. They were responsible for the development and maintenance of software systems for user departments throughout the Council, the principal users being the Directorates of Education, Housing, Social Services, and the Treasury. Although the department had a generally good reputation amongst its user community, there were pockets of dissatisfaction and ITSD had a mixed reputation for performance. The quality of software produced was generally acceptable. However, timeliness and budgeting targets were regularly exceeded and the customer departments felt that ITSD failed to provide sufficient information with regard to progress and project budgets. Interestingly, where strict deadlines were imposed, for example through legislation, those projects were delivered on time.
It was hoped that the use of CASE within a systematic methodology would achieve tangible benefits in terms of both improved developer productivity and enhanced software quality (especially in terms of user satisfaction). Productivity would be enhanced by two primary means: through the standardisation of working methods and through the technical facilities provided by CASE, e.g. integrated analysis and design tools, automatic code generation, a central code repository enabling more re-use. Improvements in software quality were sought in terms of both its technical dimensions (maintainability, cost of ownership etc.) and in terms of a better fit with user needs. It was hoped that CASE would directly improve technical quality and would indirectly support better business alignment by enabling higher levels of user participation in the development process (primarily to be achieved through the ability to prototype rapidly). The project aspired to introduce a new software paradigm, described as the "Total Team" approach. It was hoped that the above benefits would also translate into increased developer job satisfaction resulting in staff retention and improved customer satisfaction resulting in repeat business.
In a nutshell, the original CAPELLA plan was structured in two phases. It was recognised that the introduction of CASE and a new methodology represented a major change. Driving the implementation strategy was a concern to develop internal capability, not to become dependent on outside experts or consultants. This approach would also maximise feelings of ownership and minimise internal resistance. Accordingly, the first phase of the project (initial 6 months) aimed at creating such internal capability in the form of a Centre of Excellence (CoE). Key skills would be developed by using CASE on a pilot project. Once the skills were in place, CoE personnel would then play the role of internal consultants assisting in the deployment of CASE (and new working methods) throughout the rest of the department (months 6 to 18).
An unexpurgated account of the project in terms of its main events now follows. The account is broken into 6 month segments (i.e. semesters). Dates are approximate.
Although an alternative project was identified concerned with the financial management of regeneration projects ( to be known as PROJ_1), both momentum and enthusiasm had been lost. The deployment of CASE on a major project was critical to the whole strategy of CAPELLA. It would mean that many developers would ultimately be involved in its use; seeing CASE deployed on a flagship project was also of obvious symbolic value in showing how central CASE (and CAPELLA) was to ITSD’s long-term strategy. Use by a small group on a small project suggested the opposite. However, this should not be seen as a tactical error so much as a genuine change in the business environment. The loss of the Housing project was simply a reflection of a more general trend affecting ITSD (and internal IT/IS departments in general), namely a move away from in-house bespoke development to outsourcing and the use of packages. That CASE was coming to be seen as marginal was thus accurately reflecting the wider business realities.
Summarising this period, the main positive outcome was that a small group of staff had been trained in the use of D2K, and that the CoE now existed in the form of this group. Little work on methodology had been initiated, although some preparatory work on metrics had been carried out (for evaluating quality and productivity gains). The main user departments had been interviewed and an internal investigation of the use of function point analysis (Fenton, 1991) had been undertaken (with generally positive findings). Although a CoE existed, it is fair to say that it (and CAPELLA in general) lacked a high profile within the department as a whole, with most developers seeing it as having tangential relevance to their work. The primary reasons included: the loss of the Housing project and the general lack of appropriate development projects; lack of strong project leadership; low morale amongst developers leading to a general weariness in respect of any innovation.
Towards the end of this semester, HSD left his position at Salford and there was a major re-organisation within ITSD. The Customer Services and Software Development business groups were merged with the Customer Services manager (HDCS) taking on the head-ship of the newly formed "Development and Customer Services" business group. HDCS had led the original bid on the Council’s side. Availability of staff resources was obviously affected during this re-organisation, and a key member of staff also left the CoE for a position in the private sector.
Following Christmas, a concerted attempt was made to define the new methodology, drawing on best practice in past projects. A structured methodology focused on the use of CASE, with 14 stages covering the whole life cycle, was mapped out as a joint exercise by a team comprising one of the researchers, a senior practitioner (PR2) and PR1. Two of the stages were defined in detail. This work was however curtailed following a visit by PR1 and another member of the CoE (PR3) to a D2K workshop where they were introduced to CDM, Oracle’s proprietary methodology associated with D2K. This appeared to hold much promise as a potential framework. It was defined in detail, appeared to conform well to the kinds of development projects handled by ITSD, and was geared to the application of D2K. "Why invent a methodology of our own" was a persuasive argument which re-directed effort to the evaluation of CDM. The decision was made to apply it to PROJ_2 and retrospectively to another project, the ultimate aim being to customise it to ITSD’s working practices. PR1 was to take the lead in this.
This work continued over the latter part of this semester. Some progress was made but ultimately the decision was made to discontinue this line of work. PR1 showed increasing disillusionment with CDM and with CAPELLA in general. CDM appeared to consist of no more than a set of Word templates for holding documentation; in detail, it did not conform well to the standards in place within ITSD. It was also unclear as to whether there was any real need for CDM, given both the dearth of development projects and the marginal position of CASE. The sense of an impending crisis was becoming strong and in late spring a series of meetings was held which were to betoken a major change in the direction of the project.
Early summer saw a major restructuring of the project. A senior member of ITSD was appointed to lead the project. Two students were recruited from Manchester University to carry forward the bulk of the technical work, and a further researcher was engaged. Three main lines of work were mapped out: a codification of the methodological work that had been done, the development of a set of software metrics using the GQM approach (Basili, 1995), and an assessment of the lessons learned from the experience of implementing CASE. ESSI were informed of the status of the project and a request was submitted for a 6 month extension on the basis of past problems, the proposed restructuring and the future benefits. This was granted.
Work proceeded on metrics and methodology over the summer, and resulted in several reports, which formed the inputs for a 2 day workshop in early October at which the objectives and work programme for the remainder of the project were defined. Two reasons for the desultory progress made in the project were identified. The decline in in-house development was re-affirmed as a major inhibitory factor; this had rendered both CASE and the work on a development methodology of increasingly marginal business significance. A second factor also received extensive discussion, namely the low level of software engineering discipline that appeared to prevail in ITS. This had been highlighted as a result of a CMM assessment (Humphrey, 1995) performed over the summer which showed ITSD to be at level 1, i.e. chaotic. More tellingly, the lack of formality had come to light as a result of an attempt to introduce some basic mechanisms for project control; an analysis of current commitments had revealed that much of the department’s work was unofficial (up to 50%!) in the sense that no record was present in the formal order book.
Given these factors, the conclusion of the workshop was to focus the remainder of the project on process improvement areas that were desirable and attainable given the working practices and culture that prevailed in the department. Work on a monolithic development methodology was no longer regarded as appropriate and it was resolved to re-focus the methodological work on key practices. This was felt to represent a more flexible approach allowing the department to improve its performance in discrete areas as part of an on-going process of continuous improvement attuned to the exigencies of its business environment. Each year, one or more practices would be targeted for focused effort.
Two such practices were identified for the remainder of CAPELLA. The first, and the most important in terms of effort, was a metrication initiative aimed at putting in place a simple set of metrics that would provide a rudimentary degree of project monitoring and customer feedback. Part of the inspiration for this was to enable the efficacy of CASE to be evaluated (to the limited degree that it had been deployed). The second motivation was that this would be a significant move towards the creation of an effective project management infrastructure. Although some project control mechanisms were in place, we have seen that they needed strengthening and formalising. In practice many projects were initiated without plans or indeed formal approval, and where plans existed there were wide variations in the degree to which projects were monitored or controlled against those plans.
The second key practice was peer review. Two benchmarking studies recently carried out by one of the researchers had identified this as a practice which had been successfully adopted by two other level 1 organizations, and which had led to real improvements in software quality.
At this point, a further significant development will be mentioned which had occurred in the early part of the semester, namely the inauguration of a Standards and Methods Group (SMG) within ITSD. The remit of SMG was to identify existing standards and methods, to develop new ones where appropriate, and to promulgate their use. Membership of SMG was open to any member of ITSD; participation in its work was voluntary. CAPELLA and SMG were clearly related in the sense that both initiatives were aimed at improving software practices and a formal link was established. It was resolved that CAPELLA activities would be progressed in concert with this group, and that we would draw on any relevant standards work that was being done by SMG.
A presentation of the results of the workshop was made to SMG. Several staff members volunteered to work on both the metrics and the peer review strands. The two students were retained as ITSD staff (RA1 and RA2); their primary remit was to carry through the metrics thrust of the project. One of the academic researchers took lead responsibility for progressing the peer review initiative.
Major problems were encountered immediately in that only one of the 6 projects was actually live and running, namely PROJ_2. The new CASE project had been cancelled, the two package projects were in abeyance, and the other two projects were finished. Collecting the proposed set of metrics on PROJ_2 also turned out to be infeasible, for a variety of reasons. Some metrics were seen as "too complicated" (i.e. it appeared impossible to devise a standard method for assessing the complexity of design products such as DFDs given the wide variations in way that they were produced); other metrics were abandoned on the grounds that no relevant and reliable information could be found in the existing project documentation (e.g. type of deliverable, delivery dates etc.).
In the light of these problems, the original set of developer metrics was scaled back to a very limited subset, effectively providing timesheet information recording how much effort was expended on different activities (e.g. logical design, data conversion) for a given project. A detailed booklet was issued to the PROJ_2 team members (3 in number) indicating how the timesheets were to be used, and the data collection exercise was then initiated. The trial itself was disappointing in that very little work on PROJ_2 occurred over the course of the experiment, and the team members showed decreasing willingness to complete the sheets. Only 5 timesheets were ever returned. However, the experiment was invaluable methodologically in that it stimulated the development of a high level lifecycle model for CASE development, in which the development process was divided into 15 tasks (e.g. produce logical design) grouped in 8 stages. By the end of the pilot, this Process Model had been through a series of refinements and was seen to conform well to the work the developers did.
Overall, the trial was a useful learning experience, which had demonstrated the general feasibility of collecting timesheet data in real time, providing that an accurate process model of the type of project was defined. The decision was made to develop the approach further and to implement it across the whole of ITSD for a one month trial period. The RAs devised high level process models for each of the 10 types of project undertaken by the department (CASE development, non-CASE, package work, Y2K testing, database upgrades etc.) drawing on such standards that existed (e.g. a standard for package development had been developed by SMG).
In the end, this experiment ran for 8 weeks. Data was successfully collected for a high proportion of ITSD personnel, peaking at 20 individuals over the first 4 weeks before tailing off to 10 for the final 4 weeks. Exploratory analyses have shown that usable information was gleaned (e.g. accurate pie charts showing the distribution of time and effort across the different types of project work). The validity of the various process models has also been examined by checking if a valid task type has been specified against each timesheet entry. For many processes, validity was high: for CASE development, for instance, 85% of entries were valid, for non-CASE, 87%. The figures for other major processes such as package development (59%) and Y2K work (65%) were somewhat lower, indicating that further refinement of the process models and standardization of work practices is required in these areas. The RAs also investigated a number of other issues of interest, such as variations in task length across teams.
The resulting questionnaire contained a number of sections, grouped in two main areas: past experience of software development (users were asked to rate how involved had they been in the development of the system they used, how much control they had had, how responsive were ITSD etc.) and their levels of satisfaction with the resultant system (the support given, accuracy of the data, informational content, format, ease of use and flexibility). The motivation for the first set of question stemmed from our conviction that user participation was the key to successful software development (the Total Team approach) and that CASE was a means to achieve this.
Customer data was then collected by distributing the questionnaire to the users of a sample of extant systems. It had been hoped to carry out a comparison of CASE vs. non CASE systems; however, no CASE systems were operational at the time of the survey. Hence it was targeted on non-CASE systems only, 3 in all operated by two departments (Corporate Services and Social Services). Nonetheless, the exercise was seen to be of value in that it would provide a baseline for the future and that it would provide feedback on feasibility and validity of the methodology.
Questionnaires were distributed to the identified contact person for each of the systems; they were asked to pass them on to staff who were users of the system and/or had been involved in its development. Questionnaires were returned from only one of the departments; 27 responses were received from an estimated 70 distributed. Regarding the first section of the questionnaire, respondents were overwhelmingly positive about their experience: e.g. 75% of responses indicated an appropriate level of involvement and control, 85% indicated that user/developer relations and communication had been good. However, the number of respondents here was relatively small. The systems themselves were judged to be satisfactory in most respects: support (86% positive responses), accuracy against specification (100% positive), content (99%), output format (92%). For ease of use (67%) and flexibility (50%) the proportion of positive evaluations was somewhat lower.
Whilst these results are generally positive, and indicate a highly suggestive relationship between participation in development and project success, they reflect attitudes to only two systems in one department. They broadly confirm the findings of our early interview work that this department believed strongly in a user-led approach, that they committed resources appropriately, and that they were very satisfied with the systems that they currently use.
Several members of the SMG worked alongside one of the researchers in an effort to define a standard procedure for peer review and to evaluated its usefulness. Two experimental reviews were carried out, which were seen to be valuable, and a standard was defined which outlined a methodology for peer review (preparatory work, who should attend, key roles, the review format, guidance for reviewers, reporting and feedback). Quality criteria for a set of high priority topics for peer review were also outlined, including feasibility studies, project definition documents, and design documents.
The Total Team approach reflected our philosophical commitment that user participation is the key to achieving high quality software systems, and the interview/survey data confirmed the importance of a user led approach. Our original plan had been to develop a semi-structured development methodology exploiting the features of CASE (e.g. prototyping) to promote more intensive user involvement. In practice, achievements in this area have been limited, given the changing priorities over the course of the work. A methodology was defined, but only at a high level. The move away from a monolithic life-cycle methodology towards key practices reflected the need for a flexible, modular approach to improving working methods. Establishing the key practice approach is an important result; it provides ITSD with a general framework for continuous process improvement that will stand them in good stead in the face of a highly dynamic and increasingly customer-oriented business environment. The modularity of the framework is important as it enables incremental changes to be made that are attuned to the availability of resources and the prevailing business priorities.
The Peer Review initiative exemplifies the Key Practice approach. The introduction of this practice was felt to be an important step towards improving software quality, and the results of the experiment were promising. Peer review was seen to be a useful tool and a draft standard is now in place, including document templates. There is a clear intention to proceed with its implementation, following further refinement of the methodology. Questions being debated include: what products to apply it to, when should it be done (mid or end of phase), how should it be followed up? As a first step, it has been proposed that all projects must include at least one peer review built into the project plan.
Both metrics initiatives led to positive results. The timesheet experiment was a success in that data was collected over an short but not insignificant period, and that the information generated appears to be both valid and of intrinsic interest. However, the experiment also indicated that staff compliance will be problematic and that more detailed consideration needs to be given to the content of the information to be collected from both management and operational perspectives. The experiment thus demonstrated the feasibility of the methodology; it provided valuable experience and generated a set of useful document templates and process descriptions. A system for recording effort against projects in real-time is important for resource management and it is likely that, after further refinement, the timesheet mechanism will form an important component of ITSD’s embryonic project management system (see below). Timesheets were also felt to be beneficial as a "self-productivity" check.
The results of the customer metrics study were also positive in that the metrics appear to be valid and meaningful, showing that customer satisfaction is measurable not only in relation to the systems developed but that the development process is capable of metrication too. Valuable feedback on the methodology was obtained as a result of the experiment, indicating it to be generally sound, although there are problems that remain to be addressed: e.g. the absence of responses from one department was a major flaw that needs further investigation, with the methodology to be improved as a result. As a result of the experiment there is a clear intention to refine and implement the customer metrics as a routine feedback mechanism, and a further survey is planned for next year.
The October workshop identified an urgent need for improved mechanisms for project management and CAPELLA has certainly instigated important work in this area. Apart from the timesheet experiment, work on the creation of a standard for project management has commenced over the last semester, under the aegis of CAPELLA. Three concrete achievements may be noted:
As a final achievement, the inauguration of the SMG should be noted. CAPELLA in part inspired this, in the sense that SMG’s creation reflected an awareness of the need for self-reflection and process improvement that CAPELLA has helped to engender. In terms of its work, the liaison with CAPELLA has been of reciprocal benefit; standards generated by SMG have been used within CAPELLA and the results of our work have been reported back to SMG. In this sense, CAPELLA has strengthened the role of SMG. It is felt that SMG has proved itself as an effective forum, although it has declined in vigour over the last six months owing to the voluntary basis on which its technical work was done; in the future, SMG projects will be treated on the same basis as any other project and resourced fully and formally.
The first part of this report presented a detailed history of the project. The reason for recounting such a blow-by-blow narrative (rather than the usual bland and abstract account) is to provide a body of data to reflect upon regarding the lessons to be learned from the project for the introduction and management of technological change.
Arguably the most important lesson is the importance of strong alignment of new technology with the short-term demands on the business and its long-term goals. Although both are important, our experience shows that immediate exigencies tend to take precedence over future aspirations in terms of their claim on attention and resources. However important CASE and the Total Team approach might have been in the long term, the limited nature of our achievements in these areas in large part reflects the lack of relevance of CASE and the methodology to the prevailing demands on ITSD. Our experience has emphasised the high degree of dynamism in today’s business environment and that vigilance and constant effort are required if change initiatives (such as PIEs) are to remain congruent with changes in the internal and external environment. We have seen that gaps can easily open up, even over relatively short time-scales, and that delays in addressing these can seriously jeopardise the improvement initiative.
Our results underline the truism that complex technologies require a greater investment of implementation effort in terms of training, changing working practices etc than simple innovations. The benefits at an organisational and individual level will take longer to realise and the short-term costs will be greater and more inhibiting. Not only is more effort required but the learning and dislocation entailed in the immediate term are likely to engender decrements rather than improvements in performance. This is a serious problem for complex technologies such as CASE; the business risks of deploying these innovations on business-critical projects is very inhibiting. The lack of adoption of CASE is only partly explicable by the paucity of development projects. It is arguable that it could have been more widely used than it was, and we may attribute this to the investment of effort that would have been required, in terms of training and in the standardization of work practices in an organisation characterised by highly informal methods. These concerns have almost certainly held back its wider adoption, and the question is a moot one of the degree to which it would have been deployed on the original baseline, given the risks involved. These considerations indicate that incremental approaches to technological change are more likely to succeed for complex innovations. This was reflected in our original plans to roll-out CASE on a team by team basis. It is also reflected in the positive reaction to the Key Practices framework.
Leadership in terms of both vision and implementation has also been shown to be crucial to success. Arguably, leadership was lacking in the first semester of the project and this was a cause of some of the problems encountered. Implementation is critical, and our experience has shown how problematic this can be and has given important insights into the critical success factors. Ownership is at the heart of this issue; innovations will be adopted more readily if people personally identify with them. This lay behind our original decision to make use of internal personnel. Although external resources can give momentum to experimental initiatives (as we saw in the last semester) there is a danger that the these efforts can be come marginalised because they are executed by outsiders. It is significant that, although there are plans to implement metrics, the only initiatives from the last phase that are currently institutionalised are the PDD and the PRR: the implementation of both these documents was championed and led internally.
Let us examine further the success of these two innovations in terms of our previous analysis as the contrast with CASE is revealing. In terms of relevance, the benefits are clear: ITSD will find it more and more difficult to survive in an environment increasingly dominated by commercial imperatives without stronger control systems. The PDD and PRR represent important components of a nascent project management system that is crucial to survival. Relatively simple technology is being used (simple paper documents) and, as we have said, leadership is internal and at a senior level. The attention given to implementation has also been crucial and shows the importance of monitoring and accountability. Articulating a vision for change is not enough; implementation must be followed up rigorously through management structures with staff held accountable, especially middle management. PDDs and PRRs are now mandatory requirements in ITSD and team leaders are held accountable for their production. Despite early problems of incomplete recording, there is a steadily improving trend in the quality and timeliness of these documents.
[2] Checkland, P., Systems thinking: systems practice. Wiley, Chichester, 1981.
[3] Doll, W. and Torzadeh, G., The measurement of end-user computing satisfaction, MIS Quarterly, 259-274, 1988.
[4] Fenton, N., Software metrics: a rigorous approach. Chapman-Hall, 1991.
[5] Humphrey, W.S. A Discipline for software engineering. Addison-Wesley, Reading, MA, 1995.