A Total Improvement Strategy as a Basis for Software Process Improvement
Rolf H. Westgaard Solveig Gustad Kongsberg Ericsson Communications ANS P.O. Box 87 N-1375 BILLINGSTAD Norway
Introduction
Kongsberg Ericsson is working with improvements at several company levels, driven from management down and from the "floor" and up.
This paper describes:
The reason for our participation is a desire to improve our system for continuous improvement of the development processes - starting with software. We were looking for a system, which could give us objective data as a basis for improvement. Before joining SPIQ our actions to improve the development processes had been mostly based on single cases, without any statistics as a basis for priority of actions.
Our Values as a Guide
We have decided that the following values are of utmost importance for our success:
The way we obtain our results is simplified in the Business Process Model below.
Strategic Plan Structure = Organisation Structure = Business Structure:
Fig. KEC_RHW 1: The KEC Business Process Model
The company has established a Strategic Planning Wheel with a one-year cycle. The wheel consists of the following sequences:
Fig. KEC_RHW 2: The Strategic Planning Wheel
The Strategic Improvements Plan
The Strategic Improvements Plan is based on our business processes and the EFQM business excellence model. Our version of the EFQM business excellence model has added "Opportunities" to it, and we say, "We are using our market, product and technological OPPORTUNITIES by means of ENABLERS to obtain RESULTS". During the Strategic Plan Process we decide the areas within "enablers" that need improving based on long term and short term results. The conclusions are documented in improvement plans starting at company level and broken down into lower organisational levels. The plans are finally the basis for each person’s individual contribution to improvements, agreed during the yearly appraisal interview.
OPPORTUNITIES ENABLERS RESULTS
Fig. KEC_RHW 3: Modified EFQM Business Excellence Model
Follow-up of Improvement Plans
The improvement plans at the different levels contain:
Opportunities: Market & Product meetings every two weeks
Enablers: Value meetings every month
Results: Result meetings every month
We have all experienced that a well-structured plan and good intentions do not necessarily guarantee results. Therefore, we have defined some critical factors to secure success:
The most comprehensive processes in our company are the ones covering product development. Improving efficiency and quality of these processes is therefore a high priority.
The following part of the paper describes our efforts to establish a system for improvement of our development processes, starting with the software process and our participation in SPIQ with focus on software of the MRR project.
Software process improvement based on measurements
This case describes the planning, implementation and results of a software process improvement project. The company has long experience of collecting data, but not to assemble the data and use them to extract statistics and get experience from them.
What have we done?
The project
The MRR (Multi Role Radio) project is a large electronic project of 1,9 Billion NOK (230 Million EUR). The product is a portable radio for military use. The project started in 1989 and is a co-operation between Kongsberg Ericsson and Thomson CSF Norcom. The pilot project described in this paper involves only the Kongsberg Ericsson part of the MRR project. MRR was delivered in 1996 as a pre-production model with reduced functionality. After that all HW and almost all SW has been redesigned.
Because the MRR project has lasted for almost ten years, there have been some turnover in labour. The development process is an ordinary waterfall model and this model was the starting point for improvements.
The starting scenario
Most of the software was finished in the start of the SPIQ project, only testing remained. That meant that almost all the errors were already done.
To correct all errors, there was established a "system acute" that handles all error reports every day at 12 a.m. From this meeting the error report is decided whether to correct or not and the report will be sent to the responsible person with some action (usually correct a document or code). In this way we know that every error detected in test will be handled. But there is no analysis of the errors to prevent similar errors in the future. Here is where the SPIQ project comes in.
The starting point is our written development process (all procedures, checklists, rules, templates, the tools we use, language, the methods and so on). If we don’t change anything of this, we will not be better in the future. So we had to change something, but what? And when would we know that a change had a positive effect?
Work done
To answer the questions above, we joined the measurement and experience-data-group in SPIQ. We started with a GQM workshop for all the software developers. The result of this workshop was a GQM abstraction sheet with one goal, eight questions and 18 metrics. We need to answer the five questions in order to reach the goal, and we need to collect the 18 metrics in order to answer the questions. The abstraction sheet also contains a plan of measurements (what should we measure, how and who etc.).
GQM example
The main goal was to achieve reduction in the number of serious errors found in integration test and system test by 50%.
The GQM goal then became to analyse the development process in order to reduce the number of serious errors introduced before FAT (Factory Acceptance Test) seen from the software group in the environment of the MRR project.
GQM abstraction sheet:
Q1:
Fig. KEC_RHW 4: Baseline Hypothesis
Measurement plan:
Metrics:
Results from two of totally five focus questions
About every six weeks, we analyse the error reports and define actions. The actions are mostly of three kinds:
In which phase are the errors introduced? Fig. KEC_RHW 5: Errors distributed on phases
The error is per definition introduced in the earliest document that has to be corrected or changed. If a software developer has misunderstood the specification, then it is the specification that is wrong, because we have a review rule that says that a document shall not contain possibilities for misunderstanding.
The figure above shows that no errors have been introduced in the system test phase. This is because the system test has not started at the time of writing. The integration test has not finished either, so we have to wait until the end of the system test phase until we get the exact answer on this question (Q1 in the GQM abstraction sheet).
The figure above also tells us how well we know our own development process.
Fig. KEC_RHW 6: Accumulated distribution over time
This figure tells us that errors distributed over phase have become stable over time and can be used as a baseline.
Why are the errors introduced?
Fig. KEC_RHW 7: Why are the errors introduced (the cause)?
Maybe the most important question is to know the cause of the errors.
The most frequently cause is the lack of system knowledge.
Actions:
One action was therefore to check what the cause lack of system knowledge really means. What kind of competence is insufficient?
A request scheme was distributed among all software developers.
The following answers were the most common:
We wondered; which kind of source documents is poor or lacks information? Are these documents written by us (the software group) or by the system group? When we study the errors, over 60% of the errors were introduced in phase 3, which is the functional design phase. 22% belonged to the specification phase (phase 2). Additionally, the distribution of the error categories showed that most of the errors were related to sequence diagrams.
Examples of actions identified:
Fig. KEC_RHW 8: Errors distributed on categories
One thing that is noticed is that the category "other code errors" is very large. This category was meant to be a small omnibus item. We thought that the categories would cover most of our faults, but obvious it doesn’t. The action will therefore be to reconsider the categories.
Beside that, the category "errors in protocols/sequences" and "faults in parameters" covers 31% of the faults. These are faults made in Message Sequence Charts and Signal Survey. The action is therefore to improve the quality of these two document types by using a new tool (Telelogic SDT) and to learn the MSC standard.
Another action is to improve the specification documents. It is the system division that writes the specification documents, so we must appeal to them on making better specifications. The action for the software division was to organise a seminar with the system division to obtain a mutual understanding on each other’s need.
The seminar between the software division and the system division, cover at least two needs: information about the functions in the radio and an understanding for the need of better source documents (seen from the software developers).
Lessons learnt
We have not succeeded answering question Q3: What is the cost of the errors distributed to the different phases and products?
It seems to be very difficult to identify the cost of the errors. Some reasons of that are:
Using the Baseline Hypothesis in the GQM abstraction sheet was very useful. As you can see from the abstraction sheet, we did this on one question, in which phase are the errors introduced? There were very interesting discussions among the developers, and we didn’t stop the discussion until everybody agreed about the answer. Because of all the discussion, the developers were very curious about what the correct (measured) answers were. This was a motivation factor for the developers when writing error reports.
Conclusion
One of our goals was to define a baseline for further improvements. We have registered the results over time, and it seems stable enough to be used as a baseline. If we want to compare the results of this project with another project, we have to take into account the surroundings i.e. the complexity of the product and the skills of the software developers. It requires that the development process (for instance the review process) is the same.
GQM is a good method as a basis for improvements. To decide the priority of actions, Pareto analysis is being used.
We had to adjust the goal and some metrics (specially the error categories and the error causes) during the collection of data.
We have already achieved positive results. One action early in the project was to improve the module test and enforce code reading (many errors were introduced because of carelessness). After three months we could see that the number of errors with this cause was reduced.
One Improvement Process as a Basis for General Process Improvement
So far we have only registered and analysed errors detected in the test phase (integration test and system test) for one project. The result is the first baseline for measuring improvements of the software process. We are now working with improvements of the software process in general, based on the results so far. At the same time we consider improvements of the metrics. Furthermore, the other software group is adopting the established system. This department has already got the results from its first analysis meeting.
We are also working with expanding the system to all phases of the development process: specification, function design, realisation, integration test and system test. In earlier phases the ‘errors’ are comments from formal reviews of documents. We have established a checklist to be used during preparation and review of documents. The reviewers’ comments have to refer to the corresponding checkpoint in order to be accepted. The data from the reviews are collected in a database to be used for analysis and decision of actions to improve the development process or the review process. The system includes the identification of the development phase in which the errors are introduced. Needs for improvement will therefore be detected for all development steps. We do not yet have any results from this expansion. However, training courses with special focus on the checklist have already been completed. The first results are expected before the end of this year.
References
[1] EFQM, Self-assessment Guidelines, 1995
Staff: 150
Turnover: 300 mill NOK
Kongsberg Ericsson ranks among the world leaders in the area of mobile communication systems for military use. The company’s tactical communication system, EriTac, provides communication for data and voice in tactical environment, and is supplied to both army and air defence programmes across the world. The company is co-operating closely with Ericsson companies on product development and export marketing.
Tactical Communication System
The area trunk communication system is designed according to user requirement set by the Norwegian Army and the specifications recommended by EUROCOM. Operational requirements specified therein call for demanding network functions to ensure interoperability between the tactical networks of allied nations as well as a high degree of mobility and survivability.
The system’s modular design allows the user to put together any battlefield communications system, from a small point-to-point system up to a large mesh network.
Rolf H. Westgaard was born in 1938 and became M Sc. at the Norwegian Institute of Technology, University of Trondheim (NTNU) in 1964. From 1965 to 1979 he worked at Tandbergs Radiofabrikk in Oslo as radio designer, the company’s education and training manager, ending up with 2 years as technical manager at their British subsidiary in Leeds.
He joined Elektrisk Bureau as a manager in the Quality section in 1979 and has since 1985 worked as a Quality Manager of the part, which ended up as Kongsberg Ericsson Communications ANS after several changes of name and ownership.
He was part of the team who started the software group of the Norwegian Society for Quality in 1983. He chaired the second European Conference of Software Quality in Oslo 1990. Rolf is responsible for the SPIQ activities in the company.
Solveig Gustad was born in the summer of 69 and became M.Sc. at the Norwegian Institute of Technology, University of Trondheim (NTNU) in 1994. Her experience is from Kongsberg Ericsson Communications ANS.The first two years she worked as a software developer and then as a software manager. As a software manager, one of her main responsibilities is to keep the development process for software as good as possible all the time. An important part of this is continuous improvements. Solveig has been involved in the SPIQ project from the start.