Software Quality Management and Software Process Improvement in Denmark
Karlheinz Kautz, Copenhagen Business School, Department of Informatics Howitzvej 60, DK-2000 Frederiksberg, Denmark Karl.Kautz@cbs.dk, Tel.: +45 38 15 24 00 Fax.: +45 38 15 24 01
Faisal Ramzan Midtermolen 1, P.O. Box 2662, 2100 København Ø, Denmark faisal.ramzan@dk.arthurandersen.com Tel: +45 35 25 25 25 Fax: +45 35 25 20 01
On might share the opinion of Denmark’s minister for work and employment, who recently commented that the disaster with Denmark’s job center system Amanda was no reason for unease as missing functionality and exceeding budgets are an inherent element of software production ([1]).
However, one might be nevertheless concerned as software plays a more and more important role in private and professional life. According to Zahran ([2]) in the last years around 4000 people lost their lives due to software errors and in the recently emerging area of e-commerce, a mistake in an electronic payment system can have fatal consequences for organizations, which invest in such technology.
Instead of ignoring these problems, it might therefore be important for software producing companies to know where their difficulties lie and how they can solve them.
After numerous attempts to improve the situation with technical approaches ranging from structured programming over CASE tools to object orientation, managerial approaches based on the insight that a product can only be as good as its development process and thus directed towards both product and in particular process have gained growing attention. They are generally known as software quality management (SQM) standards and/or software process improvement (SPI) approaches [3].
In Denmark, single studies concerning their utilization by selected organizations have been performed ([4], [5]). No further spreading of these measure will however happen, if we do not know how wide spread knowledge and actual application of these approaches are, what the concrete reasons for their adoption and non-adoption are and what their perceived effects are. But no such study exists in Denmark and this is the objective of the investigation reported in this paper.
The paper is organized as follows. In the next section the research framework is explained and then the results organized in sections concerning demographic data, awareness and utilization of SQM and SPI, SQM and SPI and the perception of software development problems, introduction of SQM and SPI, and objectives and effects of SQM and SPI. Finally, the main conclusions of the study are compared with similar work and further research questions are outlined.
Furthermore the staged capability maturity model (CMM) [8] developed at the Software Engineering Institute (SEI) in the USA and its European counterpart BOOTSTRAP [9], which is based on the ISO-standards and the CMM are covered by the study. We were also interested in the diffusion of SPICE [10], a standard for software process assessment and improvement methodologies. In addition, the respondents could put down other wide-ranging, also self-developed, methodologies, which they used for comprehensive quality management and software process improvement.
Finally, we also asked about the use of individual improvement measure such as evaluation and amendment of planning, tracking, control and review activities and single quality assurance techniques like structured walk-throughs of specification documents, unit, integration and validation testing as opposed to or as part of the above named all-inclusive standards and process improvement methodologies.
The study does not take in metrics-driven approaches (see f. ex. [11], [12]). They are not based on formal audits and/or assessments of the current practice and problems and do not include an explicit model for improvement. As such they are different from standards and model-based approaches. Therefore, although they might be valuable supplements for software quality management and software process improvement, they are not considered here.
The questionnaire was sent to 580 organization, either to the managing director, the IT manager or the quality manager, where known. The 80 member companies of the Danish Computer Technology Forum all received a questionnaire. The remaining 500 companies were chosen from 2 databases for organizations, where software developers are employed or which provide software consultation. From a composed list of 1100 organizations, which contained a limited number of enterprises with under 5 and under 10 employees – these had been the largest portion of the originally over 8000 registered firms – 500 were randomly chosen. As 33 organizations declined to take part in the study and out of the 118 responses 7 could not be used, with 111 answers usable we achieved a response rate of 20,3%, which is quite satisfying.
The responses fall into 2 categories, namely responses from the IT sector, companies, which consider software as their primary product, and others, which develop software as part of their primary product or service. In the representation of the results, we will relate to this distinction, wherever it is significant.
The population can also be divided into two other categories, namely those, who use the standards and/or improvement approaches and those, who do not utilize them. To investigate further the reasons for adoption and non-adoption the survey respondents were asked to indicate whether they were willing to participate in an interview study. As a result 6 single case interviews with 6 representatives of 6 companies, 3 using quality management and/or process improvement, and 3 not applying these approaches, were conducted on the basis of an open, theme-oriented interview guide. Each interview lasted 90 minutes and its contents was verified by the interview objects. The results are not used for generalization, but to support some of the correlations, which the survey only could touch upon on the surface. The method triangulation supported the fulfillment of the requirements concerning representativity, validity, and reliability, which have to be posed for a descriptive and explanatory investigation like ours [14].
In the IT sector, otherwise, 80% of the companies have a software development department, whereas the number is 97% for the others. With regard to the size of software development projects and project groups, 66% of all companies report that their projects are under 3 man-years, in the IT sector the number is 73%, for the rest it is 47%. The project groups have a size of 1-5 in 55% (61) and 6-20 employees in 38% (43) of the cases. Only 6%of the organizations have projects with more than 21 project members. In the IT sector 61% (50) have project groups with 1-5 members, in the other sectors this number is 37% (11).
Of the non-IT organizations 33% (13) deal with the realization of standard products for the market, 28% (11) with tailor-made customer products and the same amount with development for company internal purposes. The IT sector is dominated by adjustment of standard products (38%) and the development of tailor-made customer systems (35%).
The data shows that larger organization regardless of their sector have larger development departments. Larger enterprises have larger projects in terms of staff and man-years. A possible explanation for the differences comes from the interviews. They show that non-IT organizations have projects of larger size and more man-years. When a product has been finished, it has to be maintained and further developed. This seems to happen in the same project rather than in a new development effort.
The results are also evidence for that larger organizations develop more frequently for internal use in the organization.
Also some other results concerning the differences of respondents from the IT sector and the rest are interesting. Relating to performed development projects 60% (48) in the IT sector have performed more than 20 development projects, for the rest the number is only 28% (8). This might be linked to the size of the projects, which is smaller and shorter in the IT sector. This assumption is confirmed by the numbers concerning development experience expressed in years: 77% (23) respondents from non-IT organizations have over 11 years of development experience, in contrast to 47% (38) in the IT sector. Concerning age and experience with quality management 44% (35) of the respondents from IT companies are over 41 years old, for the rest the number is 67% (20) and more than 50% (15) of the respondents in this segment have more than 6 years of experience with quality management. For the IT sector these are 26% (21). A possible explanation for this might be that the non-IT organizations deal and have dealt with quality management longer (and) in other than their software departments.
Concerning the single methodologies the CMM is most known: 52% know it, followed by Bootstrap with 40% , the TickIT/ISO scheme 34% and SPICE with 26%. The study also shows - although not statistically supported - that organizations, which use one methodology know about others. Out of the 40 organizations, which use SQM/SPI 30 know the CMM, 23 the TickIT scheme, 22 Bootstrap, and 17 SPICE. For the 70 organizations without SQM/SPI the numbers are lower: 28 know the CMM, 22 Bootstrap, 15 the TickIT scheme, and 12 SPICE. This indicates that the organizations base their decision to introduce a particular SQM/SPI methodology on a conscious evaluation process and knowledge about alternative approaches.
There is, however, a clear need for the Danish IT producing organizations for information: 71% give as a reason for their ignorance missing information, whereas only 11% have no interest and 9% see no relevance in SQM/SPI for themselves.
However 64% (71) of the responding organizations are not using SQM/SPI. As reasons they provide that SQM/SPI is too resource demanding (25%), that staff is lacking training and education in the area (24) and that information is missing (21%). Whereas 30% have not at all considered SQM/SPI and 8% think it is irrelevant for them, 8% are planning and 54% are considering it. These do so because, they want to identify the weaknesses of their processes (32%), experience to many mistakes (25%) or have a request from management (20%).
Concerning single improvement techniques, one or more of them are used by 89% organizations and stand-alone quality assurance in form of testing are applied by 93% of all organizations, structured walk-throughs by 78% and monitoring of functionality (acceptance test) by 71%. There are no significant differences concerning the use of these techniques with regard to the different sectors or the other characteristics of organizations with and without SQM/SPI.
The study shows also that 36% (40) of the responding organizations use SQM and/or SPI: 24% are ISO certified, 22% use Bootstrap, 18% CMM, 14% use a methodology they have developed themselves, 14% name methodologies, which we had not asked for, 6% are TickIT certified and 2% use the TickIT scheme, but without certification, 9 organizations use more than 1 approach.
On a first glance these numbers give the impression that SQM/SPI is already widely used in Denmark, however looking at the IT sector and the rest separately reveals that
in the IT-sector only 28% (23) use SQM/SPI, whereas 72% (58) do not; 23 use methodologies, which fall into the category ‘other’ and 8% use self-developed approaches, 19% are ISO certified, 19% use CMM and 15% Bootstrap; 4% use more than one approach
from the rest 32% (8) are ISO certified, 28% (7) use Bootstrap, 20% (5) have an own developed methodology, 18% (4) use the CMM and 4% (1) something else; 20% (6) use more than one approach.
Given the fact that deploying the standards and methodologies completely is an extensive and resource demanding endeavor two explanations are possible: SQM/SPI is really wide spread in Denmark or not all organizations apply the approaches in the formally pre-and described way. The interview results support the second explanation: Organizations adjust SQM and SPI standards and methodologies according to their own needs instead of applying them completely as described in the literature. Thus, the number for those applying the methodologies fully is almost certainly lower than 40.
An explanation for this might be that larger organizations have more resources and therefore have better possibilities to engage in SQM/SPI. This does however not mean that SQM/SPI is only be used by these enterprises. As a matter of fact 61% (14) of the organizations in the IT sector with under 100 employees use SQM/SPI and 30% (7) of the organizations in this sector with SQM/SPI have under 50 members of staff. Thus SQM/SPI appears to be suitable also for smaller organizations.
The IT sector has less experience with SQM and/or SPI compared to the rest: over 50% (14) organizations in the IT sector have under 4 years of experience, the number for the others is 25% (4). While over 50% (8) of the non-IT organizations have more than 8 years of experience in the area, in the IT sector these are only 22% (5) firms. This is probably due to the fact that many non IT organizations have a long history of quality management with regard to the production of their primary product.
An own quality department have 77% of the non-IT and 57% of the IT organizations and they are larger in the non-IT sector. This might be explained by that for these the quality department also works with quality management for their primary product or service.
Concerning a defined development process 90% of the organizations with SQM/SPI answer that they have a defined process, for the organization without SQM/SPI this number is 39% enterprises.
requirements specification and change management are the two areas, which clearly stand out from the others: 25% (28), respectively 24% (26) of all respondents conceive these areas as very problematic requirement specification, testing, documentation, project management, quality assurance and change management are all areas, which are regarded problematic or even very problematic by over 50% of the respondents programming is considered by 68% (75) as creating little or no problems development standards, system design and configuration management are seen by 57% (63), 52% (57) and 51% (56) as activities with little or no problems.
There is also a clear dependence between the perception of problems in the various areas. Those, who experience requirements specification and systems as problematic actually see all other areas as problematic. For all other activities there are also dependencies between them and all, but 1 or 2 areas.
In summary, there is however only little correlation between the organizations’ size and their perception of problems and no correlation between the size of software development departments, of project groups and of development and the respondents perception of the organizations’ problems with software development.
For the IT sector separately, the correlation is even stronger and shows that for this sector the difference between those enterprises, which use quality management and those, which do not is even bigger.
We also asked the organizations whether they used stand-alone quality assurance measures and single improvement techniques: over 70% do so, but their utilization has no significance for the organizations’ perception of problems within software development.
Organizations that have introduced quality management have had larger problems with software development and therefore started e deploying quality management and process improvement. Organizations that do not use SQM/SPI know that they do not have problems and therefore they are not interested in the standards and approaches. Organizations that do not utilize quality and process measures do not realize that they have problems with software development, whereas organizations with SQM/SPI are more conscious about their problems within software development.
Thus, the organizations see SQM/SPI as an approach that helps them find their problems and as such gives them an overview over their software development activities.
Our investigation shows that both organizations with and without SQM/SPI have problems with software development. While some of them do not try to solve these problems, because they are not aware of them, others are very conscious about how to work with their difficulties.
The 3. explanation is also supported by the fact that SQM/SPI according to the majority of the respondents has an overall positive or very positive effect on all the different activities in software development and this although organizations with SQM/SPI perceive software development as more problematic than those without.
Thus, SQM/SPI can be used to get an overview over the problems in software development and this contributes to a positive change of the single activities and the processes as a whole.
It can be argued that the positive effects actually confirm the first rationalization, but this is only a part of the explanation. The interview results support also that organizations with SQM/SPI are more conscious about their situation, because SQM/SPI provides the opportunity via evaluations and assessments to focus on the actual status.
In comparison, organizations without SQM/SPI are used to what they do without assessing and reflecting the situation. The existence of problems with requirements specification or project management for them is a natural element of software development.
It is not necessarily a formal assessment or evaluation, which is needed to become aware of the problems with software development, but the insight that its possible for the organization to improve is important. Such an insight is however often blocked by an understanding of quality management and process improvement as demanding extensive documentation, being bureaucratic and impeding creativity. Another reason might be that the business world just accepts that the majority of software projects overruns budgets and time schedules. Finally, as long as staff are evaluated according to whether they deliver on time as opposed to that they deliver quality, SQM and SPI will not be recognized as relevant. These attitudes have to be changed before SQM/SPI can be seen as ways to improve process and product quality.
The study shows that 53% of the organizations use external assistance for the introduction of SQM/SPI. This help is primarily used for assessments and evaluations, launch and establishment of a quality or improvement project and for consultation.
As a starting point 50% of all organizations choose to implement SQM/SPI in selected projects, while 13% start with selected departments, and 29% decide to start with a company wide implementation.
There is however no correlation between the activities, which are perceived as being problematic and those, which the organizations start their SQM/SPI activities with.
Project management, requirements specification, testing, development standards and system design are those areas, where improvements start. These are subsequently also the areas, where the introduction of SQM/SPI has gone longest and is considered as implemented.
The chosen implementation strategy - not to start company wide - indicates that most organizations are aware of the complexity of the introduction process. This is confirmed by the respondents’ answers, where 70% (28) grade the process as difficult or even very difficult.
The survey does not investigate the kind of changes, but the interviews show that the changes are of different character: some are changes of the physical and/or organizational structure, others are changes in the business and development processes.
They all have far reaching consequences on the organizations as whole and are not considered as isolated solutions to local problems. As such the introduction, implementation and utilization of SQM/SPI means organizational development.
Concerning the technical and support activities for software development the numbers are similar.
Positive or very positive effects are stated in the case of requirements specification (81%), system design (53%), programming (62%), testing (86%), documentation (69%), project management (75%), configuration management (61%), use of development standards (55%), quality assurance (70%), and change management (47%), only in 4 cases the effect was negative.
Another effect is that a fourth of the organizations (24%, 17) plan to work on more new processes. This shows that they are not only focusing on resources, but more on the new processes SQM/SPI bring about. Still, 20% (14) of the organizations intend to use more resources for SQM/SPI, and 27% (19) will work upon that SQM/SPI thinking pervades the whole organization.
The study shows that all organizations with the exception of 1 have formulated business related objectives in at least one of the following areas: customer satisfaction (20%), staff satisfaction (20%), delivery precision (19%), staff motivation (8%), resource consumption (9%), cost reductions/expenses (13%), and budgeting precision (9%).
In total, 90% of those using SQM state that they perform measurements. These are distributed as follows: 20% of all answers fall into the category of delivery precision, 18% measure customer satisfaction, 17% staff satisfaction, 14% expenses, 11% resource consumption, 10% budget precision and 9% staff motivation.
With regard to objectives for the software development activities 12 out of 40 organizations have not formulated any objectives and only 23 organizations perform measurements. The actual distribution is: testing (objectives: 14%, measurements: 14%) documentation (9%, 13%), system design (7%, 7%), project management (9%, 13%), programming (8%, 7%), quality assurance (12%, 12%), requirements specification (12%, 12%), configuration management (7%, 8%), development standards 10%, 8%), change management (4%, 5%).
The study has not investigated, which objectives the organizations’ have defined and how they perform the measurements. But, it has not been possible to find any correlation between the organizations’ defined objectives and their actual areas of measurement.
All this indicates that measurement in general is a very problematic area for all organizations. This suggestion is confirmed by the interview results: none of the organizations has a somehow functioning metrics program. However, this does not mean that organizations with SQM/SPI do not engage in measuring. They see it as a means to improve their processes, but have problems with defining the appropriate metrics, whereas organizations without SQM/SPI do not see the value of measurements.
All most all organizations have a positive attitude to SQM, but SQM standards and/or SPI methodologies are NOT known by 40% (44) organizations.
The positive attitude towards quality management is probably due to the positive connotation, which the concept of quality has. However, knowledge about SQM/SPI is not especially wide spread in Denmark. This is inline with the results from a similar survey in Norway [15], whereas in a survey from 1995 [16] comparing Italy, the UK, Germany and France more than 75% of the respondents are not aware of the SPI approaches, but only about 20% do not know the ISO9000 standard series .
SQM and/or SPI are used by 36% (40) organizations.
The study reports that 36% (40) organizations utilize SQM and/or SPI. We have already discussed that the actual number of organization might be lower as most organizations have adjusted the standards and methodologies to their needs or have developed their own approaches. In this light the 56% utilization reported from Sweden[17] look quite high. We do, however, not know the definition of the term SQM underlying that survey. In Norway [15] 17% use the TickIT/ISO scheme, whereas 8% use Bootstrap, the CMM or own developed methodologies. In Sweden the TickIT/ISO scheme and the CMM are most spread, while SPICE is more used than BOOTSTRAP. In Denmark ISO 9001, Bootstrap and the CMM are most used, and SPICE is not used at all. This difference can probably be explained by the fact that a particular Swedish change agency pushes the ISO standard and SPICE, whereas in Denmark one agency is very active promoting Bootstrap. The accumulated numbers for Italy, the UK, Germany and France are: 33% use the ISO9000 standard series for software activities, 28% the CMM – due to an extensive utilization in France (69%), 8% SPICE – again due to a very large application in France (28%), and 8% Bootstrap
SQM and /or SPI increase awareness for software development and its problems.
There exists a wide spread understanding that SQM and/or SPI create increased attention concerning the software development processes (see f. ex. [9], [10], [18]). Methodologies like the CMM explicitly aim at providing a more sophisticated picture of the development activities in accordance with the ascending level of maturity of the organization. Our study confirms that the awareness of software development grows with the use of SQM and/or SPI. In particular the study concludes that organizations, which utilize SQM and/or SPI are more aware of their problems - and as such they have an appropriate basis for improvement - than organization without SQM/SPI. In a more recent European study [19] about problems in European software production units, this supposition is also put forward, but no evidence is provided.
SQM and /or SPI have a positive effect.
In contrast to [19] where the conclusion is that SQM/SPI do not lead to a reduction of problems in software development, the respondents in this survey state that SQM/SPI have a very positive effect on both business and software development factors. This is in line with large parts of the literature (see f. ex. [2], [5], [8], [9], [20]). It has however to be considered that the survey has predominately been answered by management, who in many cases are the initiators of the SQM and/or SPI endeavors and that no assessment results and/or hard measurements have been required as evidence for the respondents’ subjective judgement.
SQM and/or SPI is to a greater extent used by larger organizations, but can be used successfully by organizations of all size.
The study shows that larger organizations have a greater tendency to utilize SQM and/or SPI than smaller organizations. At the same time the investigation indicates that both large and small organizations adjust the standards and the methodologies to their own needs. This is confirmed by f. ex. [5] and [21]. The study of the diffusion of SQM /SPI in Norway [15] provides results, which also support this research. There the conclusion is that larger organizations apply SQM/SPI to a greater extent than smaller ones. It is actually argued that it is uncertain whether small companies will adopt methodologies like the CMM and Bootstrap as these seem to better fit the needs of larger firm. However, although these methodologies originally were aimed at large organizations, [22], [23], [24] and [25] all discuss SQM and/or SPI in relation to small enterprises and show possibilities how these can be beneficially used by smaller firms. The suitability of SQM and SPI for organizations of all sizes is corroborated by the fact that regardless of size most organizations report a positive effect of the measures.
Lack of resources and missing staff engagement are the biggest problems when introducing SQM and/or SPI.
Lack of resources is in general a problem when introducing novelties, especially in small and medium-sized organizations [26]. The Danish software producing units are no exception here and confirm the results from the Norwegian study [15]. Missing staff engagement might appear as a surprise, but as mentioned above the majority of the respondents are members of management. The survey does not ask for the reasons of the personnel’s little engagement, but it be a result of insufficient empowerment and encouragement for employee participation. Active involvement, however, has long been recognized as one of the decisive factors for successful implementation of SQM and/or SPI programs [27].
Measurements and metric programs as part of SQM and/or SPI are a major problem.
Measurement is a complex process and the organizations in the study have major problems with metrics programs as a part of SQM and/or SPI.. This confirms the literature in the field (f. ex. [28], [29], [30], [31]).This literature contains some advice, but this seems to be seldom used. The Swedish investigation [17] also substantiates this conclusion. It shows that measurements and metrics are considered as most problematic in comparison to other factors, which constitute SQM and SPI. The Swedish study also includes a European wide comparison and also there measurements are seen as the most problematic area of SQM and SPI.)
SQM and/or SPI means organizational development.
This conclusion of our investigation is underpinned by the literature, which confirms that SQM and/or SPI are organizational development ( see f. ex [2], [5], [8], [32]) that has consequences beyond the software producing units of the enterprise for the organization as a whole. In our study more than half of the answers state that they performed, respectively, had to perform structural changes in their organization when they introduced SQM and/or SPI. While this insight might be well known and documented in other fields of management and in systems development (see f. ex, [33], [34]), in the context of SQM and/or SPI in practice, companies do not always seem to be prepared as the problems with the introduction of the standards and methodologies, which are reported in our study, show.
Small organizations do not seem to utilize SQM/SPI as regularly as large organizations. It would be interesting to investigate further why this is so. Organizations with SQM/SPI perceive software development as more problematic than those, which are not using these approaches. Here it is obvious to take a closer look at the individual activities and to analyze the differences in handling the various tasks. Most organizations believe that training and education can solve the problems during the introduction of SQM/SPI. It would be interesting to investigate, which particular qualifications are needed by management, project leader and staff for SQM/SPI. Finally, the study reinforces that SQM/SPI is organizational development. It is therefore obvious to investigate in more detail, which influence SQM/SPI have on an organization, in particular its business processes.
[2] Zahran, S. (1997). Software Process Improvement - Practical Guidelines for Business Success. Addison Wesley Longman, Harlow, UK.
[3] Thomsen, H.E., P. Mayhew (1998). Approaches to Software Process Improvement. In Software Process - Improvement and Practice, Vol. 3, Issue 1, pp. 3-17.
[4] Jonassen Hass, A. M., et al. (1997), Bootstrap – the real way to SPI, Quality Week Europe 1997, quoted from Center for Software Process Improvement (2000), Delta report D- 263, pp. 17- 34, Hørsholm, Denmark.
[5] Center for Software Process Improvement, 1998, Danish Experiences with the Improvement of the Software Process, (in Danish). DELTA report D-262, Hørsholm, Denmark.
[6] ISO9001 (1987). Quality Systems - Model for quality assurance in design, development, production, installation and servicing. European Standard EN29001, Brussels, Belgium.
[7] TickIT (1992). A guide to software quality management system construction and certification using EN29001, Issue 2.0. UK Department of Trade and Industry, London, UK.
[8] Humphrey, W. S. (1989). Managing the Software Process. Addison-Wesley, Reading, USA.
[9] Kuvaja, P., et al. (1994). Software Process Assessment & Improvement - The Bootstrap Approach. Blackwell, Oxford, UK.
[10] El Eman, K. et. al. (1997). SPICE - The Theory and Practice of Software Process Improvement and Capability Determination. IEEE Computer Society, Los Alamitos, Ca., USA.
[11] Basili, V. R., H.-D. Rombach (1988). The TAME Project: Towards Improvement - Oriented Software Environments. In IEEE Transaction of Software Engineering, Vol. 14, No. 6, pp. 758-773.
[12] Pulford, K. et al. (1996). A quantitative approach to Software Measurement - The Handbook. Addison-Wesley, Reading, USA.
[13] Andersen, E. B (1991), Statistics for civil engineers, (in Danish). Akademisk Forlag, Copenhagen, Denmark
[14] Andersen, I. (ed.) (1990) Choosing methods for organizational Sociology, (in Danish). Samfundslitteratur, Copenhagen, Denmark.
[15] Larsen, E. Å., K. Kautz (1997). Quality Assurance and Software Process Improvement in Norway. In Software Process - Improvement and Practice, Vol. 3, pp.71-86, 1997.
[16]European Observatory on Software Engineering (1995). Final Report, PAC France/Germany.
[17] Wang, Y. (1998). Analysis report of the Swedish Benchmarking Survey of Software Development Practices. IVF, Technical Report, Mölndal, Sweden.
[18] Paulk, M. C. et al. (1991). Capability Maturity Model for Software. Technical Report CMU/SEI-91-TR24. CMU/SEI Pittsburgh, Pen., USA.
[19] Lee, M. S. Dutta, L. v. Wassenhove (1999). A Empirical Analysis of Software Production Problems in European Software Units. In Pries-Heje, J. et al. (eds.), Proceedings of the 7th European Conference on Information Systems, Vol. II, pp. 465-481, Copenhagen Business School, Denmark.
[20] Herbsleb, J. et al. (1994). Benefits from CMM-based Software Process Improvement: Initial Results. Technical Report CMU/SEI-94-TR13, ESC, R-94-013. CMU/SEI Pittsburgh, Pen., USA.
[21] Kautz, K. et al. (2000). Applying a Software Process Improvement Model in Practice: The Use of the IDEAL Model in a Small Software Enterprise. In Proceedings of the International Conference of Software Engineering, 4-11, June, 2000, Limerick, Ireland
[22] Brodman, J. D., D. L. Johnson (1994). What Small Businesses and Small Organizations say about the CMM. In Proceedings of the 16th International Conference on Software Engineering, IEEE Computer Society, pp. 331-340, May 1994.
[23] Brodman, J. D., D. L. Johnson (1997). Tailoring the CMM for small businesses, small organizations and small projects. In Software Process Newsletter, IEEE Computer Society, No. 8, Winter 1997.
[24] K. Kautz (1998). Software Process Improvement in Very Small Enterprises? Does it pay off? In Journal of Software Process - Improvement and Practice, Special Issue on Organizational Change through Software Process Improvement, Vol. 4, No.4, pp. 209-226.
[25] Paulk, M. C. (1999). Using the Software CMM in Small Organizations. Technical Report CMU/SEI-99. CMU/SEI Pittsburgh, Pen., USA.
[26] Rogers E. M. (1983). Diffusion of Innovations, Third Edition. The Free Press, New York.
[27] Oakland, J. S. (1993). Total Quality Management (2nd edition). Butterworth-Heinemann, Oxford, UK.
[28] Fenton, N. E.., Pfleeger, S. L. (1997). Software Metrics - A Rigorous and Practical Approach, PWS Publishing Company.
[29] Pfleeger, S. L. et al (1997). Status Report on Software Measurement. In IEEE Software pp. 33-43, March/April 1997.
[30] Kautz, K. (1999). Making Sense of Measurements for Small Organizations. In IEEE Software, Vol. 16, No. 2, pp. 14-20.
[31] Iversen, J. H. , Mathiassen, L. (2000). Lessons from Implementing a Software Metrics Program. In Proceedings of HICSS 33, Wailea, Maui, Hawaii.
[32] Bang, S. et al.(1991), Quality Management in Systems Development, (in Danish). Teknisk Forlag, Copenhagen, Denmark.
[33] Borum, F. (1995). Strategies for Organizational Change (in Danish). CBS Publishing Company. Copenhagen, Denmark.
[34] Andersen, N. E., et al. (1990). Professional Systems Development - Experience, Ideas, Action. Prentice Hall, Hemel Hempstead, UK.