EuroSPI99 
Learn from the Past - Experience the Future
European Software Process Improvement
Testing
Category Index
Rated Newspaper Supported by EU Project 

THE QUEST FOR QUALITY TEST RESOURCES

· Quality and Efficiency in Software Testing by moving the technological boarder ·

Per Jørgensen
Kapital IT, Denmark
 

About Kapital IT


Kapital IT is situated in a suburb of Copenhagen and has approximately 400 IT-professionals employed.

Kapital IT is a company under Kapital Holding A/S that is the third largest financial institution in Denmark and includes BG Bank A/S and Realkredit Danmark. Kapital IT develops software for these institutions and supports their business domain strategies with IT-solutions.

Kapital IT develops a variety of financial software that is implemented on various mainframe, midrange, and Client/Server platforms. Kapital IT works on a project basis with the emphasis on an efficient development process supported by a development concept that secure consistency in the project. Projects are measured on weather they are carried out as agreed with regard to customer contentment, requirements, quality, economy, stipulated time etc. etc.
 

The starting scenario

Defect infested software is a serious quality problem. In addition, it is especially a big problem if you still have defects after spending 40% of total development time on test. It is an even bigger problem if your Compass investigation shows that the competitors spend less time on test and at the same time is having fewer defects in production than you.

When your software supports your no. 1 strategic area of growth where the competition is most vigorous and your company continued success on the market rely strongly on the quality of your software - you do have major teeth grinding problems.

The Challenge

We faced a challenge similarly to the once mentioned above. Up through the nineties our Web Banking products grew from small applications into large business and computer complex applications. At the end of the nineties once rather simple applications now involved several platforms of different technical system architecture and furthermore the Web Banking products had increased into a wide variety of products each targeting special customers profile.

At the same time the importance of the products from the business point of view grew and the products eventually became a strategic area of growth for BG Bank A/S and the Danish financial sector as a whole. In the fight for market shares, Web Banking application was now suddenly one of the key factor and of the outmost importance for continued success on the market. It is not an understatement to say that competition was and still is most vigorous.

In order to have a precise picture of our product quality we carried out in co-operation with the international benchmarking firm ‘Compass Development’ a systematic investigation of productivity, quality, economy etc. in co-operation with systems development and systems administration.

The investigation showed among other things that 30 to 40% of the total time assigned to development was spend on test. This was approximately 5-10% more than companies we compare us self with spend on test.

The investigation also showed that the quality of our products measured in faults per function point was lower than comparable products on the market [1].

To summon up the investigation in one challenging sentence >> we should not use more resources on test, but use fewer resources more effectively.
 

The Plans and the expected outcome

To use less resources on test more effectively is easy to say but hard to do. At least our investigation gave us a clue in what to do. Obviously our development and test process had to be in a degree somewhat ‘out of order’ and to many defects was implemented into our products, test and production environment. If we could ‘fix’ our development and test process and find defects early in the requirement phase then maybe we could reduce time spend in the software test phases? To find defects early in the requirement phase we could maybe involve end-users and customers in a different way? Maybe we could automate our GUI tests using modern test tools, maybe that could reduce the defects, and the times spend on test? If we automated the GUI test then why not involve the end-users in that phase too?

Finally, our QUEST began. A QUEST that gave us new knowledge and that lead us to new frontiers. In fact, the QUEST continues to this day and beyond.
 

The QUEST


Our QUEST began by asking our self the following question >>What if our products suddenly had no (Zero) defects at all and was loved and handled correct by every single customer? <<

If that once became true, several things would happen, for example:
 

Of cause, it was only a very nice dream. Nevertheless, the fact is that better product quality actually does not only create happy customers but also release human resources. Some of which are IT-professionals and others who have in depth business knowledge of various kinds but all with skills that can be used well in the development and test process. You could almost call them The Hidden Resources that just waited to be involved.

We thought a lot about that dream while we investigated and analysed our development and test process [2]. This investigation disclosed the following major opportunities for improvement. We found that:
 


To our great pleasure, we discovered the ESSI project, which was willing to support our QUEST experiment.

With these results in mind and still with our dream in fresh recollection we founded our project on 3 major building stones

1. The User and Customer Involvement in Test (UCIT)

2. The Requirement Driven Test (RDT)

3. The Automated GUI Test (AGT)

We baptised the project QUEST for ‘Quality and Efficiency in Software Testing’ and later we added ‘by moving the technological boarder’.
 

The User and Customer Involvement in Test (UCIT)


The objective of the UCIT phase was to look at The Requirement Driven Test (RDT) and the Automated GUI Test (AGT) to find the optimal way to involve end-users and customers on equal terms with the ‘IT-professionals’ and thus enhancing the process in general. If possible, it would not be such a bad idea to move the technological boarder between IT professionals and end-users and transfer traditional development activities to the business unit. We could sure use all the
IT-professionals we could get. With the national and international shortage of IT-professionals, we just had to use our resources as optimal as possible and that demanded us to rethink the end-users and customers involvement in our development and test process.

From the beginning of the project we choose to look at our end-users and customers NOT as resources in the project but rather to look at what resources that they could bring into the project to strengthen this and the end product.

We invented in our minds a phantom: ‘The All European User and Customer’.

Our phantom co-worker ‘The All European User and Customer’ had no IT education but had lived successfully through the PC revolution. Their IT interest and knowledge was no longer restrained to ‘their’ application at work. They had a private PC at home. Their IT awareness was high. Some made their own Homepage. Some programmed for fun. They all used the Internet to ‘chat’, ‘surf’, use e-mail etc. and it was a long time ago since the phantom ‘The All European User and
Customer’ had second thoughts about IT.

We would love to work together with this phantom and we believed we could find ‘it’. And we did!

We found her and him everywhere. In BG Bank they where to be found among product Hot-line staff, among account managers and among banking consultants. Among the customers they where easy to spot. Customers who had been beta-sites several times where almost for sure one of our phantoms. You could almost say that those who would benefit first from better quality where a possible phantom.

They could all contribute to the project with their different in-depth business knowledge. With that and their interest and use of IT we thought that their contribution to the end-product could be more than what end-users and customers ordinarily contributes to in a development and test process. We should soon find out if this was true.

When approached and asked about if they would like to be involve in our project in a different way than usual the vast majority of the ‘All European User and Customer’ was exited about our ideas. They where excited for different reasons. Some saw the involvement as job enrichment. Some saw new frontiers and job opportunities open up. Others just liked the idea about having influence on the new products design and functionality.
 

The Requirement Driven Test (RDT)


Our investigation showed us that the requirement phase was out of focus. We would like to get it back on track. We would like to implement into our development concept a RDT phase that made sure that the requirements for new products was detailed to a level that made test automation possible.

Furthermore, the RDT phase should make sure that all requirements was recognised and accepted throughout Kapital IT and BG Bank.

We would also like if the RDT phase could pin a name on each requirement so each and every requirement had a ‘sponsors’ with the responsibility to make sure that the requirement was tested and found as described and requested [3].

At last this new RDT phase should of cause involve the ‘The All European User and Customer’ in reviewing the requirements.
 

The Automated GUI Test (AGT)


Before our QUEST project, we had some experience with test automation tool [4] [6]. We had even back in the early nineties used these tools in a long period where we converted numerous systems between two platforms. However at that time we found the maintenance task overwhelming and not justifiable from an economical point of view. We had a few GUI test suites left though and occasional we used tools, but only on and on, and off basis. When we used tools, the
end-users were never involved because we found the tools to difficult to learn and to use for non IT-staff.

However, we knew that the tools and the tool market had evolved tremendously since we last seriously has ‘shopped’ for a test tool.

We knew that many companies had automated tests up and running but we also knew that many companies had failed in the automation process.

In spite of this we looked at the market and found that there were many tools to choose from and a lot of them looked surprisingly good and user-friendly too!

This gave us an idea. If we could find a tool that not only supported the developer requirements but also was highly user-friendly then why not involve our ‘All European User and Customer’? If we together could implement the tool and make our own user-friendly handbook in our AGT process, then why not transfer most of the business domain test from Kapital IT to BG Bank and let the ‘All European User and Customer’ take control of the AGT process?

If we could move that technological boarder between IT professional and end-users, we could release IT resources for further traditional development. With a better quality, the end-users would be released somewhat from defect handling and this ‘spare’ time could then be used on the AGT process!

Well it all seemed to fit perfectly. We had the ideas, we had the 3 building stones (UCIT, RDT and AGT) so all we now had to do was to describe the Quest project and then of cause as a minor detail ‘sell’ the experiment intern in the company and to the ESSI project.

The description part was easy. The ‘selling’ part took some hard work to accomplish.
 

The QUEST project objectives


We described the Quest project objectives as follows:

The Quest objective is to improve the test and development process of our multiple platforms that support our Web Banking systems and thus make it more efficient.

The improvements of quality and efficiency is to be carried out through:
 


In re-use the results from the AGT process in subsequent business domain tests of new releases of our Web Banking systems. (UCIT, AGT)

Notice: Our 3 building stones (UCIT, RDT and AGT) mentioned above are tied to the each objective. The Quest objectives were not all that measurable however the objectives were made more measurable in the following Business and Technical objectives.
 

The QUEST business objectives

The Quest business objectives were as follows:

To reduce the number of production errors by 33% during the first 6 months of a release as a result of a more thorough test and thus strengthen the quality of BG Bank's strategic area of growth

To reduce the production and the maintenance costs by at least 20% do to fewer errors.

To increase the percentage of Hot-Line replies to customer enquiry’s to 98%. This to take place through the staffs strong involvement in the business domain test and because fewer errors means fewer enquiry’s.

Release 10% of the BG Bank end-users in order for them to test future releases of the Web Banking system. This is to take place through fewer errors and thereby fewer customer enquiry’s

The Quest supports the business needs for competitive Web Banking products and gives the customers of BG Bank a quality product at a high technological level. Through this, existing customers are maintained and new customers attracted.

By qualifying the BG Bank end-users to "test super- users" and through their the involvement in the test, we shall obtain a "hands-on" evaluation of new releases before the implementation which will ease the pressure on the business unit in connection with the implementation of a new functionality.

By combining the business domain knowledge of the BG Bank end-users with the system and technical knowledge of the developers, we will achieve a considerable synergetic effect at a critical time in the course of testing. This will have a clear effect on the quality of the final product and will ease the implementation of the new functionality in the area of business.
 

The Quest Technical objectives:


The Quest technical objectives were as follows:

To form a business domain test of the Web Banking system complex which is end-user-oriented as far as domain-test is concerned and thus releasing at least 10% development test resources for development of new releases of the Web Banking system.

To gain experience and concrete results to be used for subsequent tests of the system complex. This includes test of the readjustment to Economic Monetary Union.

To describe and document a QUEST Best Practice for future and other end-user-oriented tests of the same character in a complex software environment.

To secure that future externally developed operational systems can be implemented in the existing end-user oriented system complex and be tested by means of the "Best Practice" described.

To secure that an automated GUI test process mainly is end-user controlled with quality support from IT professionals.

Notice: The hard numbers or should I say ‘the measurable percentages’ was founded on the investigation and analysis mentioned in the beginning of the article, the Compass experience and a handful of expectation and ‘hope’.
 

The implementation of Quest


Before we could implement the project, we had some selling to do meaning the project had to be approved by the management in Kapital IT and BG Bank. Finally our quality challenge concerning total time spend on test, faults per. function point etc. mentioned earlier and our thorough investigation and analysis of our current situation did the selling [1] [5]. OK I must admit that it did help that we could say that ESSI and thus the European Commission would finance some part of the project.

Anyway we got the Go Ahead sign and off we went
 

Implementing ‘The User and Customer Involvement in Test (UCIT)’


The UCIT phase was implemented as an integrated part of the RDT and AGT phase. You can actually say that we tried to implement a state of mind where we constantly worked to improve and increase the end-users and customers involvement.
Therefore, it is not correct to look at the phase out of context and not include the other two phases.

Anyway I will give it a try and describe the implementation plan for the UCIT phase in the below 5-step plan:
 

Find the appropriate activities where the User and Customer can strengthen the project and product the most.

We had already targeted those areas in our project objectives to be a) the requirement phase b) the domain test phase and c) the beta test phase
 

Find the ‘All European User and Customer’

We had already found them in our preliminary investigation. Among those who would benefit the most from better quality and among our beta site customers.
 

Get them assigned to the project.

When top management approved Quest, we thought that this part would be just a formality. However, we soon discovered that it was not always the case and sometime we had to fight a battle to get the end-users assigned on a serious basis to our project. Of cause when we talk about real customers we had to approach these through official channels and we often found that the customers sales manager where a perfect contact person. He or she had also a clear picture of whether our request would be turned down or not. However, as a golden rule I can say that ‘old’ beta site customers usually was a sure catch.
 

Train them well in the activities

Well when you get new staff you train them, right? We did that and kept in mind that our new co-workers had a different approach to IT.
 

Support them before, during and after

Our new co-workers had ventured into new territories and we found it important to support them in their new role. Before they began to work with us, during the project phases and after when they had returned to their normal work situation.
 

Implementing ‘The Free Test’ (UCIT)’

We implemented one activity thou that did not fit under the RDT or AGT ‘cap’. It was The Free Test or I could call it the very early beta test. What we did was to release the product for beta test very early in the development phase knowing that it was full of defects. What we actually did was that we more or less finished one part of the product and then released it to our beta sites customers. We told them something like this >>Hey, here is our coming new product! It is far from finish. It is full of defects! Can you find them? These features are almost done; do you like them? What defects do you thing they have? By the way, what you ‘put in’ you might loose and you can not be sure to upgrade with the next pre release. Please comment on the features and tell us about the defects<<

The readers of this article might thing >>Hey, the morons released a prototype! <<. However, this is not the case. Our pre releases had features that were more or less usable and almost finished. Let us say for example that the customer could do all types of payments but not look at statements. The customer could then get some of the daily work done and at the same time try out the product. Maybe find some defects? Maybe get a good idea for a new or better feature?

We on the other hand had suddenly many customers (read: testers) that did functionality test and usability test at same time and for all for FREE.

You might say >>There is no such thing as a free lunch<< and of cause you are right. We had to establish a procedure to support the customers but the cost to run this was nothing compared to the expense if we had had to hire real testers to do the job.

The procedure to run this pre beta test is pictured below.
 

                                                                              Fig. PEJO.1 : The Pre Beta Test Procedure

The sales manager is the link to the customer. The sales manager communicates with an established Hot Line or via a news group. Defects and comments are stored in a defect database. At intervals, newly recorded defects and comments are evaluated. The outcome of the evaluation is given to the customer via the sales department and the result is logged in the database.
 

Implementing ‘The Requirement Driven Test’ (RDT)’

The Requirement Driven Test (RDT) was implemented as a 3-Level requirement process. Before these RDT levels was established customers was interviewed about their view on existing products, which features they would like in the new product, what they found vital for such a product etc. etc.

This round of interviews spawned or fed some of the requirements for the RDT phase. From the beginning of the phase, we kept a high standard in the documentation of the requirements. We established a database accessible to all participants in the project and kept a strict control to make sure all documentation was linked to the database.

The 3-Level RDT is pictured on the following page:

                                                                           Fig. PEJO.2 : The Requirement Driven Test Phase

We implemented The RDT phase ad a three level process; A) Management level B) End-user level and C) Script level.

At level A. the requirements were identified accepted documented and reviewed. As mentioned earlier some requirements were identified in the interview round with the customer. Other requirements was ‘born’ as legal requirement, future business requirement etc. at level A. The level contributed at the same time to the framework for the future application cost wise and other wise.

At level B the requirements were analysed described documented and reviewed to a functional level using brainstorms, desk test and usability test. Not only did the end-users and IT professionals participate in this stage also customers were involved through contact with the sales managers. Responsibilities for the individual requirement were anchored in the end-users organisation as well as within Quest project. When the work at level B was finished a formal review was held with participation of management, IT-professionals, and end-users. When the review was completed, the change in the requirements was not possible.

In Level C the end-users broke down the requirements so, it was possible to begin the detailed planning of the automated test. You could say that level C hooked the individual requirement to the automated GUI test. In that way we had control of each requirement, what ‘state’ it was in etc.

Even thou it was implemented as three separate level the communications between the levels were high and necessary. What went on at one level had to be reviewed commented or otherwise at the other levels.

Prior to the implementation we talked a lot about switching level B to level A and thus having the end-users ‘on stage’ before the management. Anyway we did not and discovered afterwards that because of the intense communications back and forth between the two levels the discussion prior to the implementation was like talking about what came first: The hen or the egg. Actually we started out with the management but when process got going back and forth, it did not matter much which of the two levels that began the process.
 

Implementing ‘The Automated GUI Test’ (AGT)’


Our implementation of the AGT phase were divided into 4 major activities:
 

Training in test

The entire project – IT-professionals as well as end-users - went back to school and were trained in test techniques and test methods. We found it important that all participants had the same basis knowledge of test. Every one went through a tutorial that focused on the test side of the V-model, white- and black box test, preparation of test cases, suites, documentation, end conditions for business domain test etc. etc. The tutorial helped to build some sort of a test foundation in the project from where we could continue down the AGT road.
 

Selection of test tool

We put a great deal of effort into the selection of the test tool. The tool used for AGT were selected in strong co-operation between the end-users and the developers where the main requirement for the tool was that it should be as user-friendly as possible but also of a high technical development standard [4].
The selection process is described below

                                              Fig. PEJO.3 : The Tool Selection Process

The project documented the requirements for the tool before a broad selection of tools was reviewed in typical 2 to 4 hours demo sessions. Among these the three most suited tools was selected and the tool vendor was asked to give a written answer to our tool requirement list. The two tools that meet our requirements the best were selected and a 1-day workshop for each tool was arranged. IT-professionals and end-users participated in these workshops and each participant filled out a questionnaire concerning the tool.

Before making the final decision both tool vendors was asked to try-out their tool on one of our web applications. At last we was able to make our choice that eventually was QA Centre from Compuware. This tool was an over all winner and voted the most user-friendly by the end-users and best language by the IT-professionals. Furthermore, we put also a great emphasis on tool support this we also found was best at Compuware.
 

Try-out of test tool and AGT process

The end-user and IT-professionals was trained in the tools and thereafter was the AGT process including the tools tried out by automating a part of an existing web application. In the try-out, a best practice handbook was established. The handbook describes both the AGT process and how the tool is used in the process. The try-out was a great learning experience and founded the basis to implement Quest in reality as well as in our development concept.
 

The actually AGT process

A lot has been said and written a lot about automated GUI test [4] [6] [7]. So I will try to outline the differences in the AGT process we implemented. First of all the end-users in our process had been involved in Quest on equal footing with the IT-professionals from the very beginning of the project. They knew the requirements down to every detail and they had prepared the test-cases/script in the RDT phase. They had also extensive tool knowledge and they had tried-out GUI automation in the AGT try-out. Therefore, we implemented an automated GUI test not very different from others like it elsewhere. However, our AGT was to a wide extend end-user controlled. The business domain test was planned and carried out by the BG Bank end-users. Scripts were written not recorded by the end-users.
Test-suites were build and executed by the end-users. Scripts and the entire test-suite complex were maintained by the end-users. Defects were registered and pre investigated by the end-users. Of cause the IT professionals were also a part of the process but their role was – you could say – reduced to the role as consultant and thus to yield computer, tool and process assistance.
 

After the implementation of QUEST

After we implemented a great product the end-users returned full-time to their daily work in BG Bank. They took with them the task and responsibility to maintain the test cases, test-scripts and test-suites. At the same time learning-groups were established that meets on intervals where end-users and IT-professionals exchange experience, knowledge, scripts etc. etc.
 

The Quest Impact, Results and Lesson learned


In our QUEST for Quality we fought numerous battles, some we lost and most we won but most important we won the war for better software quality and we now know that:
 

The Impact and lessons learned in UCIT phase
The All European User and Customer are great ballplayers in the entire project phase and for sure, it is the undiscovered human resource. The end-user’s business domain knowledge strengthens the whole process and ensures a great product. In addition, the Customer involvement makes your product ‘stand out’.

Yes! You can move that technological boarder and train non-IT people traditional IT skills and get a better product out of the effort.

When that is stated I can say that we learned that the end-user involvement in the beginning will cost development resources it is not a free ride and you must plan to do a lot of practical work. It is important precisely to describe the objectives and activities of the involvement. Especially when approaching customers it is important to do it in a formal manner. All-ways remember to give responses to customers about what you did with the comments they gave or the defects they found. It is vital to seek and get management consent before all activities and it is maybe necessary to ensure the management consents through the entire project. In involving end-user and customers, it is a good idea to execute each activity in short intensive periods. It is important that end-users return on a regular basis to their base organisation so they can refresh their business domain knowledge. It is important to keep an open dialog about the involvement particularly with end-users not assigned to the project. Treat end-users and IT-professionals a like but be conscious about the end-users non-IT profile.
 

The Impact and lessons learned in RDT phase
The requirement phase is alpha and omega. You have to manage and test your requirements. You have to have the right people to work with the right requirements at the right time. You have to constantly look critical at your requirement phase and ask your self the question ‘Were is there room for improvement? In order to automate you GUI your requirements have to be very detailed – meaning that you have to keep a high degree of documentation. Beware that a high degree of
documentation is hard work, it takes and cost time and resources to make and maintain. To document is a boring job to a lot of people. Make sure each requirement is review, documented and that responsibility for each requirement is placed on management level as well as end-user level. Make sure each requirement is pinned (tracked) to you automated GUI test (AGT).
 
The Impact and lessons learned in AGT phase
Test automation tools are much better than rumoured, some are even easy to learn, and end-users can be transformed into ‘tool super test users’. The very difficult part is to build an automated test process that works. You have to implement an ongoing process that exist in every day production supported by well-trained staff and well documented procedures. So yes, you can do regression test, but you will regret and never regress your test suites if you do not have total control of your process.

Initially we planned or expected that the end-user could take control of the entire AGT phase. Nevertheless, we must admit that we are still is in a 70/30percentage situation where the end-user controls the 70% and the IT-professionals 30% of the business domain test. Not entirely as expected but we game on that most of the 30% can be conquered by the end-users in the years to come. Anyway, we have moved that technological boarder.
 

The measure of the Quest objectives

By the way: lets have a look at the Quest objectives one more time to see how it turned out in the end:
 

The QUEST project objectives – Final Score


In re-use the results from the AGT process in subsequent business domain tests of new releases of our Web Banking systems. (UCIT, AGT)

Notice: We reached our objectives. Nevertheless, we could have done better in involving the customer. We did a form of usability test in our pre beta test bur I am afraid to say that we started out with greater plans for usability test. However the requirement phase ate more of our time and resources that planned and therefore we had to scale down the extent of the usability test.

The QUEST business objectives – Final Score:


To reduce the number of production errors by 33% during the first 6 months of a release as a result of a more thorough test and thus strengthen the quality of BG Bank's strategic area of growth

To reduce the production and the maintenance costs by at least 20% do to fewer errors.

To increase the percentage of Hot-Line replies to customer enquiry’s to 98%. This to take place through the staffs strong involvement in the business domain test and because fewer errors means fewer enquiry’s.

Release 10% of the BG Bank end-users in order for them to test future releases of the Web Banking system. This is to take place through fewer errors and thereby fewer customer enquiry’s
 

The QUEST Technical objectives – Final Score:

To form a business domain test of the Web Banking system complex which is end-user-oriented as far as domain-test is concerned and thus releasing at least 10% development test resources for development of new releases of the Web Banking system.

To gain experience and concrete results to be used for subsequent tests of the system complex. This includes test of the readjustment to Economic Monetary Union.

To describe and document a QUEST Best Practice for future and other end-user-oriented tests of the same character in a complex software environment.

To secure that future externally developed operational systems can be implemented in the existing end-user oriented system complex and be tested by means of the "Best Practice" described.

To secure that an automated GUI test process mainly is end-user controlled with quality support from IT professionals.

Notice: To be total honest I must admit that we are still in our measuring phase. So far it seems that we will reach the objectives and for some objectives beyond them.
 

The QUEST continues

So we can proudly say that we have a test process that involves and rely on The All European User and Customer - and it works! We did not use more resources on test than expected or as used on previously projects. We did implement a great quality product almost defect free. We did implement the product on time and our cost on maintaining the product equals our competitors.

Our QUEST has begun and it will continue always. We can and we will do it better, faster and cheaper. We still carry out four annual investigations of our system development process to constantly keep the focus on activities to improve. For the moment we work on a true ‘Release when you please’ process. This combine the strength of our automated Client/Server test with the ‘On the fly’ customer requests and demands that ties the Client/Server, midrange and mainframe together in a
test process that enables new releases to be build ‘over night’.
 

 References


[1] Jones Capers, Applied Software Measurement: Assuring Productivity and Quality. On the history and evolution of functional metrics pp. 43-122, McGraw-Hill,
Inc. (New York, N.Y.) 1991.

[2] Ould Martin A., Strategies For Software Engineering: The Management of Risk and Quality. On the 14 dilemmas of software engineering, pp 186-223, John
Wiley & Sons, Inc., (New York, N.Y.) 1987.

[3] Hertzel Bill, The Complete Guide to Software Testing. On testing methods and tools, pp 43-72, John Wiley & Sons, Inc., (New York, N.Y.) 1988.

[4] Vinje Poul Staal, Test af Software. Planlaegning og styring, pp. 253-320 and Vaetoejer pp. 359-381, Teknisk Forlag, 1993.

[5] Zahran Sami, Software Process Improvement. On launching implementing and measuring software process improvement, pp 183-232, Addison Wesley
Longmann, 1998.

[6] Fewster Mark, Grove Consultant, The Selection, and Use of Test Execution Automation Tool, 2 day Seminar at Delta, 1997

[7] Kolish Tony, Segue Software Inc., Ensuring Reliability of Web Applications: A white paper for IT managers, QA professionals and software engineers. The
paper was presented at the EuroStar 1997 conference in Edinburg.


Partners in EuroSPI

Editors
ISCN LTD, ISCN GesmbH, Schieszstattgasse 4/24, 8010 Graz, and Coordination Office, Florence House, 1 Florence Villas, Bray, Ireland, office@iscn.at, office@iscn.com, office@iscn.ie, Editing Done: 19.7.2002