SUPREME – a Statistical Approach to Support Project Estimation and Management
Rikke Sunde Event AS, Oslo, Norway
Tor Vidvei Event AS, Oslo, Norway
In project planning, a core issue is to get the best estimates possible on duration, resource usage and overall costs of the project. The objective of the SUPREME project is to improve planning, estimation, monitoring and measurement of software projects by applying a statistical approach in project estimation and measuring.
As a project is developing, resource usage, status changes and revisions of the estimates are registered daily or at certain other time intervals so that the current status and the latest forecasts can be displayed at any time. This registering is done for the elementary activities and resources, and automatically aggregated to higher levels. The history of all variables is saved, possibly with annotations, so that the entire history of the project may be analysed at later stages. All persons involved in a project are encouraged to add comments to their registrations, especially when estimates or status variables are changed.
Various aggregation rules are defined for the variables describing an activity. The resource usage of an aggregate activity is calculated as the sum of the resource usage in all of its sub-activities, the starting date is found as the first starting date among the sub-activities etc.
To be registered as an activity, the task has to be of a certain size, at least one working day. Activities, which could be defined as rather small, will also be registered, but only containing actual closing dates and with no estimates for resource usage. This will typically be an activity lasting less than one day.
The activities are also organised in a tree-like structure representing the various external and internal projects from a financial point of view, reflecting how the resources spent on the activity will be accounted for. Aggregates within this hierarchy are called accounts. An account may represent a particular contract while the sub-accounts may represent various sub tasks within this contract. The accounts are used for budgeting and reporting to the clients. Together with the activity hierarchy, the accounting hierarchy represents a two-dimensional WBS of the activities. This is especially important within an object-oriented framework where code is written for reuse, possibly across various external projects. For projects that are to be accounted for on a per hour basis, the accounting will not be in accordance with the physical organisation of the activities.
For each registration (data item), it is also possible to add annotations. Annotations should be added when they may be significant to the analysis of the project in later phases. (New variables can be added at later stages. If the annotations contain information that is needed in the statistical analysis, it is possible to create new variables and code the information.)
The variables may be aggregated following the tree-structure that the various activities or resources are organised within. The user controls the aggregation rules.
It is possible (and normal) to add new activities or resources during a project period. Such changes do not create any problem for the estimation and accounting routines in most cases. It is also possible to move and even delete an activity, but one should be aware of the following:
When an activity is moved from one place to another within a tree, its entire data history (even the data that are collected before the move) will follow and be aggregated within the new tree-structure, and consequently removed from the old tree-structure. The aggregates will display a history of its variables as if the activity has always been placed in its new structure.
When an activity is deleted, the user will have the opportunity to move its content to another activity. Despite the problems just mentioned this could be useful, e.g. in cases when a project plan appears to be too detailed as the project evolves. Then the activity tree can be "Cleaned" even after the project has started.
Say, one is told that over a period of time one systematically underestimates projects with about 20%, and that most of these mistakes are caused by tasks added during the project period. On the other hand, one could at the same time hit fairly well with the estimates for programming and documentation efforts. One may be helped significantly in performing corrective actions and improving future estimates by this feedback, even without a formal estimation model.
The need for status reports and statistics with focus on the comparison of estimates made and results achieved is enforced by the common experience of most software projects: the early estimates tend to contain a significant amount of "guesstimates", usually more than in other types of projects e.g. within construction. This element of guessing is more prevalent the more "Creative" or the less routinised the project is. Estimation relies on quantitative measures for the outcome of the project or its activities at some level (e.g. code lines, modules, reports, pages of documentation), assuming some kind of repetitiveness.
It should also be emphasised that even quantitative estimation models like the COCOMO model relies heavily on judgement or qualified guessing. Although the expected costs in man hours are calculated from the relation between lines of code and the expected resource usage, which is the result of statistical research, one does not really know the number of codelines in advance when one is going to estimate a new project. The element of guessing is not eliminated from the estimation process; it is rather isolated to some parts of the estimation procedure, in this case to the assessment of how many codelines that is required to fulfil a certain task. After all this procedure might in many cases prove better than relying entirely on qualified guessing, probably because predicting the number of codelines is simpler than predicting the costs directly. But you may also have cases where the opposite is true, namely that it is better to assess the costs directly rather than first trying to assess the number of codelines, then to select an appropriate estimation model and finally calculate the costs.
The method itself will usually be applied on historical data and not to perform the estimates (prediction) during the planning phase. This will make it possible to use the method to compare the estimates with actual recourse usage etc and thus give us a sort of feedback on the estimates.
This event history analysis is relevant while analysing event history data, i.e. data describing sequences of dated discrete events. A typical field of application for this method is demographical analysis, with the analysis of fertility and mortality depending on age and other factors as one of the core issues. Demographic data typically consists of individual "event histories" of birth, marriage, death, changes in health status etc. Another typical application of this method - which is also known as "failure time analysis" - is the analysis of the duration of products until a failure (or some other interesting event) occurs. The method focuses both on the analysis of which events or transitions that takes place, and on the duration between the various events, according to some timescale or any other variable that grows as time passes.
The "history" of a project can be described in similar terms as sequences of dated events associated with the individual activities within the project: registration, decision, start, completion, cancellation etc. We are interested in the transitions that takes place (what is the probability for an activity that has actually started, also to be completed?), and the duration according to various timescales and resource usage (how is actual resource usage and duration of activities distributed, compared to the original estimates?). We think that event history analysis is well suited to perform the analysis necessary to answer this type of questions.
In SUPREME we observe all events of interest for each activity, and aggregate them as individual event histories. A set of event histories is called event data. When aggregating the stories for a group of activities or a project, one will have a summery of the transition between several states. On the contrary the cross section data will only give information about how many activities that is in the state at any point of time.
An event history model is a schematic way of showing possible courses of events. The model used in SUPREME is containing 5 states: registered, approved, in progress, completed or rejected.
Suppose one is observing the events in a group of activities. The phenomenon that is of interest is "Completed". The event history for two activities may be as follows:
Fig. RSU.1: Example of event histories containing 5 events.
By registering these events we will have an exact registration for the time each event occurs.
A set of states as described above are characterised as a case where there exist more than one transition from one state (competing risks of e.g. being "Completed" or "Rejected"). There are several entrances to some states (all activities are under risk of being "Rejected" at any point of time). In addition it is likely to assume that the intensities are depending on more than one time scale e.g. calendar time (date), time in state (for example time from "Approved" to "In progress") or time in "Approved" and cumulative time in state (aggregated time an activity has been exposed to e.g. being rejected).
The possible states and transitions can be illustrated as follows:
Fig. RSU.2: Transitions and states
The boxes represent the possible states each activity can be in. An activity can not be in all of them simultaneously, as an activity can not be "In progress" and "Completed" simultaneously. The transition between the states is marked with arrows.
To illustrate what is happening in the transition, we can regard a group of activities in a project. As time goes by, all activities will be completed or rejected. At the start none of the activities will be in the state "Completed" or "Rejected", but as times go by an ever-increasing number of activities in the project will be in the state "Completed" or "Rejected". But after some time, all activities will reach these final states. In our terminology all activity that enter the registered-state will be the same activities which are entering the completed or the rejected-state.
The transitions between the states can be illustrated as a survivor function. An example of this is shown in figure 4. For a given number of activities, the graph shows the estimated probability for an activity still being ongoing at each point of time.
Fig. RSU.3: The survivor function
In this example an activity has a probability of 90% of still being ongoing after 7 months. After 10 months the probability of still being ongoing has fallen to 15 percent.
When an activity leaves for example the state "In progress", it will either reach the state "Completed" or "Rejected", which make it necessary to define the intensities for each of these events. At any point of time the activity in progress is being exposed to the risk of getting completed and to the risk of being rejected. In the terminology of event history analysis, there are competing risks for a transition to either "Completed" or "Rejected".
The use of event history analysis can be illustrated by one of the baseline projects - PETRA. We want to investigate how the resource usage in the project compares to our original estimates. In terms of the state diagram (fig RSU.2) we are going to investigate the transition from "In progress" to "Completed" or "Rejected". As the project is still running, we will use our last estimates for the resource usage and assume that all activities will be completed. (Obviously, when the project is completed, the final accounts on resource usage and status for each activity should be used instead of these revised estimates.) As we want to look at the actual resource usage compared to the estimates, we divide the resource-usage (actual or last estimate) by the original estimate – and use this as our "timescale". (See the last column in the table RSU.6) As the size of each activity varies, the data for each activity should also be weighted with the original estimates (taken as a measure of the "size" of each activity).
Table. RSU.4: The PETRA project
Table RSU.5: The Event history data for the transition from
"In progress" to "Completed"
In the RSU.5 the "Weight" column is containing the original estimates for each activity. The tE column shows the resource-usage relative to the original estimates. In the Occ column a 1 or 0 indicates that the activity was- or was not completed.
The event history data derived from this is displayed in table RSU.5. Data for the aggregates are excludes in order to avoid counting the same activity multiple times. A hazard function and an associated "survivor function" expressing the probability distribution for the resource usage relative to the estimates, can be estimated from these data. The survivor function is displayed in figure RSU.6:
Fig RSU.6: Survivor function for the transition from "In progress" to "Completed"
The figure indicates that the project is somewhat underestimated (the survivor curve is "skewed" to the right in the figure - the area above the survivor curve to the left of t=1 is less that he area below the curve to the right of t=1. (The average resource usage is 15% higher than the estimates.) Further, the resource usage compared to estimates varies between 0.40 and 1.75.
This type of figure gives in a glance a summary of the performance of the project. The figure RSU.7 contains some other stylised examples of "survivor functions".
Fig RSU.7 Examples of survivor functions
Graph 1 and 2 both indicates that estimates are unbiased, as the average time to complete an activity is equal to the estimated time. But graph 2 indicates less uncertainty: the curve is closer to the t=1 line on both sides. Graph 3 indicates that estimates are biased: for most activities and on average the time to completion is longer than estimated in advance. Graph 4 also indicates that the average time to completion is longer that the estimates, but in this case most activities are completed within the estimated time.
As mentioned above each activity that is "In progress" will be under risk of being "Rejected" or "Completed" at any point of time. On this basis one can estimate a hazard rate for each of these risks to illustrate the probability of an activity being "Rejected" vs. "Completed".
One can obviously assume that an activity’s progress depends on several explanatory variables. This could be e.g. the activity-size and -type. Being aware of this may help to identify critical activity types or other states, which are sources of uncertainty (or risk).
Meanwhile one has to consider what will happen if one is facing major deviations from the original schedule. When an activity is running out of time, does that means that one will use all available (and not available) resources on that particularly activity, forgetting the other activities. And what if an activity is doing more than just fine, so it will be completed "too early"? Will one still use all the planned resources just to get oneself a well-deserved breathing space?
[2] H-P. Blossfeld, G. Rohwer: Techniques of Event History Modeling, LEA, Mahwah, New Jersey, 1995.
[3] J. E. Matson, B. E. Barrett, J. M. Mellichamp: Software Development Cost Estimation Using Function Points, IEEE transactions on software engineering, Vol. 20 no 4, April 1994.
[4] C. A. Behrens: Measuring the productivity of computer systems development activities with function points, IEEE transactions on software engineering, Vol. SE-9, no 6, November 1983.
[5] Breien, Andersen, Jonassen, Stålhane, Urstad: Estimering- og fremdriftsmålemetoder for utvilkingsprosjekter, ITUF R-29, 1992.
The COCOMO method is based on inputs relating to the size of the resulting systems and a number of "Cost drivers" that affect the productivity. The cost function in COCOMO can be written:
Cost = a • LOCb • ?fi,
where "Cost" is the total project cost, "a" and "b" are constants and dependent on the development modus only, "LOC" is the size of system, measured by Lines Of Code and " fi" are the cost drivers. The COCOMO model does estimate the cost of production from the system size (measured by lines of code (LOC)), the type of development modus (organic, semidetached and embedded) and the cost drivers. Examples of cost drivers that affect the productivity are product attributes (data base size), computer attributes (execution time constraints) and personnel attributes (programmer capability)
Unfortunately the COCOMO model does not provide an opportunity to balance the interests of total cost and the time of realisation in a project. The uncertainty of the estimate will be enlarged as a result of implementing the cost drivers.) ….til hit!
As method to be used to make proper estimates the function point analysis seems to be the most relevant. This method which is quantifying the size and complexity of a software system in terms of the functions that the system delivers to the users, is unrelated to the language or tools used to develop a software project. As for projects, which are based on object orientations, this is of great importance
To measure productivity, a product and a cost is defined and measured. The product is the function value delivered to a user. The number of inputs, inquires, outputs, master files and interfaces delivered are counted, weighted, summed and adjusted for complexity. The collection of function point data has two primary motivations. One is the desire to monitor levels of productivity, for example number of function points achieved per work hour expended. Another use is in the estimation of software development cost.
Appendix 2: Presentation of the authors
Rikke Sunde: Cand Polit (Master of Science degree in economy), born 1969
Education:
Tor Vidvei: Cand oecon (profession oriented degree in economics), born 1955
Appendix 3: Presentation of the company
Event AS is a small enterprise developing software systems for statistical modelling. The applications are tailor made for each customer, while the development process relies heavily on the use of a software library called EVENT, which is being developed continuously as part of each of the application projects. This approach communicates the management of the development processes in many ways, especially due to the ties and dependencies between the different application projects that are crated through the use of the software library.
The company presently has three employees.
Among our major projects are: