ARCHIVED - Management of Large Public IT Projects - Canada

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

2. Case Studies

2.1 The 1995 Auditor General's Report

The first of the cases referred to in this section is the 1995 Report of the Auditor General of Canada, referred to in the first section of this paper. The report was entitled "Systems Under Development – Managing the Risks", and it identified some of the problems (and some of the successes) the government had in managing large systems development projects at that time, and presented some recommended solutions. This section covers the highlights of the report.

Audit scope

The audit examined four major systems then under development. These systems were chosen for review because: a) they were complex, long-term initiatives that would take several years to implement; b) they would either affect the infrastructure of government, or have an important impact on the operational capabilities of a department; and c) they represented a significant investment on the part of the government.

The findings of the audit were presented under three main headings: project management and monitoring; the nature of large IT projects; and environmental factors.

Project management and monitoring

The audit found a number of management-related weaknesses associated with the introduction of the systems. These were as follows:

• Inadequate analysis of underlying business issues: for example, major projects had been launched without full understanding of the complexity of the business environment, and without resolving the business issues which were the real root cause of the problems the system was intended to solve.
• Inconsistent support from management and project sponsorship: in fact the importance of the project sponsorship role was not widely recognized in government at that time.
• A lack of experienced resources on project teams: in all four projects, project teams started out without sufficient experience of projects of similar size and complexity, and in some cases without experience of the associated technologies.
• Inconsistent user involvement and acceptance: in one of the projects, user involvement had been very effective, and as a result there was widespread support for the project among users at all levels. In another project user involvement was less well addressed, and the resulting system ended up with fewer users than expected. Other potential users had not been convinced to convert from their existing systems.
• A lack of effective ongoing monitoring of systems under development: projects were able to produce adequate status data, although this was not always possible right from the start of the project. However issues such as the currency and continued relevance of the business case for the system were not so well monitored.

The nature of large IT projects

The audit noted several factors related to the inherent nature of large multi-year, multi-million dollar information technology projects that influenced the risk of successful implementation. These were as follows:

• Project size: the risk of failure of a systems project increases disproportionately with increased size and complexity of the system. It is important to try to measure this before development starts. One method of measurement is Function Point Analysis. Any system containing more than 5,000 "function points" is considered complex; research had shown that any project of more than 10,000 function points had about a 50% chance of being cancelled before completion. The estimated size of the systems reviewed in the audit was between 14,000 and 16,000 function points (in one case after previous scaling down).
• Technical complexity: One of the four projects was based on a commercial software package. This should have reduced the complexity; however the extent of modification carried out to meet departmental needs significantly increased the complexity. The other three systems were custom-developed and had aspects that were technically very complex.
• Risk associated with uncertainties about functional requirements: in each of the four systems reviewed, the functional requirements had been articulated at the time the project had been approved and initiated. However, as all four projects progressed, it became clear that the requirements, as originally stated, needed to be either restated, refined or more clearly communicated. The impact of any uncertainty about requirements affects both project budgeting and project approval: the scope may have to be reduced to stay within budget. It was apparent from the audit that the inherent nature of large, multi-year projects did not allow project sponsors to predict with confidence what the ultimate cost of the project would be.

Environmental factors

The business environment within which the government operates is continuously changing. At the time of the audit, the number of changes and the pace at which they are occurring, both globally and within government, had sometimes outpaced the ability of systems developers to respond to them. The audit identified two key environmental factors, as follows:

• A changing business environment: for each system that was reviewed, the business environment, under which the initial decision to build the system was made, was no longer relevant. It is not possible to successfully implement large systems that take years to develop, in a situation where the business environment is continuously changing.
• Advances in technology: while some of the projects were under way, user expectations expanded dramatically in response to the availability of new and emerging technologies. The developers were not able to keep pace with these growing expectations.

Conclusions and recommendations

The audit identified many factors that affect the successful development and implementation of systems. Those considered among the most critical to the successful management of project risks were:

• effective project sponsorship;
• clearly stated requirements;
• effective user involvement in and commitment to the success of the project; and
• the expertise and experience of resources dedicated to the project.

The auditors recommended that an integral component of any new approach must focus on the implementation of long-term information technology strategies through smaller, more manageable components, each of which provided an improved capability (efficiency and/or effectiveness) to the organization.

Such an approach would provide certain key benefits:

• it would be easier to more clearly define requirements for a smaller component;
• requirements would be less likely to be affected by changes in business environment;
• more complete and accurate estimates of costs and schedules could be developed; and
• it would be easier to obtain project resources with appropriate levels of experience.

Follow-up Reports

Since the 1995 report, the Auditor General has continued to review large government IM/IT projects, and has also followed up the status of those projects reviewed previously, as well as tracking the implementation and application of the Enhanced Management Framework. Four more major systems under development were reviewed in 1996, and three in-house projects in 1997. Follow-up reports were published in 1998 and 1999. By 1999, 16 out of 20 large departments had submitted plans to TBS for the implementation of the EMF, and most of the Auditor General's recommendations for corrective actions on individual projects had been acted on.

• Previous Page
• Table of Contents
• Next Page