ARCHIVED - Management of Large Public IT Projects - Canada

This page has been archived.

An Enhanced Framework for the Management of Information Technology Projects

Management of Large Public IT Projects

October 26-27, 2000
Chief Information Officer Branch
Treasury Board of Canada Secretariat

Introduction

This report has been prepared by the Enhanced Management Framework Division of the Chief Information Officer Branch, Treasury Board of Canada Secretariat, in support of the OECD project on Management of Large Information Technology (IT) Projects in the Public Sector.

The OECD project aims at finding out what it takes for Governments to secure success in their management of large public IT investments, and what lessons can be learnt from past failures. It addresses the issues of governance, monitoring and risk management of such projects.

This country report is structured into four sections. Section 1 provides a synopsis of the general institutional framework put in place by the Canadian government to improve the success rate of Information Management/Information Technology (IM/IT) investments. Section 2 of the report contains a number of case studies which provide actual examples of successes and failures, while Section 3 identifies the key lessons learned from recent projects on how failures can be avoided and success facilitated. Finally Section 4 contains references to further material on the management of large public IT projects.

1. General Institutional Framework

1.1 Background

The Canadian government is committed to deliver its programs and services more efficiently and effectively through the use of information technology. In 1994 the Treasury Board Secretariat published a "Blueprint for Renewing Government Services Using Information Technology". Since that time, the development and rapid growth in use of the Internet has revolutionized approaches to IM/IT usage. The government recognizes the need to adapt its services to the new technology. In the October 1999 Speech from the Throne, the government committed to having all its information and services on-line by 2004. The government's goal is "to be known around the world as the government most connected to its citizens, with Canadians able to access all government information and services on-line at the time and place of their choosing."

However, for some time past, failures of major information technology investments and key systems development projects have raised concerns for the achievement of service improvement through information technology. In 1995, in recognition of these concerns, the Treasury Board Secretariat (TBS) carried out a review of 25 significant government IM/IT projects, with a total estimated cost of $2 billion. The review identified business, project management, risk management, and human resource issues influencing the outcome of these projects.

Also in 1995, the Auditor General of Canada, as part of his annual review of government, studied four large systems development projects, with a combined value of $490 million, being undertaken by government departments, and found that only one of the four was being managed in a way that dealt effectively with the risks associated with the project.

The report described the results of these reviews, and also quoted private sector research (the Standish Group's CHAOS research) indicating that the likelihood of large multi-year systems development initiatives being completed on time, within budget and with the desired functionality (what the system should do for its users) in the public and private sectors was extremely low. In 1994, the chance of a project developed by a Fortune 500 company coming in on time and within budget was only 9%. At that time 31% of IM/IT projects surveyed by the Standish group in both the private and public sectors were cancelled before completion. Subsequent CHAOS research shows that higher proportions of projects are being completed successfully, but the proportion cancelled has remained about the same.

The focus of reviews by the Treasury Board Secretariat and the Auditor General was on large IM/IT projects. These may be defined by a number of criteria, for example:

• with a value over $10 million;
• taking 18 months or longer to implement;
• involving extensive business or organizational change;
• involving significant new technology and/or technical complexity;
• requiring implementation in numerous and widely dispersed locations, or across more than one organization.

However, it was evident that the conclusions emerging from the reviews of large IM/IT projects were applicable to all sizes of project in this area. These reviews led to a reexamination of the government's policy and procedures for the management of systems development projects, and ultimately to a new focus on the management of IM/IT investments in general - portfolio management.

1.2 Enhanced Management Framework for IM/IT (EMF)

In 1996 the Treasury Board Secretariat developed the Enhanced Framework for the management of IT projects, which has since evolved to become the Enhanced Management Framework for IM/IT (EMF). The objective of the EMF is to help departments better manage their IM/IT projects and improve the success rate of IM/IT investments. Departments were directed to apply this Framework to all projects that have a significant information management or technology component, regardless of size. The EMF was designed to ensure that government information technology projects fully meet the needs of the business functions they are intended to support, deliver all expected benefits and are completed on time and on budget.

Subsequently the EMF has expanded from focussing solely on project management disciplines to include portfolio management, which addresses the overall management and governance of IM/IT investments. This helps improve the strategic use of IM/IT investments through a stronger alignment with business directions and priorities.

EMF enhances the government's ability to manage its IM/IT investments, successfully deliver IM/IT projects, and minimize risks. It provides for a more effective, efficient and successful management of IM/IT allowing for optimal selection of high-return IM/IT investments, balanced decision-making, comprehensive risk management and successful delivery of projects.

EMF is an integrated management model that includes processes and key practices for executives, as well as for business and project managers. The framework is supported by a set of principles, best practices, methodologies, tools, templates, handbooks, guides, and standards.

The conceptual model above shows the components of the EMF, and the way in which they are related.

EMF is based on four guiding principles:

1. Alignment of IM/IT investments with business strategies;
2. Establishment of clear accountabilities for managing IM/IT investments;
3. Development of corporate project management disciplines; and
4. Identification and management of risks on a continuous basis.

EMF addresses two broad areas: portfolio management and project management. Portfolio management stresses the importance of aligning business planning with an integrated IM/IT strategy. This strategy should set priorities and budgets for the organization's IM/IT investments as a whole, allowing it to assess and successfully manage projects, existing operations, enhancements and innovative pathfinders. Within this context, organizations should review their IM/IT investments and – based on this review and available funds – select those investments that will deliver optimal value. EMF also promotes the application of project management disciplines to all approved initiatives, as well as the implementation of risk and performance management throughout the entire process.

The implementation and the institutionalization of EMF across government is led by an Implementation Council – a government-wide partnership with representatives from 28 departments, the Office of the Auditor General, and central agency functional areas like policy, risk, audit and procurement. The EMF Division within the CIO Branch of TBS, is a centre of excellence assisting departments in their EMF implementation efforts. In partnership with the Implementation Council, it evolves the framework, rolls out well-researched solutions and toolkits, promotes and shares best practices, conducts symposia and workshops, addresses policy issues, and provides leadership, advice and support.

Alliances have been formed with leading international standard setting organizations, U.S. government agencies, professional associations, educational institutions, and central agencies. Special Interest Groups are created to facilitate sharing of experiences, best practices, and lessons learned on specific subjects and issues such as risk and process improvement. Ad-hoc working groups are set up to help develop specific solutions, including templates, tools, guidelines and methodologies.

Details on the EMF and its suite of solution sets can be obtained from http://www.tbs-sct.gc.ca/emf-cag/index-eng.asp.

Since September 1997, all new projects submitted by departments for approval by the Treasury Board Secretariat have been required to conform to the best practices in the Enhanced Management Framework. In addition, the Secretariat applied the appropriate principles from the Framework to all Year 2000 projects.

1.3 Procurement philosophy

The government has adopted an innovative new approach to buying goods and services to help ensure the success of complex IT acquisition projects, traditionally characterized by significant risk. The approach is called Benefits Driven Procurement (BDP), stressing a focus on results, on the benefits that the government – but also its suppliers – must gain from each acquisition project. BDP is a philosophy more than a method, although its aims are achieved through very methodical means. Although developed to solve problems with information technology acquisitions, the BDP has a broad application and is relevant to a wide range of high-risk procurement endeavours.

The BDP approach is designed to avoid the pitfalls that beset many complex projects – the delays, cost overruns and end results that often fall far short of expectations. It is a new concept for today's new era of rapid-fire technological change, competitive marketplaces and pressures on governments to be more efficient and effective than ever before.

BDP addresses many of the problems related to traditional procurement approach by focusing on the overall desired outcomes rather than on detailed requirements. Basically, the BDP approach is to ask suppliers to deliver certain agreed upon results rather than follow a blueprint.

The other key distinguishing feature of BDP is thorough and rigorous front-end planning to remove or mitigate potential problems in the procurement process. Both the front-end planning and the management of the entire acquisition life cycle are based on four elements: i) a solid business case; ii) risk analysis; iii) clear delineation of accountabilities; and iv) a compensation structure closely tied to the supplier's performance.

1.4 Funding

The majority of government IM/IT projects are funded from the budgets of their sponsoring departments. Departmental budgets are established and approved by Parliament through the government's Budget and Estimates process.

Certain IM/IT applications used by more than one department, typically financial management and human resource management systems, are designated Shared Systems. The Chief Information Officer Branch of the Treasury Board Secretariat oversees the whole Shared Systems program, including the acquisition of system licences for multi-departmental use. Each system is managed by a Cluster Group made up of representatives from each department that uses the system. The Cluster Group is responsible for managing the common maintenance and enhancement of the core system, although departments can still make their own individual enhancements to the core. The activities of the Cluster Group are generally funded from individual departmental budgets, by contributions from the cluster member departments.

In some cases, government-wide IM/IT initiatives, which apply to all departments, are funded centrally. These initiatives are normally administered by Treasury Board Secretariat, and departments can get access to funding via a Treasury Board submission (as described in the next section) demonstrating how their proposed expenditures meet the objectives of the government-wide initiative. A current example of a government-wide initiative is the Strategic IM/IT Infrastructure (SII) Initiative. This is an interdepartmental program that will design and deliver a common information technology and policy foundation to government departments and agencies. It will act as the backbone for Electronic Service Delivery and will support key service improvement initiatives such as Government On-Line.

Departments receiving funding from Treasury Board Secretariat for IM/IT projects as part of a government-wide initiative are normally required to set up a Memorandum of Understanding (MOU) with TBS, which sets out the objectives of the projects and identifies how project success will be measured. The MOU specifies how often and in what form departments must report back on project progress to TBS, so that the value obtained for the funds disbursed can be monitored.

1.5 Decisions and Assessment

It is government policy that all IM/IT and capital projects, with a total estimated cost above the level that the sponsoring Minister of the department concerned can approve, must be approved by Treasury Board (the committee of government ministers set up to review and approve expenditures). This applies even though the project may be funded from the department's authorized budget. Approval is sought by the preparation of a Treasury Board Submission, which is thoroughly reviewed by officials in the Treasury Board Secretariat before being submitted to Treasury Board Ministers.

All projects submitted for approval must be supported by documentation that adequately describes the full scope of the project including the associated management framework. All submissions relating to IM/IT are reviewed by the Portfolio Management Division (PMD) of the CIO Branch, in a business-team setting to ensure that they represent an effective and efficient solution to the operational needs as set out in the department's defined priorities or in the Long-term Capital Plan. PMD also reviews for compliance with the EMF to help departments and agencies meet their projected budget and scheduled time lines.

The approval process normally consists of two stages: Preliminary Project Approval (PPA) and Effective Project approval (EPA). Departments normally request PPA when the initial project planning and identification phase is completed but before the project definition phase starts. The formal Treasury Board approval process may be tailored to individual projects and departments, depending on the nature of the risks involved in those projects.

The requirements for PPA are as follows:

Proposal: It must list all authorities being sought from the Treasury Board, including:

• the cost objective for the project definition phase, which will establish the total amount of funds approved by the TB for the purpose of defining the project;
• any other objectives deemed to be sufficiently critical to require specific authority by the TB.

Supporting documentation: consisting of the following:

• A background section identifying the program's scope and requirements and providing justification that it directly relates to the department's or agency's goals and responsibilities;
• Indicative budget and total project cost estimates (total cost and annual cash-flow) for the overall project;
• An overall project schedule which should provide an estimated schedule of milestones for the overall project, a work breakdown structure, and a sequential implementation plan for all products/releases;
• The project management approach;
• The Enhanced Management Framework questionnaire (used to assess that all appropriate elements of the framework have been addressed);
• The Project Charter;
• A Business Case;
• A summary of comprehensive cost-benefit and options analyses;
• A risk assessment and mitigation strategy;
• A Human Resource plan;
• A Communications plan;
• An outstanding issues section;
• Gating of the project (if it is large enough to warrant it);
• Other objectives, such as
• Schedule objective
• Performance objective
• Procurement strategy.

Under certain circumstances, PPA may be deemed unnecessary. This is acceptable so long as all the conditions required for PPA are met within the EPA submission.

The EPA submission must include the above PPA requirements, and in addition:

• Any updates to the PPA documents;
• Substantive budget and total cost estimates (total cost and annual cash flow) for the overall project; and
• Full details of any agreements associated with the project; and
• Explanations for any variances between PPA and EPA schedules, costs etc.

After receiving Project Approval, departments have reporting obligations to the Treasury Board Secretariat. These follow the "gating" concept. Gates are significant completion events or quality milestones, placed at key points in the project lifecycle. Gates assess either the quality of the products produced so far, or the adequacy and completeness of the process to date, and a gate can only be "passed" if the products or process meet a predefined performance standard. Gates may take the form of technical reviews, risk assessments, completion of documents, demonstrations or test cases, or project audits. Gates are identified in a project's plan/schedule, and a gate review is required to formally "pass" each gate.

If, at a gate review, the gate's criteria have not been met, it is possible that a significant change in project direction may need to take place to address the lack of performance. The project should not continue "as is" unless each gate is passed.

Once projects are completed, audits are normally carried out by departmental review branches, which report directly to senior management. On a government-wide basis, projects of significant scope and/or cost may be reviewed by the Auditor-General of Canada. Large and/or long-term projects may be subject to audits while still in progress.

1.6 Management Models

As is evident from the preceding sections, the central agency (Treasury Board) is responsible for setting overall departmental budgets, approving individual IM/IT projects where they are of sufficient size, and monitoring expenditures and project performance on an overall basis. Otherwise, individual departments are completely responsible for managing their own IM/IT projects. The only exception is in the case of government shared systems, where the departments have established cluster groups to carry out joint management of common system maintenance and enhancement projects.

The Enhanced Management Framework describes a governance structure that is to be used government-wide and by departments to select and manage IM/IT projects. This governance structure satisfies the key factors for project success as identified by the Standish Group research: early agreement on project requirements; commitment and involvement of the user community; and commitment, attention and decision-making by top executives. A sound governance structure is expected to help government avoid typical project failures.

The EMF governance structure consists of four elements: planning and governance at the portfolio level; a business case; a project charter; and a review schedule tied to gates for each project.

Planning and Governance at the Portfolio Level identifies the organization' s process for selecting, prioritizing, and monitoring each project in the context of the organization's overall objectives and business priorities.

The Business Case puts the investment decision in a strategic context and identifies the business objectives and options that will affect both the decision and the investment itself. It provides the information necessary to decide whether a project should proceed. It provides an analysis of all the costs, benefits and risks associated with a proposed investment and with the reasonable alternatives to the proposed investment.

The Project Charter is a signed agreement between all stakeholders that defines the objectives, roles, responsibilities and level of participation of each stakeholder.

The Review Schedule tied to Gates establishes in the project charter the project review gates. The review gates are the major decision points of a project – to continue or to walk away. For each gate, the deliverables for that review, the type of review, the stakeholders responsible for reviewing each deliverable and the appropriate approval authorities must be defined.

Stakeholders in the governance structure are:

• Business Program executives and Information Technology executives responsible for strategies, priorities, and decisions;
• Business Program managers who implement information technology solutions in their program delivery value chains;
• Information Technology process support teams who provide the common standards and infrastructure for projects; and
• Information Technology project teams who need an action-ready project infrastructure so that they can concentrate on delivery of the business solution.

1.6.1 Departmental Management Model

A typical governance organization structure for a large department comprises a hierarchy of committees and working groups with specific responsibilities. One such department has the following structure.

Information Management Committee (IMC): a committee of Assistant Deputy Ministers responsible for IM/IT project approval and policy setting; reports to the departmental Business Board (the senior decision-making body).

Business, Information and Technology Alignment Sub-Committee (BITASC): a Director General/Director-level committee which identifies business and IM/IT support requirements from the various branches, and is responsible for ensuring the development of IM/IT plans, policies and practices in support of business and technology integration.

Infrastructure Sub-Committee (ISC): a Director General/Director-level committee responsible for identifying and co-ordinating IT support infrastructure objectives for the department, and assessing impacts of new technology developments/changes.

These committees are supported by a variety of lower-level committees and working groups addressing areas such as the Internet, security, architecture, and office automation.

The use of sub-contract staff on government IM/IT projects is increasingly common, either working in project teams alongside government employees, or as a complete project team where the whole project is contracted out. It is more and more difficult for departments to retain highly specialized IT staff in the face of strong demand for resources from the private sector. The rapid development of new technologies can render knowledge and experience obsolete after only a short time, and the cost of constant updating of knowledge is high. It is recognized that with outside staff working on the delivery of IM/IT projects, the importance of thorough application of the principles and practices of the Enhanced Management Framework, especially as regards user ownership and involvement, is even more critical.

1.6.2 Central Agency IM/IT Governance

A number of other committees and individuals operate within Treasury Board Secretariat to give advice and guidance on IM/IT projects.

TIMS (TBSAC Information Management Subcommittee): TIMS is a subcommittee of the Treasury Board Senior Advisory Committee (TBSAC). Its membership is comprised of several Deputy Ministers (heads of government departments). TIMS mandate is to provide leadership in the search for better service to the public and improved productivity through the use of IM/IT. This senior management committee represents the business end-users of information technology, and can ensure that the government's strategies for the deployment of IM/IT (including the EMF) meet government-wide requirements from a business perspective.

Government Chief Information Officer (CIO): The CIO is responsible for determining and implementing a strategy that will accomplish government's IM/IT goals. The role of the CIO includes:

• providing leadership, co-ordination and broad direction in the use of IT;
• facilitating enterprise-wide solutions to horizontal IT issues;
• serving as technology strategist and expert advisor to Treasury Board Ministers and senior officials across government.

ACIM (Advisory Committee on Information Management): ACIM provides advice to the CIO and TIMS on IM/IT issues. ACIM members are heads of IM/IT from large departments. ACIM, chaired by the CIO, provides feedback to the EMF program to ensure that the EMF continues to be aligned with government strategic objectives. ACIM members promote and fund the use of the EMF within their departments.

IMB (Information Management Board): IMB is an oversight body, recently set up by the Treasury Board Secretariat, for its strategic IM/IT infrastructure initiative. Members are selected from senior management across government. The IMB is chaired by the CIO and has a mandate for making decisions and overseeing the adoption and implementation of a government-wide IM/IT infrastructure. The IMB reviews and approves proposals from departments seeking central funding for projects that contribute to this objective.

2. Case Studies

2.1 The 1995 Auditor General's Report

The first of the cases referred to in this section is the 1995 Report of the Auditor General of Canada, referred to in the first section of this paper. The report was entitled "Systems Under Development – Managing the Risks", and it identified some of the problems (and some of the successes) the government had in managing large systems development projects at that time, and presented some recommended solutions. This section covers the highlights of the report.

Audit scope

The audit examined four major systems then under development. These systems were chosen for review because: a) they were complex, long-term initiatives that would take several years to implement; b) they would either affect the infrastructure of government, or have an important impact on the operational capabilities of a department; and c) they represented a significant investment on the part of the government.

The findings of the audit were presented under three main headings: project management and monitoring; the nature of large IT projects; and environmental factors.

Project management and monitoring

The audit found a number of management-related weaknesses associated with the introduction of the systems. These were as follows:

• Inadequate analysis of underlying business issues: for example, major projects had been launched without full understanding of the complexity of the business environment, and without resolving the business issues which were the real root cause of the problems the system was intended to solve.
• Inconsistent support from management and project sponsorship: in fact the importance of the project sponsorship role was not widely recognized in government at that time.
• A lack of experienced resources on project teams: in all four projects, project teams started out without sufficient experience of projects of similar size and complexity, and in some cases without experience of the associated technologies.
• Inconsistent user involvement and acceptance: in one of the projects, user involvement had been very effective, and as a result there was widespread support for the project among users at all levels. In another project user involvement was less well addressed, and the resulting system ended up with fewer users than expected. Other potential users had not been convinced to convert from their existing systems.
• A lack of effective ongoing monitoring of systems under development: projects were able to produce adequate status data, although this was not always possible right from the start of the project. However issues such as the currency and continued relevance of the business case for the system were not so well monitored.

The nature of large IT projects

The audit noted several factors related to the inherent nature of large multi-year, multi-million dollar information technology projects that influenced the risk of successful implementation. These were as follows:

• Project size: the risk of failure of a systems project increases disproportionately with increased size and complexity of the system. It is important to try to measure this before development starts. One method of measurement is Function Point Analysis. Any system containing more than 5,000 "function points" is considered complex; research had shown that any project of more than 10,000 function points had about a 50% chance of being cancelled before completion. The estimated size of the systems reviewed in the audit was between 14,000 and 16,000 function points (in one case after previous scaling down).
• Technical complexity: One of the four projects was based on a commercial software package. This should have reduced the complexity; however the extent of modification carried out to meet departmental needs significantly increased the complexity. The other three systems were custom-developed and had aspects that were technically very complex.
• Risk associated with uncertainties about functional requirements: in each of the four systems reviewed, the functional requirements had been articulated at the time the project had been approved and initiated. However, as all four projects progressed, it became clear that the requirements, as originally stated, needed to be either restated, refined or more clearly communicated. The impact of any uncertainty about requirements affects both project budgeting and project approval: the scope may have to be reduced to stay within budget. It was apparent from the audit that the inherent nature of large, multi-year projects did not allow project sponsors to predict with confidence what the ultimate cost of the project would be.

Environmental factors

The business environment within which the government operates is continuously changing. At the time of the audit, the number of changes and the pace at which they are occurring, both globally and within government, had sometimes outpaced the ability of systems developers to respond to them. The audit identified two key environmental factors, as follows:

• A changing business environment: for each system that was reviewed, the business environment, under which the initial decision to build the system was made, was no longer relevant. It is not possible to successfully implement large systems that take years to develop, in a situation where the business environment is continuously changing.
• Advances in technology: while some of the projects were under way, user expectations expanded dramatically in response to the availability of new and emerging technologies. The developers were not able to keep pace with these growing expectations.

Conclusions and recommendations

The audit identified many factors that affect the successful development and implementation of systems. Those considered among the most critical to the successful management of project risks were:

• effective project sponsorship;
• clearly stated requirements;
• effective user involvement in and commitment to the success of the project; and
• the expertise and experience of resources dedicated to the project.

The auditors recommended that an integral component of any new approach must focus on the implementation of long-term information technology strategies through smaller, more manageable components, each of which provided an improved capability (efficiency and/or effectiveness) to the organization.

Such an approach would provide certain key benefits:

• it would be easier to more clearly define requirements for a smaller component;
• requirements would be less likely to be affected by changes in business environment;
• more complete and accurate estimates of costs and schedules could be developed; and
• it would be easier to obtain project resources with appropriate levels of experience.

Follow-up Reports

Since the 1995 report, the Auditor General has continued to review large government IM/IT projects, and has also followed up the status of those projects reviewed previously, as well as tracking the implementation and application of the Enhanced Management Framework. Four more major systems under development were reviewed in 1996, and three in-house projects in 1997. Follow-up reports were published in 1998 and 1999. By 1999, 16 out of 20 large departments had submitted plans to TBS for the implementation of the EMF, and most of the Auditor General's recommendations for corrective actions on individual projects had been acted on.

2.2 Integrated Resource Management System (IRMS) Program

The Integrated Resource Management System (IRMS) integrates human resources, finance and materiel management services for the Canadian House of Commons. Implemented within budget over a 14-month period, IRMS has fundamentally changed the way the House works by providing Members of Parliament and the Administration with single-window Intranet access to enhanced tools, services and information.

Objectives and scope

A 1997 review of the services and systems of the finance, materiel management and human resources functions of the House of Commons concluded that existing systems were limited in their long-term capability. The seven existing systems were stand-alone and centralized, and required duplicate data entry and data reconciliation. The age and limitations of the technology employed meant that essential requirements could not be met efficiently or at reasonable cost.

Management identified the objectives of developing a new integrated management system that would: align and streamline finance, materiel and human resource management services; transform administrative staff from transaction processors to service providers; and provide clients (Members and managers) with secure desktop access to timely, accurate and integrated information on budgets, purchases, assets and staff costs, plus information management tools.

The IRMS Program was supported by a sound governance structure led by two project sponsors. From the outset, a key to the program's success has been the active participation of both clients and functional users (i.e. management and staff of the finance, materiel management and human resource functions) in all stages from planning the overall strategy to implementation of sub projects. More than 200 clients and users participated in extensive consultations, and many performed a variety of roles in individual IRMS projects.

Costs and benefits

IRMS was the largest investment made to date in people, tools and services for Members and the institution. The business case projected the total cost to be $13.6 million over five years. If the cost of maintaining the existing systems ($5.4 million) is offset, the net new funding required was $8.2 million. Planned savings were $2.2 million in administrative costs.

Qualitative benefits delivered to date include: facilitating House-wide information management policy; building linkages between policies and procedures across functions; and making available tombstone information for a corporate web-enabled telephone directory. Members will enjoy single-window Intranet access to frequently used forms, and "next day" budget information. On-line ordering and payment for purchases is the next step.

Critical success factors

The impact of the IRMS program has been far-reaching in its effect on internal policies and procedures. Recognizing the potential impact of these changes on people, the program plan set out a comprehensive strategy for managing change. This included targeted communications, training and user support plans. One priority was to provide a seamless transition from old to new with minimal disruption for Members.

Management of the program required a special blend of experience and skill. The program was complex, requiring a huge operational transition from old to new systems of financial management, coding and fiscal year reporting. IRMS integrated the services, information systems and cultures of three previously discrete functions, and included many external players.

The governance of IRMS reflected the complexity of this challenge. Initially an Ad Hoc team of the House Management Forum was set up. Since then governance has shifted structure and focus to meet the program's evolving needs, from consultation, inclusiveness and dialogue, to the practicalities of implementation.

The responsibility for day-to-day management of IRMS has been shared, reflecting the scope of the challenge. An in-house manager provided essential knowledge about the House, its organization, key players and management processes, program functions and requirements, while outside consultants offered technical and functional expertise in implementing large-scale systems.

Risk Mitigation Strategy

The IRMS program was regarded as a high-risk project. It involved a significant investment of resources over a number of years; and its scope was virtually unparalleled in the history of the House of Commons. Risk management was an essential component of the management process. The risk management strategy included the following major elements:

• The overall strategy identified risk management as a critical element of the program as a whole as well as for individual projects.
• A formal risk assessment was conducted of the IRMS Software Evaluation Project. It identified approximately 50 program risks, as well as appropriate risk mitigation strategies.
• A comprehensive business case review was undertaken of various options for addressing the objectives of IRMS. The review included a detailed assessment of the risks associated with each of four main options.
• Two Systems Under Development (SUD) reviews of the full IRMS program were conducted: during the planning phase; and during the analysis phase. The objective was to identify potential obstacles to the success of the program. The reviews prioritized risks and proposed appropriate mitigation strategies in four key areas: project management framework, implementation plan, data integrity and security, and change management, communication and organizational readiness.
• A comprehensive evaluation framework ensures on-going assessment of the IRMS program.

By helping to avert problems before they occur, the risk management approach has resulted in significant savings for the program.

Project management plan

The IRMS management framework was based on the Treasury Board Secretariat's Enhanced Management Framework. The project management framework included detailed project plans, regular progress reporting, well-defined objectives and a formal risk management strategy. A change management process ensured that proposed changes to the IRMS implementation plan had to be documented and reviewed.

The program's governance framework established a clear accountability structure, specifying roles and decision-making responsibilities for each authority level. One of the strengths of the IRMS management plan was its emphasis on "up-front" analysis and planning, and careful review and assessment at key stages during both the planning and implementation phases. Decision-making approaches included the following:

• Development of a business case defining the system requirements and analyzing the benefits to Members and managers, as well as the risks and costs of four options for addressing current and future service and system needs;
• Analysis of software options including a detailed assessment of costs and benefits associated with suitable commercial software products, evaluation and selection of a systems integrator, and development of an implementation plan and budget;
• Analysis of the fit between the new system and user service requirements to ensure that user interests were identified and represented;
• A roll-out approach to implementation, beginning in functional areas and proceeding to House managers and then to Members and their staff.

2.3 SIGNET Renewal Project

SIGNET 2000 + is the computing environment that allows the staff of the Department of Foreign Affairs and International Trade (DFAIT) and eleven other departments to exchange messages and access application tools at any of 158 Canadian government offices around the world. Delivered ahead of time and under budget, the success of its development and global deployment is attributable to excellent project planning and management, plus an effective industry/government partnership.

Objectives and scope

Late in 1998, testing confirmed that the existing 16 bit SIGNET platform was not Year 2000 compliant. This mission critical platform supported all non-classified messaging (about 30 million messages annually) and applications serving some 8,500 users worldwide. The legacy system was no longer supported by the vendor. Quick, effective, and economical design and implementation of a replacement was essential.

Project challenges included:

• The team had 18 months from identification of the problem to select a product, design, develop, test. procure components, and implement the system at 158 locations around the world before the fixed deadline of the Year 2000 transition.

• The nature of business conducted in offices overseas required that SIGNET be updated with minimum disruption to their operations which included presidency of the UN Security Council and the Kosovo crisis;

• The global scale of the system, representation of foreign nationals among both users and technical staff, and mobility of staff in general produced special challenges including: enhanced security; remote administration and user access; single sign-on anywhere in the world; compatibility with data from 1500 existing applications; training in several languages; and local alphabet support (including Russian, Chinese and Japanese).

Costs and benefits

The project had a budget of $46 million, including engineering, replacement of 345 servers, client and enterprise software, travel and training. The project was completed four weeks early and $1.9 million under budget.

The introduction of a standard workstation platform and remote systems maintenance is reducing system operational costs. This is allowing system administrators abroad to expand their user support role, as they have to spend less time on network and server maintenance.

SIGNET 2000 + has increased service availability from 91.8% to 99.5%, which has reduced lost client time by an estimated $2.7 million annually.

Total productivity increases of $94 million, mainly from new application functionality, are anticipated over the five-year life of the project. In addition, the productivity deterioration expected if the existing platform was retained for a further five years was even greater than this.

Critical success factors

The success of SIGNET 2000 + was largely attributable to close attention paid to the following factors:

• The imminence of the Year 2000 required careful scope planning, analysis and resolution to meet the stringent schedule and avoid "scope creep", yet deliver cost-beneficial user functionality;

• Detailed project planning, management and monitoring that included comprehensive and timely procurement, logistics and communications planning and execution;

• Adequate resources of funding, appropriately skilled technical and client staff and facilities available as identified in the project schedule, requiring considerable attention to resource acquisition and planning early in the project life; where contractors were used, technology transfer from contractors to in-house staff was built in as required;

• Effective quality assurance and control of deliverables, documentation and a formal change management process at all stages of the project to avoid delays or costly retrofits overseas;

• Close consultation with client groups and representatives; and unrelenting communications with the user community to prepare them for change;

• Effective and appropriately timed education for clients and technical training for field technical and support staff including "distance education" for reduced costs and greater flexibility.

Potential risks and risk management

The major risks facing the project were schedule risks: the risk of not completing the project before Year 2000. These risks were compounded by procurement risks in the form of three trade tribunal challenges and the risk of procurement delays which are common in government.

The premium on resources (contractors and hardware) associated with the Year 2000 "crisis" heightened the risk of cost overruns, as did the conflicting demands for resources to continue maintaining the unique but arthritic legacy environment that was no longer supported by the vendor.

There were situational risks such as the intrusion of local and international events into the implementation schedule (for example, UN Security Council activities, Team Canada visits, and the Kosovo crisis), and the risk of losing technical expertise at a time of intense demand for such resources.

There were also technical risks of not being able to solve compatibility and software integration problems, or being unable to migrate the myriad of existing applications in use in missions around the world.

To counter these risks, a risk management process was established and rigorously followed, including two independent risk assessments of the project. The component build schedule and QA reports on the build integration tests helped measure earned value, and contingency plans, including "off-ramps" were prepared.

Project management plan

The project used two main sources for its project management regime: Microsoft Solutions Framework, and the Treasury Board Secretariat Enhanced Management Framework (EMF).

From Microsoft the team adopted three main guidelines:

• The project should be a team of peers; project management should stress shared responsibility for the project, with individual accountability for components;
• There must be a strong commitment to good quality management practices;
• The project must employ good internal and external communications.

From the EMF the management team adopted the following:

• Top down and bottom up schedule and cost estimation, based on a detailed work breakdown structure;
• Project Charter and integrated project planning system including a basic mechanism for earned value tracking;
• Formal deliverable, schedule and cost tracking and reporting.

The adoption of these guidelines led to a project management team that had, as equals, managers for: product and communications; user education; implementation; development and infrastructure; the project office; and quality assurance. They reported to a Project Director, who in turn reported to a Board of IMT Directors chaired by the Department's CIO. The Project Champion was the ADM for corporate services.

Twice weekly "war room" exercises were scheduled, in which any team member could raise any issue they felt was critical, have it discussed and an action plan struck. Because senior management always attended the war room meetings, team members had the means to get quick action on these issues. This increased people's involvement in and commitment to the project.

2.4 Government of Canada Year 2000 Project

Objective

The Canadian government has been investing heavily in building computerized systems to support program and service delivery since the early sixties. Because most systems developed prior to the mid 1990's were designed to use two digits only to identify the year, there was a danger of some government systems failing in the new millennium if they were not corrected or replaced. In addition to fixing the systems problems, many government departments required new contingency plans, testing facilities, and training.

Dimensions

Estimated Year 2000 expenditures in the Canadian government, as of April 2000, were $1.9 billion. This consisted of:

• $1.5 billion for Year 2000 remediation and testing of IT systems;
• $300 million for contingency planning; and
• $100 million for other specifically funded initiatives. These included monitoring the private sector and other governments, setting up and running the central project office, establishing resource centres for embedded and building systems, contingency planning on a national scale, and communications with Canadians.

The scope of the Year 2000 project included:

• 43 Government-Wide Mission Critical (GWMC) functions, delivered by 23 departments;
• over 10,000 people dedicated to fixing the problem;
• 100 million lines of code, with more than 1,000 external IT interfaces;
• an estimated 100,000 embedded systems;
• 6,000 buildings across the country;
• contingency plans in 147 institutions.

Results

The government protected its service delivery capability through the transition period. Departments and agencies ended up with better information about their systems than they had ever had previously, and IT infrastructures were strengthened. Many organizations prepared comprehensive contingency plans for the first time. The partnerships that developed between the central agencies and departments were very effective.

Timeframe

Although some departments had started Year 2000 activities, up until the end of 1997 the Canadian government had taken only very limited steps in an integrated fashion, to tackle the threat to its IT investments. However during 1998 decisive action was taken to set up a management environment to deal with the issues in a coordinated manner. Almost all government-wide mission critical (GWMC) systems were remediated and tested by the end of July 1999, and all departmental mission critical systems well before December 1999, leading to a successful transition into Year 2000. Most departments closed down their Year 2000 projects by March 2000. Treasury Board Secretariat has maintained a small Post-Year 2000 Office, to provide information on the Year 2000 experience and to leverage lessons learned in current government IT projects.

Organization

The Prime Minister of Canada designated four departments to take the lead in preparing for Year 2000. These were National Defence, Foreign Affairs, Industry, and Treasury Board Secretariat. Senior committees at the ministerial and deputy ministerial levels were set up to direct the effort nationally.

The Treasury Board Secretariat established a Year 2000 Project Office to monitor and report on departmental progress, and to coordinate the resolution of horizontal issues affecting all departments. This included the early identification of Government-Wide Mission Critical (GWMC) functions (those essential to the health, safety, security and economic well-being of Canadians).

Departments were responsible for doing the work to maintain the integrity of their own programs, including the GWMC functions they delivered. They defined their budgets, made the critical business decisions, acquired the resources, managed their projects, and delivered the results. Departments established formal Year 2000 Project Management Offices with clear, department-wide roles, responsibilities and authorities.

Project management

The Year 2000 challenge provided an opportunity for government departments to reinforce, and in some cases to re-establish, a project planning function integrating IT planning into departmental program planning. Departments and agencies set up special Year 2000 project management offices (PMOs). Most PMOs had responsibility for all aspects of Year 2000 remediation and preparation, including contingency planning. In most cases separate budgets were established. In some departments the Year 2000 project office was part of the systems organization, but in a significant number of cases it was independent, reporting directly at a high level, and emphasizing that this was a business issue, not solely a systems matter.

Departmental Year 2000 project offices had the difficult task of cutting across organizational and regional boundaries to obtain the necessary action. Only the strong support of senior management, and the fixed Year 2000 deadline, allowed them to do this successfully.

As resources and time were both limited, departments adopted a risk management approach, paying attention to their mission-critical systems, and their most vulnerable technologies, first. For the 23 departments delivering GWMC functions, these were the top priority, followed by department-wide mission critical (DWMC) systems.

Departments that did not have the organizational capacity to perform the work took action to bring in the specialized skills. All departments made use of contract staff. They were able to utilize fast-track procurement mechanisms put in place by government purchasing authorities to expedite the work.

A focus on business continuity required Year 2000 project teams to extend their views beyond IT and address the broader issues of logistical support for program delivery, including supply chains, utility supplies, building and other embedded systems, and contingency planning. Some departments developed comprehensive contingency plans for the first time during the run-up to Year 2000.

Central Year 2000 project reporting was orchestrated by the Treasury Board Secretariat Year 2000 Project Office; this ensured timely monitoring and reporting of results. Departments were also encouraged to conduct independent reviews of their Year 2000 programs, and all the GWMC departments carried out at least one independent review or audit related to the Year 2000 project. Furthermore, most departments put in place an ongoing review program that continued until the Year 2000 project was complete.

To prepare for the transition, most departments instituted a freeze on all except essential system changes during the latter part of 1999. After the transition, all departments went through a formal project close-down phase, which often included an assessment of lessons learned.

Critical Success Factors

With respect to the specific success of the Year 2000 initiative, the factors that were seen as most critical in determining this outcome were:

• The declaration of Year 2000 as a government imperative established it as a clear priority for government departments;
• Establishing Deputy Minister accountability ensured management commitment at the most senior levels, and provided the context and requirements for the creation of a governance structure (e.g. committees and working groups) with roles and responsibilities that would deliver the results;
• The adoption of a risk focus reinforced the first two items and provided a mechanism for determining mission critical business functions, a context for decision making and issues management, and the framework for the development of a robust set of business continuity plans;
• The results of these three items enabled the initiative to proceed and provided for the funding, disciplined use of project management, the creation of partnerships, and many other support elements;
• Performance management, through very visible reporting mechanisms and monitoring at senior levels within departments and across government, was an essential tool for ensuring progress was achieved and in establishing credibility for government-wide initiatives;
• The control loop in this chain of events was provided by the public visibility that was accorded to Year 2000 through the extensive communications and formalised monitoring and reporting that was introduced;
• The influence and efforts of individual people over the life of the project significantly influenced the direction and outcome of the initiative; and
• The establishment of Project Management Offices provided effective oversight and management.

3. Lessons Learned

This section describes lessons learned from the cases featured in this report and is presented in two categories: government-wide lessons learned, and lessons learned at the level of the individual organization or project.

3.1 Government-wide Lessons Learned

Government-wide projects should:

• fit within a clearly expressed strategic plan for government investment in technology;
• be joint-venture partnerships between the central agencies and the departments and not the exclusive domain of one or the other;
• be driven by a clear risk and opportunity assessment supported by the program and service delivery community within government;
• be sustained by the highest executive and political management;
• be adequately funded; and
• be monitored and measured by precise, pre-defined and negotiated reporting requirements.

Business alignment

• Business should be engaged right from the start of large IT projects. The project needs to be aligned with, and support, the business directions and priorities.
• Planning for information technology investment must be fully integrated with the business/program objectives and planning processes of an organization.

Governance

• Establishing the initiative as a government priority is fundamental to success.
• Establishing Deputy Ministers' (DM) Accountability as a subsequent step ensures management commitment and decision-making at the most senior levels. It provides the context and requirements for creating governance structures (e.g. increased role of the TIMS sub-committee of DMs, other special committees and working groups) across the government, and the hierarchy set of roles and responsibilities that would deliver the results.
• Clear accountabilities and decision-making authority should be defined at all levels for all stakeholders. Appropriate frameworks should be established to ensure that if problems cannot be resolved promptly, they are elevated to the appropriate higher level of resolution.

Role of central agency (Treasury Board Secretariat)

• Large, complex multi-organizational government initiatives are most effectively managed through a collaborative effort that focuses responsibility for end results on departments; the central agency provides facilitation, co-ordination and monitoring.
• Strategic interventions ("swat" type actions) provide valuable opportunities to resolve issues.
• Even though accountabilities and responsibilities between central agencies and groups may be defined, they should be clearly communicated to departments.
• Policy direction is essential but it must be more flexible and be accompanied with appropriate supportive guidelines and mechanisms
• Efforts should continue to further entrench and institutionalize the EMF in all departments and agencies.

Risk management

• A continued focus on risk in planning and carrying out the initiative provides a critical context for identifying and managing concerns and potential issues.
• Risk management initiatives for government-wide use in major initiatives needs to be timely and directed to meet the needs of departments with varied levels of experience in risk management.
• Risk management methodologies and techniques should be adapted to the initiatives. The adaptation of the risk management methodology to the Year 2000 project, which was made available as part of the EMF suite of solution sets, was useful. However, the ease of implementation varied across departments and in some cases was dependent upon the availability and skilled resources.

Performance management

• A very visible reporting and central agency monitoring is an essential tool in ensuring that progress is achieved and maintained, and in establishing credibility for the whole endeavour.
• Reporting processes need to be simple and timely.

Communications

• Creation of a formal communication strategy and plan is essential.
• Effective communications are more complex to establish and more difficult to maintain than expected.
• All opportunities and possible vehicles for communicating must be utilised.

Procurement

• Government-wide Standing Offers continue to provide an appropriate and expedient vehicle for obtaining professional services
• Standing Offers need to be in place as early as possible in order to be effective and need to be more flexible in order to cope with changing situations
• Departments need to be fully cognisant of the contract selection process and contract content in order to maximize their utilisation of these vehicles.

3.2 Organization/Project-level Lessons Learned

They are presented roughly in chronological order as they apply during the life of an IT project.

Business alignment

• The alignment of the objectives of a project with business goals should be expressed in the form of a business case, which supports the investment decision. The business case puts the investment decision in a strategic context. It provides the information necessary to make a decision about whether a project should proceed. It provides an analysis of all the costs, benefits and risks associated with a proposed investment, and with the reasonable alternatives to the proposed investment.

Senior management commitment and involvement

• Involvement and commitment of senior program/business management to a large IT initiative at an early stage is essential;
• A "project sponsor" should be identified, who is typically a senior official responsible for the business function the project will support. An effective project sponsor ensures that the organization understands the value and importance of the project, and is ultimately responsible for realising the expected benefits of the project.
• Solid senior management support is demonstrated through the provision of adequate financial and human resources, and active participation in project governance.

Project governance

• Project governance should reflect the complexity of the challenge, and change as the project commences. Typically it will start with a senior management committee. From then on governance should shift structure and focus to meet the program's evolving needs, from consultation, inclusiveness and dialogue, to the practicalities of implementation.

Project Management Office (PMO)

• The creation and use of a Project Management Office (PMO) significantly improves chances of success in the project endeavour. The PMO's role is to facilitate co-ordinate and monitor on-going activities.
• A central PMO in an organization can manage portfolios of projects, provide effective project oversight, and act as a source of expertise and support to individual projects.

Project planning

• A Project Charter should be the first step in project planning, once the business case has been developed and the investment decision taken. The Project Charter is an agreement between the technical groups providing the product or service, and the business organization requesting and receiving the project deliverable.
• The Project Charter is a tool to obtain commitment from all affected groups and individuals within a specific project, and also a communication vehicle that can be referenced throughout the project. It acts as a quick reference and overview of what the project is about, why it is being conducted, who is involved and in what capacity, and the general approach and timeline that exists for the project.
• Project planning should include top down and bottom up schedule and cost estimation, based on a detailed work breakdown structure. Detailed project planning will provide clear definition and documentation of the project. It will encompass management structures, monitoring mechanisms, comprehensive and timely procurement plans, logistics and communications planning.
• Careful scope planning, analysis and resolution are necessary to meet project schedules and deliver cost-beneficial user functionality while avoiding "scope creep".

Project resources

• The responsibility for day-to-day management of a project can effectively be shared between in-house and contracted resources. Core project management responsibilities and functions should not be outsourced. An internal manager provides essential knowledge about the organization, its key players and management processes, program functions and requirements, while outside consultants offer technical and functional expertise in developing and implementing systems.
• To be successful a project requires adequate resources of funding, trained staff and facilities available as identified in the project schedule. This requires considerable attention to resource acquisition and planning early in the project life.
• The right mix of resources will involve program and information systems staff. The team should ideally include experienced and knowledgeable departmental staff from both the program and systems disciplines. If outside technical resources are needed, they should be used strategically to supplement staff skills. A requirement for technology transfer from contractors to in-house staff should be agreed from the start.
• Including systems management, design and architecture staff very early in the analysis of business processes and development of user requirements gives them a better understanding of business needs.

Risk management approach

• A risk-based process has been found very effective for determining departmental priorities and identifying mission critical functions.
• The chances of project success are measurably improved if risk management techniques are performed on a continuous basis throughout the project. Formal risk assessments at key points during the development life cycle can identify potential problem areas and suggest corrective action to be taken before the problems occur.
• Risk management for a project can start at the business case stage, with a review of the options for achieving the required business objective, including an assessment of the risks associated with each option.
• A risk assessment performed early in the development process can identify exposures against which contingency plans can be prepared, including "off-ramps" and alternative delivery strategies.

Phased approach

• The recommended approach to the implementation of long-term information technology strategies is through small, manageable components, each of which provides an improved capability (efficiency and/or effectiveness) to the organization. Such an approach provides certain key benefits:
• it is easier to define requirements more clearly for a smaller component;
• requirements are less likely to be affected by changes in business environment;
• more complete and accurate estimates of costs and schedules can be developed;
• it is easier to obtain project resources with appropriate levels of experience.
• Systems development should be done in well-scoped phases. The use of Function Point Analysis can eliminate guesswork in managing the scope of work that can reasonably be completed within the established time frame.
• Careful selection of achievable objectives for each phase, with diligent adherence to these objectives, keeps the development package stable; scope creep can be minimized.

Co-location

• Wherever possible, all project staff should be co-located to ensure rapid and accurate communication and information exchange. This includes program and information systems staff, in-house and consultant resources.

Involvement of users

• Project management and communications should involve functional and end users from as early as possible. Close consultation with client groups and representatives helps build ownership and commitment. Extensive user participation in systems development and testing is essential for a viable end product.

Performance measurement and reporting

• Achievement of project deliverables and schedules should be measured by a precise, pre-defined, and regular monitoring and reporting process. This should be established at the earliest possible stage in project start-up.
• Open and visible tabling of project status information at senior management committees on a regular basis focuses everyone's attention, and ensures continued management support.

Scope management

• A formal change management process, together with effective quality assurance and control of deliverables and documentation at all stages of a project can help avoid "scope creep".
• Involvement of business and user staff, leading to a thorough understanding of the links between project quality, schedule and cost, can also reduce pressures to expand the scope of a project. Careful selection of achievable objectives for each phase, agreed to by all parties, with diligent adherence to these objectives, enables a stable development environment to be maintained.

Communications and training

• Establish appropriate logs to document and communicate decisions, changes, issues and problems etc. to all project team members.
• IT initiatives must recognize their potential impact on people and their jobs. A comprehensive strategy for managing change should be part of project planning. This will include targeted communications, training and user support plans.
• All opportunities and possible vehicles for communicating, both internally and externally, must be utilised in prepare the user community and other stakeholders for change. Ideally the project team should include one or more communications specialists.
• Project planning should reflect the need for effective and appropriately timed education and training for users, and technical training for technical and support staff. All means of training should be investigated as appropriate, including computer-based or internet "distance education" where staff are geographically dispersed, for reduced costs and greater flexibility.

Independent reviews

The use of independent reviews at key stages of a project's life cycle can provide an extremely valuable snapshot of the "health" of the initiative. Experienced but independent observers with no stake in the project can identify issues that need to be addressed, but which might not be visible to project staff in the heat of the action.

4. References

• Enhanced Management Framework for IM/IT
• Creating and Using a Business Case for Information Technology Projects
• Treasury Board Project Approval Process
• IM/IT Investment Evaluation Guide
• Project Charter Guide
• Project Management Guide
• Project Planning, Tracking and Oversight

Auditor General Reports:

• http://www.oag-bvg.gc.ca/domino/reports.nsf/html/9932ce.html
• http://www.oag-bvg.gc.ca/domino/reports.nsf/html/mp9723e.html
• http://www.oag-bvg.gc.ca/domino/reports.nsf/html/9828ce.html
• http://www.oag-bvg.gc.ca/domino/reports.nsf/html/9624ce.html