Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - IM/IT Investment Evaluation Guide


Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.

IM/IT Investment Management Process Overview

Planning Phase: Choosing the Best IM/IT Investments
Tracking & Oversight: Manage the Investments by Monitoring for Results
Evaluation Phase: Learn From the Process
Questionnaire  

Planning Phase: Choosing the Best IM/IT Investments

How can one select the right mix of IM/IT projects that best meet mission needs and improvement priorities?

The goal of the planning phase is to assess and organize current and proposed IM/IT projects and then create a portfolio of IM/IT projects. In doing so, this phase helps ensure that the organization:

  1. Selects those IM/IT projects that will best support mission needs; and
  2. Identifies and analyzes a project's risks and returns before spending a significant amount of project funds.

A critical element of this phase is that a group of senior executives makes project selection and prioritization decisions based on a consistent set of decision criteria that compare costs, benefits, risks, and potential returns of the various IM/IT projects.

Steps of the Planning Phase

  1. Initially filter and screen IM/IT projects for explicit links to mission needs and program performance improvement targets using a standard set of decision criteria.
  2. Analyze the most accurate and up-to-date cost, benefit, risk, and return information in detail for each project.
  3. Create a ranked list of prioritized projects.
  4. Determine the most appropriate mix of IM/IT projects (new versus operational, strategic versus maintenance, etc.) to serve as the portfolio of IM/IT investments.

Management Tools and Techniques Applicable to This Phase

  1. An executive management team that makes funding decisions based on comparisons and trade-offs between competing project proposals, especially for those projects expected to have organization wide impact.
  2. A documented and defined set of decision criteria that examine expected return on investment (ROI), technical risks, improvement to program effectiveness, customer impact, and project size and scope.
  3. Predefined dollar thresholds and authority levels that recognize the need to channel project evaluations and decisions to appropriate management levels to accommodate unit-specific versus agency-level needs.
  4. Minimal acceptable ROI hurdle rates that apply to projects across the organization that must be met for projects to be considered for funding.
  5. Risk assessment that expose potential technical and managerial weaknesses.

Tracking and Oversight: Manage the Investments by Monitoring for Results

What controls are you using to ensure that the selected projects deliver the projected benefits at the right time and the right price?

Once the IM/IT projects have been selected, senior executives periodically assess the progress of the projects against their projected cost, scheduled milestones, and expected mission benefits. The type and frequency of the reviews associated with this monitoring activity are usually based on the analysis of risk, complexity, and cost that went into selecting the project and that are performed at critical project milestones. If a project is late, over cost, or not meeting performance expectations, senior executives decide whether it should be continued, modified, or cancelled.

Steps of the Tracking and Oversight Phase

  1. Use a set of performance measures to monitor the developmental progress for each IM/IT project to identify problems.
  2. Take action to correct discovered problems.

Management Tools and Techniques During This Phase

  • Established processes that involve senior managers in ongoing reviews and force decisive action steps to address problems early in the process.
  • Explicit cost, schedule, and performance measures to monitor expected versus actual project outcomes.
  • An information system to collect projects cost, schedule, and performance data, in order to create a record of progress for each project.
  • Incentives for exposing and solving project problems.

Evaluation Phase: Learn From the Process

Key Question: Based on your evaluation, did the system deliver what was expected?

The evaluation phase provides a mechanism for constantly improving the organization's IM/IT investment process. The goal of this phase is to measure, analyze, and record results, based on the data collected throughout each phase. Senior executives assess the degree to which each project met its planned cost and schedule goals and fulfilled its projected contribution to the organization's mission. The primary tool in this phase is the post-implementation review (PIR), which should be conducted once a project has been completed. PIRs help senior managers assess whether a project's proposed benefits were achieved and refine the IM/IT selection criteria.

Steps of the Evaluation Phase

  1. Compare actual projects costs, benefits, risks, and return information against earlier projections. Determine the causes of any differences between planned and actual results.
  2. For each system in operation, decide whether it should continue operating without adjustment, be further modified to improve performance, or cancelled.
  3. Modify the organization's investment process based on lessons learned.

Management Tools and Techniques During This Phase

  • Post-implementation reviews to determine actual costs, benefits, risks, and return.
  • Modification of decision criteria and investment management processes, based on lessons learned, to improve the process.
  • Maintenance of accountability by measuring actual project performance and creating incentives for even better project management in the future.

Processes

This is an assessment of the investment management processes that the organization is following to plan IM/IT investments, control and monitor progress of these investments, and evaluate final results. The central question to be answered is:

"Does the organization have defined, documented processes for planning, selecting, controlling, and evaluating its IM/IT investments?"

The goal in assessing an organization's processes is to identify to what extent the organization has a structure in place for managing and evaluating IM/IT investments.

  • There should be documented evidence (in guidance or policy) that an IM/IT investment management process is in place (consisting of planning, tracking and oversight, and evaluation pieces) that it is repeatable and implemented consistently throughout the organization, and that decision-making roles, responsibilities, and authority have been clearly defined.

An important point to remember when making an assessment of existing processes is that the evaluation should be focused solely on the organization's policies, practices, and procedures, not on actual decisions. Having institutionalized management processes, honed to work in the culture of the organization, is critical to producing consistently good results. The investment processes should accurately reflect the way the organization actually functions and makes decisions.

Data

An IM/IT investment process cannot operate without accurate, reliable, and up-to-date data on project costs, benefits, and risks. It is the basis for informed decision-making. In addition, documentation of management decisions is essential to begin to assemble a track record of results. Evaluating the data involved in the IM/IT investment management process requires evaluating two different types of data:

  • Ex ante – the information that is being used as inputs to the IM/IT investment process (e.g., the cost/benefit/risk analyses that are used to justify the planning and continued funding of projects, the performance measures that are used to monitor a project's progress, etc.).
  • Ex post – information that is produced based on decisions that are made (e.g., project review schedules and risk mitigation plans should be developed once a decision is made to fund a project).

All projects (proposed, under development, operational, etc.) should have complete and accurate project information – cost and benefit data, risk assessments, links to business/program goals and objectives, and performance measures, as well as up-to-date project-specific data, including current costs, implementation plans, staffing plans, and performance levels. In addition, the organization should have qualitative and quantitative project requirements and decision criteria in place to help screen IM/IT projects, assess and rank projects, and control and evaluate the projects as they move through the various phases of their life cycle.

All management actions and decisions that are made should be documented and maintained. Moreover, some decisions require that additional information be produced. For instance, after a project is selected, project-specific review schedules and risk mitigation plans should be developed.

Decisions

One of the most important goals of this guide is enabling evaluators to assess the effectiveness of the organization's IM/IT investment process and the extent to which it is contributing to the improved mission performance of the organization. After evaluating the processes that the organization uses to plan, monitor, and evaluate IM/IT investments and the data that are used to make decisions evaluators will be in a much better position to reach conclusions about the specific decisions that the organization is making. The central focus of analysis is on whether management decisions and actions are being taken using the investment control processes and requisite project data.

The IM/IT investment portfolio should represent a mixture of those projects that best meet the mission needs of the organization. Projects in the portfolio should be consistently monitored and decisions should be made at key milestones to ensure that the project is continuing to have its expected business or programmatic impact with a focus on minimizing risk and maximizing return. Completed projects are evaluated to compare actual performance levels to estimated levels and to feed lessons learned back into the Planning phases.