Treasury Board of Canada Secretariat
Symbol of the Government of Canada

 

ARCHIVED - IM/IT Investment Evaluation Guide

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.



An Enhanced Framework for the Management of Information Technology Projects

IM/IT Investment Evaluation Guide

May, 2000
Chief Information Officer Branch
Treasury Board of Canada Secretariat



Table of Contents

Overview

1. Planning Phase: Choosing the Best IM/IT Investments

2. Tracking and Oversight: Manage the Investments by Monitoring for Results

3. Evaluation Phase: Learn From the Process

QUESTIONNAIRE



IM/IT Investment Management Process Overview

Planning Phase: Choosing the Best IM/IT Investments
Tracking & Oversight: Manage the Investments by Monitoring for Results
Evaluation Phase: Learn From the Process
Questionnaire  

Planning Phase: Choosing the Best IM/IT Investments

How can one select the right mix of IM/IT projects that best meet mission needs and improvement priorities?

The goal of the planning phase is to assess and organize current and proposed IM/IT projects and then create a portfolio of IM/IT projects. In doing so, this phase helps ensure that the organization:

  1. Selects those IM/IT projects that will best support mission needs; and
  2. Identifies and analyzes a project's risks and returns before spending a significant amount of project funds.

A critical element of this phase is that a group of senior executives makes project selection and prioritization decisions based on a consistent set of decision criteria that compare costs, benefits, risks, and potential returns of the various IM/IT projects.

Steps of the Planning Phase

  1. Initially filter and screen IM/IT projects for explicit links to mission needs and program performance improvement targets using a standard set of decision criteria.
  2. Analyze the most accurate and up-to-date cost, benefit, risk, and return information in detail for each project.
  3. Create a ranked list of prioritized projects.
  4. Determine the most appropriate mix of IM/IT projects (new versus operational, strategic versus maintenance, etc.) to serve as the portfolio of IM/IT investments.

Management Tools and Techniques Applicable to This Phase

  1. An executive management team that makes funding decisions based on comparisons and trade-offs between competing project proposals, especially for those projects expected to have organization wide impact.
  2. A documented and defined set of decision criteria that examine expected return on investment (ROI), technical risks, improvement to program effectiveness, customer impact, and project size and scope.
  3. Predefined dollar thresholds and authority levels that recognize the need to channel project evaluations and decisions to appropriate management levels to accommodate unit-specific versus agency-level needs.
  4. Minimal acceptable ROI hurdle rates that apply to projects across the organization that must be met for projects to be considered for funding.
  5. Risk assessment that expose potential technical and managerial weaknesses.

Tracking and Oversight: Manage the Investments by Monitoring for Results

What controls are you using to ensure that the selected projects deliver the projected benefits at the right time and the right price?

Once the IM/IT projects have been selected, senior executives periodically assess the progress of the projects against their projected cost, scheduled milestones, and expected mission benefits. The type and frequency of the reviews associated with this monitoring activity are usually based on the analysis of risk, complexity, and cost that went into selecting the project and that are performed at critical project milestones. If a project is late, over cost, or not meeting performance expectations, senior executives decide whether it should be continued, modified, or cancelled.

Steps of the Tracking and Oversight Phase

  1. Use a set of performance measures to monitor the developmental progress for each IM/IT project to identify problems.
  2. Take action to correct discovered problems.

Management Tools and Techniques During This Phase

  • Established processes that involve senior managers in ongoing reviews and force decisive action steps to address problems early in the process.
  • Explicit cost, schedule, and performance measures to monitor expected versus actual project outcomes.
  • An information system to collect projects cost, schedule, and performance data, in order to create a record of progress for each project.
  • Incentives for exposing and solving project problems.

Evaluation Phase: Learn From the Process

Key Question: Based on your evaluation, did the system deliver what was expected?

The evaluation phase provides a mechanism for constantly improving the organization's IM/IT investment process. The goal of this phase is to measure, analyze, and record results, based on the data collected throughout each phase. Senior executives assess the degree to which each project met its planned cost and schedule goals and fulfilled its projected contribution to the organization's mission. The primary tool in this phase is the post-implementation review (PIR), which should be conducted once a project has been completed. PIRs help senior managers assess whether a project's proposed benefits were achieved and refine the IM/IT selection criteria.

Steps of the Evaluation Phase

  1. Compare actual projects costs, benefits, risks, and return information against earlier projections. Determine the causes of any differences between planned and actual results.
  2. For each system in operation, decide whether it should continue operating without adjustment, be further modified to improve performance, or cancelled.
  3. Modify the organization's investment process based on lessons learned.

Management Tools and Techniques During This Phase

  • Post-implementation reviews to determine actual costs, benefits, risks, and return.
  • Modification of decision criteria and investment management processes, based on lessons learned, to improve the process.
  • Maintenance of accountability by measuring actual project performance and creating incentives for even better project management in the future.

Processes

This is an assessment of the investment management processes that the organization is following to plan IM/IT investments, control and monitor progress of these investments, and evaluate final results. The central question to be answered is:

"Does the organization have defined, documented processes for planning, selecting, controlling, and evaluating its IM/IT investments?"

The goal in assessing an organization's processes is to identify to what extent the organization has a structure in place for managing and evaluating IM/IT investments.

  • There should be documented evidence (in guidance or policy) that an IM/IT investment management process is in place (consisting of planning, tracking and oversight, and evaluation pieces) that it is repeatable and implemented consistently throughout the organization, and that decision-making roles, responsibilities, and authority have been clearly defined.

An important point to remember when making an assessment of existing processes is that the evaluation should be focused solely on the organization's policies, practices, and procedures, not on actual decisions. Having institutionalized management processes, honed to work in the culture of the organization, is critical to producing consistently good results. The investment processes should accurately reflect the way the organization actually functions and makes decisions.

Data

An IM/IT investment process cannot operate without accurate, reliable, and up-to-date data on project costs, benefits, and risks. It is the basis for informed decision-making. In addition, documentation of management decisions is essential to begin to assemble a track record of results. Evaluating the data involved in the IM/IT investment management process requires evaluating two different types of data:

  • Ex ante – the information that is being used as inputs to the IM/IT investment process (e.g., the cost/benefit/risk analyses that are used to justify the planning and continued funding of projects, the performance measures that are used to monitor a project's progress, etc.).
  • Ex post – information that is produced based on decisions that are made (e.g., project review schedules and risk mitigation plans should be developed once a decision is made to fund a project).

All projects (proposed, under development, operational, etc.) should have complete and accurate project information – cost and benefit data, risk assessments, links to business/program goals and objectives, and performance measures, as well as up-to-date project-specific data, including current costs, implementation plans, staffing plans, and performance levels. In addition, the organization should have qualitative and quantitative project requirements and decision criteria in place to help screen IM/IT projects, assess and rank projects, and control and evaluate the projects as they move through the various phases of their life cycle.

All management actions and decisions that are made should be documented and maintained. Moreover, some decisions require that additional information be produced. For instance, after a project is selected, project-specific review schedules and risk mitigation plans should be developed.

Decisions

One of the most important goals of this guide is enabling evaluators to assess the effectiveness of the organization's IM/IT investment process and the extent to which it is contributing to the improved mission performance of the organization. After evaluating the processes that the organization uses to plan, monitor, and evaluate IM/IT investments and the data that are used to make decisions evaluators will be in a much better position to reach conclusions about the specific decisions that the organization is making. The central focus of analysis is on whether management decisions and actions are being taken using the investment control processes and requisite project data.

The IM/IT investment portfolio should represent a mixture of those projects that best meet the mission needs of the organization. Projects in the portfolio should be consistently monitored and decisions should be made at key milestones to ensure that the project is continuing to have its expected business or programmatic impact with a focus on minimizing risk and maximizing return. Completed projects are evaluated to compare actual performance levels to estimated levels and to feed lessons learned back into the Planning phases.



Planning Phase: Choosing the Best IM/IT Investments

IM/IT Investment Management Process

IM/IT Investment Management Process

1.1 Planning Processes

The IM/IT investment management process begins with the project planning process. Projects being proposed for funding are put through a "coarse-grained" screening process to:

  1. eliminate proposals that fail to pass minimal acceptance criteria; and
  2. ensure that projects are being reviewed at the most appropriate organizational level (Department, Branch, Sector, Region, Directorate, Unit etc.).

Proposals that pass this screening process have their costs, benefits, and risks analyzed in-depth. Once this is accomplished, all of the projects are compared against some common decision criteria and ranked based on their relative benefits, costs, and risks. Using this prioritized list as a guide, senior managers make decisions about which projects will be proposed for funding for the upcoming year. This post-prioritization decision-making on the appropriate mixture of projects is the essence of IM/IT portfolio analysis. Finally, after these funding decisions have been made, schedules for reviewing projects are established or updated.

1.1.1 Screening New Projects

The organization should have a process that outlines how to introduce projects for funding and how these projects will be screened for relevancy to business/program goals and objectives and technical soundness. Specifically, the organization should:

  • define what constitutes an IM/IT project;
  • identify initial requirements that projects must meet in order to be seriously considered for funding,
  • explain how screening will be conducted, and
  • establish roles and responsibilities for conducting the screening.

The screening process should be established in policy guidance (to ensure that it is conducted consistently) and used at all levels of the organization. As part of the initial screening process, there should be documented screening criteria (minimal requirements) that all projects are expected to meet.

The screening criteria should serve three functions. They should:

  • a) Identify whether the project meets initial acceptance requirements;
  • There should be some initial requirements that projects must meet before they are reviewed in greater detail. These initial requirements may include return-on-investment (ROI) thresholds (or minimum cost/benefit ratios), identification of the project's link to objectives in the business or strategic plans, evidence of compliance with the organization's information technology architecture, identification of business/programmatic sponsorship, and assurance that all necessary project proposal and justification steps have been performed.
  • b) Ensure that the project is being reviewed at the most appropriate organizational level
  • Not all projects need to be reviewed at the same organizational level. The criteria for screening projects should be structured so a determination can be made about where in the organization a decision would best be made. Certain cost and risk thresholds can be used to determine what requires centralized (Department or Branch level) versus decentralized approval (Sector, Region, Directorate, or Unit level).
  • c) Identify what level of management scrutiny is appropriate given the project's type, size, and risks.
  • There should be flexible but defined rules explaining how project review and approval may vary based on the project's relative costs, benefits, and risks. Low-cost, low-risk projects should not need to have the same justification provided for them as projects of greater cost, risk, and organizational impact.

On the basis of this screening process, projects will either move on for more in-depth analysis or will be sent back to the originating program group.

1.1.2 Analyzing and Ranking All Projects Based on Benefit, Cost, and Risk Criteria

The cost, risk, and benefit information of all projects (initial concept, proposed, under development, operational) should be analyzed and assessed in detail.

Each project should have a business case developed that provides the sponsor's justification for the project. The business case should identify the organizational needs that the project is meeting or proposes to meet; provide information on the benefits, costs, and risks of the project; and establish proposed project development time frames and delivery schedules. The information in the business case should be continuously updated to ensure that it always reflects the current situation.

The organization should have some established group or audit function that is responsible for verifying and validating the various analyses (cost/benefit analyses including feasibility studies, risk assessments, and alternatives analyses) and information that are submitted as part of a project's business case. This validation should include:

  • reviewing assumptions that were made;
  • assessing all of the alternatives that were analyzed and determining whether others should have been included;
  • reviewing the cost and benefit estimates to ensure that they were accurate and realistic;
  • evaluating the risks that were identified and determining whether others may be applicable;
  • evaluating the sensitivity analyses that were conducted.

The organization should have a management information system (MIS) or some other mechanism where all project information is collected and maintained. Such a mechanism, if kept accurate and up-to-date, can make data verification and validation easier by allowing the organization to track costs, risks, etc. over time.

This mechanism for collecting and maintaining project information will also be essential during the Control and Evaluation phases to:

  1. help assess whether projects are still aligned with mission needs and organizational objectives,
  2. determine whether projects are meeting planned performance goals, and
  3. identify possible revisions to the overall investment management process based on previous experiences and lessons learned.

After each project's cost, risk, and benefit information has been examined and validated, all of the projects should be compared against some common decision criteria in order to weigh the relative merits of the projects and develop a prioritized listing of projects.

The criteria used for assessing and ranking projects should consist of elements related to three essential areas - benefits, costs, and risks. Often organizations will establish broad categories related to these three areas and then develop more specific sub elements that come under each broad category. For example, an organization may establish risk as a categorical heading and then include schedule risks, cost sensitivity, technical risks, organizational risks, and risks if the project is not undertaken as sub elements under the risk heading.

Different organizations will break these broad categories and sub elements out in different ways. For instance, some organizations may include a project's costs as one of several factors under risk, while others break project costs out as a separate category.

Decisions should rarely be made based on one project factor, such as the project's estimated cost or a projection of reduced cycle time. Using an assortment of decision criteria to make decisions allows an organization to take into account and compare the different subtleties of a wide variety of projects.

The organization may assign weights to each of the broad categories, as well as any sub elements related to each category, in order to help prioritize those factors that the organization considers to be the most significant (e.g., a company that has limited experience developing systems may give technical risk a greater weight than projected cost). The mixture of weights among the ranking criteria will vary from organization to organization. The weights that are given should take into account the organization's unique mission, capabilities, and limitations. The weighting schema that the organization establishes should be defined and documented. Such documentation is even more important if different weighting approaches are used for different kinds of projects (operational, infrastructure, applications development projects, etc.).

To provide senior managers with an understanding of the relative costs, risks, and benefits of each project compared to the other projects, the organization may develop a scoring model or decision support tool. Such a tool compares the costs, benefits, and risks of each project against the cost, benefit, and risk criteria and assigns a score for each factor. The scores that the project receives for each factor are then added up to produce a cumulative score that establishes the project's relative worth and allows comparison against all other projects.

An important point for an organization in developing such a scoring model or decision support tool is to precisely define the scoring elements. The purpose behind these definitions is to ensure more consistent or uniform objectivity in the scoring process, which helps to eliminate widely varying interpretations and implementation.

The criteria for comparing and ranking projects should be used uniformly across the organization (i.e., unit, division, directorate, Region, Sector level decisions should be made using a set of criteria that are similar to criteria used for Branch or Department-level decisions). Although different levels of the organization may use additional criteria, the organization should have a set of minimum criteria that are used enterprise wide. Using some common decision criteria provides greater assurance that the organization is selecting projects consistently and helps to avoid "apples versus oranges" project comparison problems.

There should also be incentives to ensure compliance with the process and to dissuade gamesmanship. The organization should identify who is responsible for enforcing the process and there should be explicit consequences for noncompliance.

1.1.3 Selecting a Portfolio of Projects

The organization should have a senior management decision-making body, made up of program, and financial managers, that makes decisions about which projects to fund for the year based on its determination of where organizational needs are greatest. Such a determination will usually be made by analyzing the gap between the organization's goals and objectives (as highlighted in its strategic and annual performance plans) and the organization's existing capacity.

The roles and responsibilities of the IM/IT investment review group should be clearly identified and documented. The organization should also identify how this group will go about making decisions. This should include establishing how decisions will be reached, how conflicts will be handled, and how stakeholder input will be brought into the process.

The investment review group will make decisions about which projects to propose for funding, using the list of ranked projects as a key input. As the group goes about making these decisions, a number of trade offs will have to be made. For instance, the group will need to decide how much should be spent to continue operating and maintaining existing systems, versus funding enhancements to current systems, versus funding systems that are currently under development, versus funding new projects, versus funding research projects that assessing the applicability of emerging technologies. The group must also determine the proportions that will be spent on the various IM/IT types (i.e., research and development, administrative, mission critical, infrastructure, etc.). And, the group must take into account dependencies among projects.

The decision-making process should help address difficulties associated with using different units of measure for analyzing different kinds of IM/IT projects, as well as a balancing of "soft" versus "hard" quantitative data.

To aid the investment review group in making trade offs between various project types and phases, the organization may maintain a data repository that contains historical information on expenditures in different IM/IT investment categories (operations and maintenance, enhancements to current systems, new systems development, research into developing or applying emerging technologies, etc.). By maintaining this information, the organization can review how much was spent previously and factor this in to current spending decisions.

As part of the process of making trade offs and determining spending priorities, the organization may also conduct a review (in- house or via outside consultant/expert) of its current IM/IT spending portfolio to assess alignment with mission needs, priorities, strategic direction, major process re-engineering, etc. This review may include a trend analysis to show how patterns of investment and spending have changed, as well as an analysis to estimate how the spending pattern may change with the proposed IM/IT portfolio.

No matter how rigorous or structured the organization's decision- making process is, decisions about which projects to select for funding are ultimately managerial decisions. If senior managers select projects that score low when compared to other projects (e.g., high-risk, high-return projects) the justification for these decisions should be documented and the project's progress should be closely monitored during the Control phase. Making such exceptions should be kept as minimal as possible, however, to preserve the integrity of the decision-making process.

The process of reviewing and selecting IM/IT projects should be explicitly linked with other business processes (e.g., planning, budgeting, acquisition). Most investment decisions should mirror a planning decision or business objective and should be reflected in related budgeting documents and decisions.

The investment review group's responsibilities will usually not end once it has decided upon the mix of projects that will be proposed to comprise the current year's investment portfolio. Instead, the group should meet on a regular basis (often quarterly) to discuss the status of projects and to make further project decisions. The group may also be responsible for reviewing investment portfolio decisions that were made by lower-level organizational units.

1.1.4 Establishing Project Review Schedules

After making funding decisions, each project that was selected should have a review schedule established for it, or should have its current review schedule assessed and updated as needed. The time frames for these reviews will depend on various project-specific factors (amount of risk, investment size, mission importance, capability of the project team, etc.).

It is important that these reviews be conducted on a regular, scheduled basis. These reviews do not necessarily have to coincide with major project milestones. Moreover, "review triggers" should be established that automatically require a management review meeting. For example, a cost, schedule, or performance deviation of 10% or greater might require an immediate project review.

1.2 Planning Data

Good decisions require good data. Ensuring that each project meets established screening and ranking requirements and that the project's information is accurate and up-to-date is essential for ensuring that the most critical needs of the organization are being met by the projects and systems that are selected. In addition, the ex post information that is generated during this phase, such as project review schedules or risk mitigation plans, based on the planning decisions that are made, is critical for controlling and evaluating projects during the next two phases.

1.2.1 Evidence That Each Project Has Met Initial Project Submission Requirements

The efficiency of the investment management process depends initially upon how well the organization is ensuring that all projects meet initial project acceptance requirements and that necessary project proposal and justification steps have been performed. There should be evidence that each project that is submitted has been screened, analyzed, and evaluated according to processes and criteria established by the organization.

The information that is analyzed may include verification that all requisite planning data were submitted, that answers were received for all relevant questions, that projects met business/program goals and conformed to the agency's information technology architecture, and that projects that did not meet these requirements were not allowed to move on for further review and consideration. There should also be evidence demonstrating that all business units adhered to organizational policies and procedures regarding the screening and acceptance of projects.

Much of the evidence that will be reviewed will consist of cursory completeness and quality checks. For instance, if the organization has requirements that all projects over a certain cost threshold must

  1. submit complete cost/benefit and risk analyses,
  2. identify the business objectives that the project is meeting, and
  3. provide assurance that the project conforms to the organization's technical architecture, then a review of projects that went on for further review should not identify any projects that did not meet these initial requirements. Evidence should also be available demonstrating that each project adhered to the documented process.

There should also be evidence that information that was submitted was validated by a quality assurance/control function. Such validation can be performed by in-house quality control/quality assurance staff, internal audit staff (e.g., inspector general), etc. The project information should also be verified to ensure that it is accurate and reflects the most up-to-date information.

All project information should be up-to-date, cost numbers should be accurate, benefits should be quantified to the extent possible, risks should be spelled out, alternatives should be identified, and sensitivity analyses should have been conducted.

1.2.2 Analysis of Each Project's Costs, Benefits, and Risks

Each project that is submitted should have a business case prepared that provides justification for the project. Included in the business case should be identification of the project's functional requirements and estimates of the project's life-cycle costs, benefits, and risks (to the extent possible), as well as the corresponding analyses that were conducted to develop the estimates. Making accurate cost savings estimates and benefit determinations requires having at least a rudimentary understanding of the baseline costs and benefits from existing IM/IT capabilities.

A key analysis that should almost always be submitted with project proposals is a cost/benefit analysis. A complete cost/benefit analysis should

  • identify and quantify benefits and costs,
  • identify assumptions and constraints that were used when developing these figures,
  • evaluate alternatives using net present value, and
  • perform risk and sensitivity analyses.

The amount of rigor and types of analyses that are conducted will depend, in part, on the size of the investment and the amount of risk. It may not be economical to conduct an in-depth cost-benefit analysis for a low-cost, low-risk project that only affects a specific division or office or a limited number of users. The organization should have a process that outlines what project data are required given each project's type, cost, and risks; variation in the quality or type of data should not be ad hoc.

Listed below are some of the cost, risk, and benefit elements that an organization should keep in mind as IM/IT develops project estimates.

Costs (recurring and non-recurring)

  • Up-front costs, such as hardware and software purchases, costs to design and develop the project, transition costs, etc...
  • Ongoing costs, such as salaries, software upgrades, training, supplies, operations and maintenance, etc...
  • Indirect costs, such as initial productivity losses, computer support (network management, data administration, hotlines), etc...

Risks

  • Project risks, such as the size of the investment, project size and longevity (is the project designed in modules, does the project rely on the implementation of other systems, do other systems rely on this project), has the project group managed other projects of similar risk and complexity.
  • Organizational risks, such as mission risks if the project does not proceed, program management's commitment to the project, political expectations for the project, whether the project is legislatively mandated, etc.
  • Technical risks, such as skills required, hardware and software dependencies, application software, etc.

Benefits (will usually consist of both tangible and intangible benefits)

  • Tangible benefits include benefits that can be explicitly quantified. Such benefits may include reducing costs, increasing productivity (e.g., reducing errors, eliminating duplication or needless work steps, etc.), decreasing cycle time, or improving service quality (e.g., timeliness, convenience, access, reliability, etc.).
  • Intangible benefits include benefits that may be easy to identify, but that are difficult to quantify. These benefits may include faster, more efficient decision-making; greater data accuracy; improved data security; reduced customer burden; or increased organizational knowledge.

In identifying and measuring IM/IT benefits, it is important to always remember the business function or process that is being supported by the technology. For instance, the benefits that are gained from implementing EDI technology are derived from the increased capability and efficiency that the technology provides to the organization and its customers.

All of the information in the business case should be as up-to-date and accurate as possible. If the analyses are to yield meaningful results, it is essential that the project team carefully formulate assumptions, identify feasible alternatives, and provide realistic cost and benefit estimates.

Most agencies have criteria or methodologies detailing how cost/benefit analyses are to be conducted and what should be included. In addition, OMB Circular A-94 provides guidance and discount rates for conducting cost/benefit analyses.

1.2.3 Data on the Existing Portfolio

Because all projects (ongoing, under development, etc.) go through the Planning process (usually on an annual basis), portfolio data from previous years should be available to assess and compare previously selected projects. The spending, cost, and obligation data in this portfolio should be up-to-date and categorized in ways that are most meaningful to organization management.

An agency's cost accounting system should be able to distinguish between what has been obligated to date and what is still available, as well as identify what the incurred costs to date were for. In addition, the system may be able to split spending into more specific categories, such as development, operations, maintenance, etc. (Activity-based cost tracking, for example, should provide this detail.)

1.2.4 Scoring and Prioritization Outcomes

There are several pieces of information that should arise out of the Planning phase, based on the actual decisions that are made. This information includes:

  • the initial project scores and ranked list of projects,
  • the investment review group's scores based on any additional decision-support tools,
  • the investment group's final list of projects that will make up the investment portfolio,
  • documented justification for selecting projects that scored below accepted thresholds (e.g., high-risk, high-return projects), and
  • funding information, as well as acquisition and development schedules, for all projects that were selected.

The organization should also be maintaining net cost and benefit information on the complete portfolio of IM/IT investments.

Finally, all of the projects that were selected for funding should be included in the Organization Capital Plan that is submitted to OMB. Information that is submitted in this plan should include baseline cost, schedule, and performance goals for each project.

In addition to the Capital Plan, decisions that are made on the mix of existing and new projects should be clearly identified in the agency's annual performance plans. Actions described in the Capital Plan to implement the funding, procurement, and management of the IM/IT projects should also be articulated in these performance plans.

1.2.5 Project Review Schedules

All projects that are selected for funding should have project review schedules, risk management plans, and project-specific performance measures established. All of this information will be particularly critical for assessing performance, identifying risks, and making decisions during the Control and Evaluation phases.

The timing of reviews, as well as the number of reviews that will be conducted, will depend on the investment size of the project, the amount of risk, the capability of the project team, etc.

In addition, the investment review group may identify additional project management or investment review reporting requirements (data, information, analysis), beyond what is specified by existing processes, for projects that it determines are particularly high-risk. These additional requirements should be clearly documented and communicated to the responsible project team. The project team should also be given explanation detailing how this information ... and its assessment by senior management ... may influence project continuation, delay, or cancellation.

At some point the project team should develop an outline or strategy describing how any necessary acquisitions will be handled. Key tenants of a sound acquisition strategy are that it appropriately allocates risk between the organization and contractor, effectively uses competition, ties contract payment to accomplishments, and takes maximum advantage of commercial technology.

1.3 Planning Decisions

The purpose of the Planning phase is to put the organization in the best possible position to make decisions about which IM/IT proposals or projects to fund. Getting to this final decision requires that initial decisions be made about whether proposed projects should be moved on for further consideration. It then requires decisions to be made about the relative merits of each individual project. This is followed by the most important decisions, in which tradeoffs are made between the various projects and systems in order to develop the IM/IT investment portfolio that will be funded for the upcoming year.

1.3.1 Determining Whether Projects Met Process Stipulated Requirements

All new projects should have a decision made about whether the project meets all minimal project requirements, at what organizational level the project should be reviewed, and the level of analytical rigor necessary for decisions. While these screening decisions should be relatively straightforward, driven primarily by project-level data sufficiency, they should not be thought of as simply a cursory exercise. The overall efficiency and effectiveness of the entire Planning phase depends to a large extent upon these initial screening decisions.

The organization should also have a process for determining where in the organization a funding decision should be made. The efficiency of the investment management process is significantly affected by how well the organization identifies which projects should be reviewed where. Senior decision makers should not spend their time in lengthy, in-depth reviews of projects that could have been easily assessed and decided upon at lower organizational levels.

1.3.2 Deciding Upon the Mixture of Projects in the Overall IM/IT Investment Portfolio

Decisions made at this stage are the most important of all. The projects that are proposed to make up the investment portfolio for the year should represent the best match with organizational needs and business objectives or, in instances where exceptions were made, an explanation should be provided detailing reasons for the exception.

In making the planning decisions, senior managers should be taking into account tradeoffs between the various projects and systems that are going to be funded. Making these tough choices requires the organization to develop and maintain an understanding that not every project or system can be funded. Spending more for operational systems may mean that there is less money for research and development. The relative merits of each project should be rigorously assessed and analyzed in order to prioritize and select those projects that best match the most critical business needs of the organization.

In addition, projects selected for the portfolio should have decisions made about how often they will be reviewed and how associated risks are going to be managed.



Tracking and Oversight: Manage the Investments by Monitoring for Results

2.1 Tracking and Oversight Processes

Achieving maximum benefits from a project, while minimizing risks, requires that the project be consistently monitored and managed for successful results. During the Tracking and Oversight phase, agency executives should be actively engaged in monitoring all of the projects in the investment portfolio, making decisions and taking actions to change the course of a project when necessary and incorporating their experiences back in to the Selection phase to further refine and improve the process.

2.1.1 Consistently Monitoring Projects

Each project should be reviewed at key milestones in its life cycle (a project review schedule should have been approved when the initial funding decision was made). The focus of these reviews should expand as projects move from initial design and pilot through full implementation and as the dollar amounts that are expended increase.

A low-cost, small-scale research and development project being conducted to determine the applicability of a systems technology to a business process requirement might receive limited review other than assessing whether the general approach is sound and feasible. However, projects that are preparing for limited field or full-scale implementation should be reviewed in depth including cost and performance to date - to ensure that the project delivers promised benefits within cost and risk limitations and to correct any problems before significant dollars are expended.

In addition, as the reviews are conducted, the context of the program that the system or project supports should be factored in. For instance, a project may exceed performance expectations, but if it is contributing to a program that is failing or is no longer needed, then little is gained for the organization.

The project reviews should assess several aspects of the project and its development. Below are examples of assessment categories that should be considered as part of the project reviews:

  1. Deliverables - results achieved to date versus expected results.
  2. Methodology - problems that have arisen concerning the systems development process (including contractor management issues).
  3. Technical - technical issues or problems, concerning such factors as hardware, software development, or telecommunications.
  4. Schedule - estimated time frames versus actual, including schedule slippages and/or compressions.
  5. Costs - estimated costs versus costs spent or obligated to date, any changes in funding, and the impact of these changes.
  6. Business/program alignment - evaluation of the project's mission improvement effectiveness and relationship to business objectives.
  7. Risk - risks that were previously identified are being appropriately mitigated, new risks have been evaluated and steps taken to address them.

The organization should have some standard policies that help ensure that these different categories are assessed uniformly across the organization; however, the measures that are used to evaluate each project will be specific to that project. For instance, the organization may have a requirement that all projects have their schedules reviewed, but the schedules that are reviewed will be different for each project.

The problem with many progress reviews is that they focus almost exclusively on cost and schedule concerns. While these factors are important, the prime focus of progress reviews should be on ensuring that benefits are being accomplished, that risks are being managed, and that the project is still meeting strategic needs. As noted earlier, "review triggers," such as updated benefit/cost ratios or ROI thresholds, done in conjunction with schedule and spending checks, can help the organization determine when actions need to be taken.

The organization should have a documented process detailing how reviews will be conducted, what data and project information is required, and how decisions will be made based on the results of the project reviews. This process should include identifying roles and responsibilities for making decisions, as well as rules for how the project decisions will be made.

Some organizations use a traffic-light method to help make project decisions. Projects are given red, yellow, or green lights depending on how the project rated against expected performance measures. Yellow lights indicate that management action is necessary to avoid potential problems affecting project outcomes. Red lights indicate that major problems have already occurred. (As with all reporting and scoring mechanisms, it is critical that the organization define the conditions associated with element.) The following is an example of a traffic light tracking and oversight process:

The organization should also have an independent audit team, quality assurance group, or independent validation and verification (IV&V) contractor who is responsible for ensuring that project information is valid and verifying that corrective actions have been taken. In addition, the organization should have procedures in place to ensure that information from this quality assurance function is integrated in to the project review process.

Finally, the organization should have mechanisms in place to ensure that project teams are complying with the tracking and oversight process. This may include incentives for raising problems to senior managers and disincentives for noncompliance.

Project reviews, while helping to ensure accountability, should not be totally viewed as a "gotcha" opportunity, in which project managers are punished when problems are identified. Rather, the reviews should be considered opportunities for raising problems early, when they may be easier to address, rather than allowing the problems to be buried, creating a risk that they will arise later when costs are higher and the potential impact is greater.

2.1.2 Involving the Right People

Senior managers (particularly program managers) should be actively involved in the ongoing project reviews and are responsible for making decisions about whether to continue, accelerate, modify, or cancel a project. While members of the development team can, and should, be part of the decision-making process, they should not have unilateral responsibility or authority to make all project decisions. In addition, site executives and project managers should take part in devising and approving the solution to any problems that are identified.

2.1.3 Documenting All Actions and Decisions

All of the information in the business case, including the various analyses that were conducted to justify the project, should be updated to reflect the current state as project implementation continues and dollar amounts increase.

Some leading organizations estimate that often they cannot accurately estimate costs or quantify benefits until almost 40 percent of the way into the project.

The organization should have a uniform mechanism (e.g., management information system) for collecting, automating, and processing data on expected versus actual outcomes. Specifically, this mechanism should: provide the cost and performance data needed to monitor and evaluate investments individually and strategically, provide feedback on the project's adherence to strategic initiatives and plans, and allow for the review of unexpected costs or benefits that resulted from investment decisions.

Data in this system should be easily accessible to both the program team and senior managers.

Collecting and maintaining project information is important, not only from a project review standpoint, but also from the standpoint of establishing an organizational memory. Decisions in all three phases of the investment cycle (Planning, Tracking and Oversight, and Evaluate) will depend on this information being accessible and up-to-date.

2.1.4 Feeding Lessons Learned Back in to the Planning Phase

Information learned during the Tracking and Oversight phase should be fed back in to the Planning phase to help make future selection decisions and to modify and enhance the screening and selection decision criteria. To make this easier, there should be some mechanism in place for aggregating decisions and actions in order to identify patterns of problems or, conversely, patterns of excellence.

Document the warning signs that, with hindsight, preceded the problem, identify what remedial steps were taken, and what the outcome of this approach was. Such documentation will help to make future acquisition decisions and identify recurring problems on existing programs.

2.2 Tracking & Oversight Data

Because the Selection phase usually occurs only once a year during the annual budget process, project information for that phase tends to be collected and assessed on a periodic basis. In contrast, information in the Tracking and Oversight phase is continuously collected, updated, and fed to agency decision makers. The data in the Tracking and Oversight phase should consist of such items as comparisons of actual results achieved to date versus estimates and an assessment of benefits achieved as part of project pilots or prototypes. Data collected during this phase will also consist of ex post documentation such as executive decisions and changes made to projects to address risks or better meet business requirements. The type and depth of data that are collected and maintained in this phase should be commensurate with the type and size of the project.

2.2.1 Measures of Interim Results

As projects move from one phase of the project's life cycle to the next, and as the dollars that are expended increase, interim results should be compared against estimates to ensure that the project is progressing as expected and to indicate when actions should be taken as problems arise or requirements change.

The organization should have project-specific measures established to help analyze actuals versus estimates, ensure that the project is meeting business requirements, and identify where improvements may be needed. These measures will consist of items such as cost and schedule information, quantitative and qualitative benefit measures, status of deliverables, risk elements, etc. The measures should be updated as actual costs, risks, and benefits are identified.

Using these measures, the organization should identify and monitor interim results that are achieved. The following are examples of the kinds of data that should be analyzed:

Accumulation of actual cost data and comparisons to estimated cost levels.

Evidence that results for the phase (or results to date) have been compared against initial estimates for cost, schedule, performance, risk, and return. Documentation of the change between the current number and scope of requirements and the original requirements baseline established for the project. Documentation of the comparison between the current business conditions and assumptions and the projects' initial assumptions and context. After the release of each new increment, each project participating in the increment should be analyzed to determine what interim benefits have been achieved in comparison to the previous increment.

Documentation of differences between the actual performance of the software organization or contractor and their claims at the beginning of the project (e.g., schedule, costs, functionality, technical solutions, etc.).

Aggregate data covering costs, benefits, and project performance for all IM/IT projects in the investment portfolio.

2.2.2 Updated Analyses of Each Project's Costs, Benefits, Schedule, and Risks

The cost, benefit, schedule, and risk information that was included in the business case, including the various analyses that were conducted to justify the project, should be updated as project implementation continues and as dollar amounts increase. For instance, it may have been difficult to precisely estimate costs and benefits when the project was first proposed, but such quantification may be improved as prototype and pilot project results become available.

Information and analyses in the business case should also be updated to provide justification for adding additional functional requirements to the project. This justification should weigh the costs of adding the requirement late in the development process versus the anticipated benefits that are expected from the added functionality.

Older versions of these analyses should be maintained for later comparisons and to feed lessons learned back in to the Selection phase.

2.3 Tracking and Oversight Decisions

The primary focus of the Tracking and Oversight phase should be on making project management decisions. Actions should be taken quickly to address problems as they are identified and senior managers should be actively involved in making decisions about all of the projects in the investment portfolio. While many of these decisions will be implicit (the project is right on course, no problems have been identified, requirements have remained the same and, thus, the decision will usually be to continue the project as is), it is critical, nonetheless, that a conscious decision be made about the future of each project.

2.3.1 Deciding Whether to Cancel, Modify, Continue, or Accelerate a Project

As each project is reviewed at various stages in its life-cycle, decisions should be made about the future of the project. These decisions will be unique for each particular project and should be based on the particular merits of the project. In addition, some explanation or documentation of the decision should be included. Even implicit decisions should have some documentation to show that a conscious decision was made to continue the project.

Projects that have deficiencies or problems identified (actuals exceed estimated levels, risks are increasing, requirements have changed, etc.) should have a decision made by senior managers about what to do with the project. Decisions will usually involve one of four alternatives: modify the project cancel the project continue the project as is accelerate the project's development

Decisions may also be made to suspend funding or make future funding releases conditional on corrective actions being taken. These decisions should be documented, along with an explanation or criteria stating how funding can be reobtained. The decisions should also be reflected in budget information. For instance, if a project's development is halted while the feasibility of an alternative is assessed, budgetary spending information should reflect such a halt in funding. There should also be an explanation or documented criteria stating what must occur before funding is reinstated.

In addition, depending on what decisions are made about a project, future "cascading" actions resulting from these decisions should be clearly identified and delineated. For instance, halting the development of a project will impact a number of other areas, including project management, personnel and staffing decisions, budget decisions and spending priorities, etc. These cascading actions should also be reviewed and monitored to ensure that money is not continuing to be spent and that all development activities have ceased.

An independent review should be conducted prior to funding being reinstated to ensure that all corrective actions have been taken and to determine whether additional changes or modifications are still needed.

2.3.2 Aggregating Data and Reviewing Collective Actions Taken to Date

A review of past activities and decisions made about a particular project can influence both current and future managerial decisions. This is a primary reason why aggregating information is important. Aggregating allows trends to be more easily identified. Looking at projects across an agency or bureau, for example, can help pinpoint those divisions that have had repeated success at developing and managing IM/IT projects, and those that have had more trouble. This in turn can be used as inputs for decision makers when weighing organizational capability risks and determining project review schedules.

Aggregating can also help as the organization looks to refine and improve the screening and selection criteria and performance measures. Data can be aggregated by project, or can be grouped along unit, divisional, bureau, or agency lines.

Problems that are identified from this analysis may be serve as an indication of specific endemic weaknesses with project management, contractor oversight, or cost-estimation practices that need revision and corrective actions. In addition, positive trends that are identified can provide valuable lessons for highlighting and reinforcing organizational strengths.



Evaluation Phase: Learn From the Process

3.1 Evaluation Processes

The Evaluation phase "closes the loop" on the IM/IT investment management process by comparing actuals against estimates in order to assess performance and identify areas where future decision making can be improved. Lessons that are learned during the Evaluation phase should be geared towards modifying future Selection and Control decisions. Central to this process is the post-implementation review with its evaluation of the historical record of the project.

3.1.1 Conducting Post-implementation Reviews (PIRs) Using a Standard Methodology

Once a project has reached a final end point (e.g., the project is fully implemented, the project has been cancelled, etc.), a post- implementation review (or post-investment review) should be conducted. This review will usually occur about 3 to 12 months after a project has reached its final end point and should be conducted by a group other than the project development team to ensure that it is conducted independently and objectively.

Organizations often spend significant time and resources focused on selecting IM/IT projects, but less attention is given to evaluating projects after they are implemented. Yet the information gained from PIRs is critical for improving how the organization selects, manages, and uses its IM/IT resources.

Each PIR that is conducted should have a dual focus - it should:

  1. provide an assessment of the implemented project, including an evaluation of the development process, and
  2. indicate the extent to which the organization's investment decision-making processes are sustaining or improving the success rate of IM/IT projects. The following are three essential areas that should be evaluated as part of a complete PIR:
  • a) Customers : Surveys should be conducted to determine users' satisfaction with the end product. There should also be a focused look at how well the project supports the organization's various business processes. Many of the intangible benefits that were identified at the outset will relate to how customers and end users feel about the final project.
  • b) Mission/Program Impact : A close look should be taken to determine whether the implemented system has achieved its intended impact, and whether this impact is still aligned with mission goals. An assessment should also be made of other project-specific aspects, such as an estimate of cost savings that have been achieved, compliance with the information technology architecture, evaluations of the information product (accuracy, timeliness, adequacy, and appropriateness of information), and identification of additional maintenance or security issues.
  • c) Technical Capability: Finally, an evaluation should be made of the technical aspects of the project, both current and future. Such an evaluation may focus on such factors as the competency of the workforce to use the new system and employee satisfaction or retention, the extent to which advanced technology was used, and the methodological expertise of the development team.

To ensure that each project is evaluated consistently, the organization should have a documented methodology for conducting PIRs. This methodology, which should be used at all organizational levels, should spell out roles and responsibilities for conducting reviews and for taking actions based on the results. PIRs should be required on a regular basis to ensure that completed projects are reviewed in a timely manner. The organization should also have policies or procedures that document how information from the PIRs is to be relayed back to decision makers.

Finally, because there is a great deal of knowledge that can be gained from failed projects, evaluations should also be conducted for projects that were cancelled prior to being fully implemented. Although project accountability is important, these evaluations should focus on identifying what went wrong with the project, in order to learn from mistakes and minimize the chances of their being repeated.

3.1.2 Feeding Lessons Learned Back Into the Planning and Tracking & Oversight Phases

All of the PIR information gained in the Evaluation phase should be collected and maintained with project information gathered during the other two phases. Developing this complete library of project information helps to establish an organizational memory in which both successes and failures can be used for learning.

There should be some mechanism or process to ensure that information is being aggregated and fed back in to improve the investment management process. For instance, the cost, risk, and benefit criteria (including the weights given to each) for the Selection phase may be refined to ensure greater implementation success of future projects. The organization should also determine whether there may be different or more appropriate cost, benefit, and risk measures that could be established that would help better monitor projects.

3.2 Evaluation Data

Data collected during the Evaluation phase will be primarily historical in nature focusing on the outcome of a project as compared to executives' expectations for the project. In addition, ex post information that is developed should include modifications made to the Selection and Control phases, as well as the institutionalized lessons-learned information. This information should be used to revise the Selection and Control phases and to help make future investment decisions.

3.2.1 Measurements of Actual Versus Projected Performance

The focus of the PIR should be on evaluating a project's actual results compared to estimates in terms of cost, schedule, performance, and mission improvement outcomes. An attempt should also be made to determine the causes of major differences between planned and end results. And the PIR should be used to help identify any inappropriate systems development and project management practices.

The PIR should provide a wide range of information regarding both the project and the process for developing and implementing the project. Specific information includes the following:

  • an assessment of the project's effectiveness in meeting the original objectives,
  • an identification of benefits that have been achieved,
  • an assessment of whether they match projected benefits, and a determination of reasons for any discrepancies.
  • an evaluation of whether original business assumptions used to justify the project were valid,
  • a comparison of actual costs incurred against projected costs, a determination of how well the project met time schedules and implementation dates, management and user perspectives on the project, and
  • an evaluation of issues that still require attention.

Outputs of the PIR should include user evaluations of the effectiveness of the project, actual costs broken out by category, measurements used to calculate benefits, a comparison matrix of actuals to estimates, and business-as-achieved documentation.

3.2.2 Documented "Track Record" (Project and Process)

The organization should be maintaining documentation of all decisions, changes, actions, and results that occurred throughout the project's life cycle, as well as other relevant project information, such as the business case and updated cost/benefit analyses. The organization should also be tracking recommendations (for both improving the project and refining the overall investment management process) that arise out of the PIRs.

This "track record" will be invaluable for helping the organization refine and improve its processes as more and more information is collected and aggregated.

3.3 Evaluation Decisions

A number of key decisions will be made during the Evaluation phase, including an assessment of how well the project met its intended objectives, a determination of what changes or modifications to the project are still needed, and an identification of ways to modify or improve the overall investment management process to better maximize results and minimize risks. In addition, the organization may assess the overall performance of its IM/IT investments in improving mission performance. To make these decisions, agency executives must gauge the degree to which past decisions have influenced the outcome of IM/IT projects, understand why these decisions had the effect that they did, and determine how changing the processes for making decisions could create a better outcome for current IM/IT projects and future IM/IT proposals.

3.3.1 Assess a Project's Impact on Mission Performance and Determine Future Prospects for the Project

The results and recommendations that arise out of the PIRs, combined with other project information, are a critical input for senior decision makers to use to assess the project's impact on mission performance. In making this assessment, senior managers will need to ask a number of questions about the project, including the following:

  • How effective was the project in meeting the original objectives?
  • Are these objectives still valid?
  • Were the original business assumptions used to justify the project valid?
  • What is the current status of the system?
  • Are further changes or modifications necessary?

Even after a project has been implemented, decisions should be made on a regular basis about the status of the project. Senior managers should regularly question whether

  1. the current system meets organizational needs,
  2. the system should be modified to better meet these needs,
  3. a new system is needed to best meet these needs, or
  4. the needs could best be met by outsourcing the work.

In addition, because operation and maintenance (O&M) costs, for such activities as hardware upgrades, system software changes, and ongoing user training, can consume a significant amount of IM/IT resources (some have estimated that ownership costs, operations and maintenance costs, and disposition costs can consume as much as 80 percent of a project's total life-cycle costs), a plan should be developed for the continued support and operation of each IM/IT project.

3.3.2 Revising the Planning and Tracking and Oversight Phases Based on Lessons Learned

An organization's investment management process will usually not remain static, but will evolve and change over time as the organization learns more and more about what has been successful and what still needs to be improved. Modifications that may be made to the process include the following:

  • changing the mixture of members on the organization's decision-making investment review group
  • changing the Planning phase decision-making criteria (both screening and ranking criteria)
  • changing the Tracking and Oversight phase criteria used for monitoring the progress of projects
  • modifying the time frames for reviewing projects during the Tracking and Oversight phase
  • modifying the triggers for identifying projects for review
  • modifying the PIR methodology.

The results from one project will often not provide enough information to allow significant modification to be made to the agency's IM/IT decision-making processes. However, a significant , recurring system development problem found across multiple projects over time would because for refining or even significantly revising the organization's decision-making processes and criteria.

The causes for differences between planned and actual results should be determined and corrective actions to the overall IM/IT management process, decision criteria, etc. should be identified and documented. Once the causes for differences between planned and actual results have been determined, steps should be taken to address these causes in order to ensure greater success in the future.

All alterations or updates that are made to the Planning and Tracking and Oversight phases, based on the results of PIRs, should be documented.



Questionnaire

Information Technology Investment Evaluation Guide Questions

PLANNING PROCESS

Screening New Projects

  1. Does the organization have a defined process for submitting and screening new funding proposals for management consideration? Is this process established in policy guidance?
  2. Does the process define:
    • what information is to be submitted?
    • who must approve (screen) the information prior to formal submission?
    • how a determination will be made about where in the organization the project will be reviewed?
    • the stages of management review?
  3. Are roles, responsibilities, and authority for people and offices involved in the screening process clearly defined?
  4. What information is required for submitting projects for funding? (Please check off those that apply and include others.)
    • For most projects this information may include:
      • business case justification, including:
        • clear, designated senior management sponsorship from a program/business unit
        • links to business/program/mission objectives that the project is helping to achieve, as well as an explanation of how the IM/IT investment will directly or indirectly help achieve the intended outcomes associated with these objectives, and
        • clear identification of proposed benefits (both quantitative and qualitative)
        • cost/benefit estimates
      • alternative and sensitivity analyses
      • compliance with the information technology architecture
      • risk assessments
      • other

  5. Do defined thresholds for benefit/cost ratios, return on investment calculations, risk assessments, etc. exist? Are these thresholds clearly defined and understood?
  6. Are all funding proposals treated the same or does the organization have different process requirements for funding proposals of different size, scope, etc.? Are these different requirements adequately documented?
  7. If exceptions to the screening process are allowed, are the conditions for allowing exceptions clearly documented?
  8. Does the process clearly stipulate potential actions that can be taken for projects that are funded without evidence of following the screening process?

    Analyzing and Ranking All Projects Based on Benefit, Cost and Risk Criteria


  9. Does the organization require that the information and data submitted with funding proposals be validated (accuracy, reliability, completeness)? Does the process stipulate who is responsible for performing this validation (sponsor, project team, independent validation and verification teams, etc.)?
  10. Does the process stipulate where exceptions to validation are permitted? If exceptions are allowed, are other clearly defined conditions required to be met?
  11. Does the organization have an established, documented process for comparing and ranking all proposed IM/IT-related funding?
  12. Has the organization defined explicit criteria that will be used to help compare and rank projects? If no, go to 17. Do these criteria include cost, risk, and benefit elements (e.g., benefit/cost ratios, anticipated or actual impact on mission improvement priorities, risks versus benefits, etc.)? Do the criteria include both quantitative and qualitative criteria (e.g., return on investment calculations, risk modeling scores, capability assessments, alignment with critical needs, etc.)?
  13. Are the decision criteria weighted? If no, go to 14. If scoring weights are attached to different items for management consideration, are these clearly defined and understood by participants? Has management consensus been established on use of the weighting scheme? Are these weights being applied consistently to all project proposals? If not, has the organization established different weighting schemas for different project types?
  14. Does the process explain how the decision criteria are to be applied?
  15. If the organization uses a scoring model or decision support tool associated with the decision criteria to help measure the relative costs, benefits, and risks of each project, are the scoring elements precisely defined and differentiated?
  16. Is the process for analyzing and comparing IM/IT projects required throughout the organization, regardless of where actual funding decisions/approvals are made?
  17. Are the criteria used to compare and rank projects weighted? How were the weights determined? Is the weighting scheme reassessed periodically?
  18. Does the process include incentives or disincentives to ensure compliance? Are roles and responsibilities for enforcing the process defined?
  19. Does the organization require that management evaluations, as well as scoring, ranking, and prioritization results be documented (either manually or through the use of automated applications such as a decision support tool)?
  20. Does the organization require that this information, together with approved project data on cost, schedule, and performance, be tracked at an aggregate level? At a project level?

    Selecting a Portfolio of Projects


  21. Does the organization require that this information be entered into a recognized management information system? Does it require the data to be maintained in a uniform fashion?
  22. Does the organization have a formal systematic process for determining priorities and making funding decisions? Does the process clearly establish who in the organization has the responsibility and authority for making final IM/IT related funding decisions? Is the process clear in establishing this responsibility and authority for various organizational levels (department/branch/ sector/directorate/region/units, etc.)?
  23. Has the organization established an IM/IT investment review group (or some other review function)?
    • Who is on this review group?
    • Does the process cover the roles, responsibilities and authority of this group?
    • Is it clear on the role the group will play in:
      • selecting, controlling and evaluating IM/IT investments, and
      • suggesting changes to organizational policies, procedures and practices?
    • Does the group have authority to:
      1. approve, cancel or delay projects
      2. approve plans for mitigating risks
      3. validate expected returns
      4. place constraints on investment proposals regarding project size and duration?
    • Does the process stipulate the operating rules and procedures for this group, (i.e. explaining when it will meet, how it will function, how decisions will be made, what steps will be taken to resolve conflicts, etc.)?
    • Is it clear what projects the investment review group (or similar management group) will review (eg. all IM/IT spending proposals or IM/IT spending proposals that meet or exceed defined decision thresholds based on cost, level of risk, cross-functional, bureau, or office impact, or involving common infrastructure needs such as telecommunications, data centres or networks)?
  24. Are IM/IT decisions made as part of an overall capital planning process or are IM/IT projects separated out? Does the process explain how decisions made on IM/IT spending will be incorporated into the organization's overall budgeting or capital programming decision making process?
  25. Does the process require that data on obligations, outlays, and actual expenditures be maintained for all IM/IT spending? Are categories of IM/IT spending defined within the organization, eg. hardware, infrastructure, telecommunications, operations and maintenance, applications (development data processing services, personnel, etc.)?
  26. Has the organization conducted a review (inhouse) or via outside consultant/expert) of its current IM/IT spending portfolio to assess alignment with mission needs, priorities, strategic direction, or major process re-engineering?
    • Has the rate and type of IM/IT spending been aligned with management expectations?
    • Has trend analysis been done to show how patterns of investment and spending are changing?
    • Has an analysis been conducted to show how spending patterns could be affected by the proposed IM/IT portfolio?
    • Does the organization have a process for documenting and disseminating results of this review?
  27. Does the process define how unit or office level IM/IT decisions will be reviewed?

    Establishing Project Review Schedules

  28. Does the process stipulate how approved projects are to be monitored by senior management in regular investment meetings? Are there any procedures for informing project managers of decisions about monitoring schedules made by the investment review group?
  29. Is the review process clear on criteria for deciding the kinds of projects that will receive regular management monitoring and oversight by an investment review group versus those that will be monitored exclusively by management sponsors?
  30. Does the process allow the investment review group to call special project reviews outside of regular meetings if the group deems it necessary?
  31. Does the process require any additional certification or reviews before high-risk projects are allowed to proceed (eg. risk mitigation plans, additional cost certifications, etc.)?

    Evidence that Each Project has Met Project Submission Requirements

  32. For IM/IT proposals that were submitted for funding consideration, was all of the required data/information prepared and submitted in accordance with the prescribed process?
  33. Is there evidence that the data/information (cost, schedule, performance, risk) for submitted projects has been validated either independently or using a self assessment process?
  34. Did the information/data presented in the proposals come from organization recognized information systems (automated or otherwise)?
  35. Is the information/data easily accessible for further review and consideration?

    Analyses of Each Project's Costs, Benefits and Risks


  36. Are project cost data fully identified direct, indirect, ongoing? Do the data include full lifecycle costs? Did these cost data come from a recognized financial system?
  37. Have benefits of the investment been identified using quantitative and/or qualitative data/information that relate directly to mission support and performance improvement? Are expected cost savings, productivity gains, and improvements in quality identified and timeframes specified as to when these should occur?
  38. Have all foreseeable risks been identified? These risks may include technical, managerial, capability, procurement, organizational impact, stakeholder, etc. Have all concerns and potential problem issues been resolved or accounted for in a risk mitigation strategy for the project?
  39. Have business owners been involved in constructing and certifying the accuracy of the data that are submitted.
  40. Were baseline cost and performance data used as a basis for analysis? Is this information reliable and accurate? If baseline data were not available, were the estimates generated using a prescribed approach or method?
  41. For projects that are requesting continued or additional funding (for a new phase, new module, or as a result of increased costs), is there evidence that the newly submitted data reflect any changes that may have occurred to the cost, benefit, or risk information?
  42. Has project schedule information been reviewed in light of competing priorities; skills, capabilities, availability of organization staff; contractor expertise and experience, etc.
  43. Was the cost and return information that was submitted constructed using accepted methods and techniques (prescribed by the organization, legislative provisions, and/or accepted industry practice)?

    Data on the Existing Portfolio


  44. Does the organization maintain data on its current IM/IT spending portfolio (eg. are major categories of spending and investment defined and tracked such as operations and maintenance, applications and systems development, hardware acquisitions, telecommunications, personnel, contracted services, data administration, research and development etc.)?
  45. Are the costs and returns of the IM/IT spending portfolio presented on an aggregate basis (past, current, future)? On a project basis?
  46. Was the portfolio information (IM/IT spending categories, cost/benefit estimates average development costs, etc.) derived from recognized organizational management information systems? Are standard definitions and reporting elements used to maintain consistency and uniformity?

    Scoring and Prioritization Outcomes


  47. Are summary data presented on each project's costs, benefits, risks for senior management (investment review group, etc.) to consider?
  48. Does the management review group conduct scoring exercises to vote on the relative strengths and weaknesses of proposals? Are these scoring exercises recorded and documented? Are the criteria that are used as the basis for the scoring instruments defined and used consistently?
  49. Can the costs of the approved list of projects for funding be tracked to available funds and/or reflected in the budget requests?
  50. When management approves funding for projects that fall outside accepted thresholds (high risks, high project costs, noncompliance with the architecture etc.), is an explanation/rationale provided for this decision? Are additional management and project reporting requirements stipulated?

    Project Review Schedules


  51. Once projects are approved for funding by an investment group and/or organization head, are any additional project management or investment review reporting requirements (data, information, analysis) established for high risk, high cost projects (beyond what may be specified by existing processes)?

    • If so, have these requirements been clearly documented and communicated to the responsible project team?
    • Is it clear why this data/information is being requested and what it will be used for?
    • Has an explanation been given to the project team explaining how this information and its assessment by senior management may influence project continuation, delay or cancellation?
  52. Did each project that was approved have an outline or strategy developed establishing how any necessary acquisitions would be handled? Is this strategy appropriate given the project type?

    Determining Whether Projects Met Process Stipulated Requirements


  53. Are decisions being made about project readiness for review using the data/information requirements established by a project screening process?

    • Are project submissions being reviewed consistently (using the process and information required)?
    • Are screening decisions being recorded?
    • Is there evidence of projects being rejected.
    • Were explanations for submission rejections documented and communicated to the business sponsor?
    • If exceptions are being made to screening criteria, is the explanation documented and forwarded with the project proposal?

    Deciding Upon the Mixture of Projects in the Overall IM/IT Investment Portfolio


  54. Do the systems that were selected for the current portfolio represent the best match with mission needs. Do all of the selected projects support objectives or goals established in the organization's strategic plan or annual performance plans? Are all of the selected projects justified on the basis of their relative costs, benefits and risks? Are there any other factors that appear to have influenced the executives' decisions?
  55. For ongoing projects, have projected versus actual data on costs and interim results been used to make decisions about project continuation?
  56. Were project decisions made at the appropriate organizational level?
  57. Do you know the ratio of funding between the various project types (new, proposed, under development, operational, etc.) that were selected? Does this ratio appear to effectively support the organization's business plan and objectives?

    TRACKING & ASSESSMENT

    Consistently Monitoring Projects

  58. Does the organization have a defined, documented, and repeatable process for monitoring and reviewing IM/IT projects? Does this process define what the focus of the investment reviews will be? Some key elements that may be included in the review include the following:

    • project status including where the project stands in relation to the development of other projects
    • business sponsor evaluation of the project estimated vs. actual costs to date
    • estimated schedule vs. actual schedule
    • actual outcomes from modular testing pilots, prototypes or limited site testing vs. estimates
    • technical performance as well as estimated impact on program or business performance
    • updates on risk mitigation efforts including identification of any new risks and steps to address technical issues or problems that have arisen
    • contractor performance and delivery review of the methodology or systems development process
    • new unexpected issues affecting project progress or outcomes
  59. Does the process stipulate what project data and information must be submitted for management evaluation?
  60. Does the process indicate how data and information being presented for management review are to be verified and validated? Are roles and responsibilities for conducting this verification and validation spelled out?
  61. Does the process define a specific group (or groups) of managers that are responsible for conducting IM/IM/IT investment control meetings? What is this group called? Are procedural rules for how the investment review group will operate and make decisions clearly defined and documented? Are the roles, responsibilities, and authority of this group(s) clearly defined? Is the purpose of the investment review group clearly stated? Are the types of decisions that will be made at the IM/IT investment control meeting defined (e.g., project continuation, delay, cancellation, termination, acceleration, etc.)?
  62. If investment review processes are used across different units of the Branch (e.g., department, branch, sector, region, directorate, unit, etc.), do the different units have consistent policies, practices, and procedures for monitoring projects and reaching decisions?
  63. Does the process make accommodations for flexible reviews (frequency, required report submissions, etc.) for different kinds of projects (high risk/low return vs. low risk/low return)?
  64. Does the process define who is accountable for acting on review decisions? Is it clear the types of actions that are under different people's responsibilities (e.g., project team, CIO staff, business sponsor, financial staff)?
  65. Does the process define how open action items are to be addressed? Does the process define roles and responsibilities for making these decisions, as well as criteria for evaluating actions that are taken and determining whether the open item has been resolved?
  66. Are there mechanisms in place to help ensure compliance with the review process? What are the mechanisms? Are there disincentives in place for noncompliance? Is there someone responsible for overseeing the process? If so, who is that person? Is the process consistently maintained?

    Involving the Right People


  67. Who is involved in ongoing project reviews and decisions? Do the review groups include staff from program, IM/IT, and financial offices? Is there membership by a quality assurance or some other outside assessment group?
  68. Are project managers or site executives included in devising and approving actions to address deficiencies that were identified?

    Documenting All Major Actions and Decisions


  69. Does the organization define the various pieces of information that are to be updated and maintained? Does the organization define the various pieces of information that are to be updated and maintained. This information may include the following:

    • project related decisions that are made
    • actions that are to be taken as well as criteria or measures for evaluating improvement
    • project outcomes
    • cost/benefit analysis and other associated project information
    • business case information
  70. Does the process stipulate where these data are to be maintained (e.g., official organization information system with uniform data standards and entry procedures)?

    Feeding Lessons Learned Back Into the Planning Phase


  71. Does the organization have a process for evaluating current decision-making processes and suggesting changes to these processes based on lessons that are learned from investment control reviews?

    • Does the process account for and distinguish between senior management decision-making changes and project-level management changes?
    • Does the process specify someone who is accountable for identifying lessons that are learned from investment control reviews?
  72. Is there a process for refining or updating the selection criteria (both screening and ranking) based on lessons that are learned?
  73. Does the organization have a process for aggregating data/information across all major IM/IT projects (or spending categories) in order to compile an overall organizational track record on costs and benefits attributable to IM/IT?

    • Is someone (or office) charged with this specific responsibility?
    • Does the organization have procedures for presenting this information to the IM/IT investment review group?
    • For presenting it to all agency executives?

    Measures of Interim Results


  74. Are specific measures of performance being used to track costs, schedule, benefits, and risks? Are these measures updated throughout each project's life-cycle as costs, benefits, and risks become better known?
  75. Are data being used to track actual project performance (interim results) against estimates that were used to justify the project?
  76. Are gaps or differences between estimates and actuals being analyzed and explanatory factors documented for positive or negative variances?
  77. Is there documentation to support that interim cost, schedule, benefit, and risk information has been validated or verified? If risks have changed, is this supported by documented data or analyses?

    Updated Analyses of Each Project's Costs, Benefits, Schedule and Risks


  78. In investment control meetings, has the information in project business cases been updated to reflect the current state (including project costs to date, current risks and mitigation plans, interim benefit or performance results achieved, etc.)?
  79. Are project-level data (current and historical) being maintained and updated using organization approved databases/information systems? What are the databases or information systems that are used?
  80. Are changing business assumptions or environmental factors (political, funding, stakeholder support) identified and documented? If so, are the impacts of these factors on project outcomes evaluated?
  81. If a project is behind schedule, do risk mitigation plans get updated and are explanatory factors thoroughly evaluated?
  82. If contractor assistance is being utilized, are contractor performance reports conducted and results made available for management consideration?

    Deciding Whether to Cancel, Modify, Continue or Accelerate a Project


  83. Is the organization reviewing projects and using data analysis to make decisions about the future of each project? Were the decisions that were made reasonable given the situation and condition of the project? For those projects whose status remained stable, was a conscious decision made that the project should proceed or was it implicitly assumed that the project would continue?
  84. If problems were identified, was a decision made about what actions should be taken? Who made this decision? Is there some explanation or justification given for this decision? Were the actions that were taken appropriate to the problem (i.e., was the problem an IM/IT problem or a business/process problem)?
  85. What evidence is there, if any, that actions have been taken based on the results of project reviews? For instance, if the organization determined that new requirements have been introduced, what actions were taken to respond to these additional requirements? Were these actions documented? Did these actions fully address the requirements?
  86. If decisions are made that affect a project's funding, such as suspending project funds or cancelling a project, is there evidence in budget documents and spending information that reflects this decision? Are there criteria identifying what must be done for funding to resume?
  87. Are future "cascading" actions resulting from project decisions clearly identified and delineated?
  88. Is a management group following organization policies, procedures, and practices in making regular decisions about the continuation of major IM/IT projects?
  89. Are project data being used to make decisions and take action on IM/IT projects that are subjected to investment reviews?
  90. Are decisions being made at the right time (i.e., as prescribed by the agency process or as agreed to by senior management as part of project approval)?
  91. Are decisions regarding projects being executed? Is accountability and follow-up clearly defined?
  92. Is an independent review conducted afterward to ensure that actions were taken and to make further changes as necessary?
  93. If projects are allowed to proceed when investment review data and analyses raise serious questions about the project, has documentation been provided detailing how management reached its decision?

    Aggregating Data and Reviewing Collective Actions Taken to Data


  94. Has the organization aggregated data in order to assess organizational performance and to identify patterns or trends? At what levels C unit, division, agency, departmental C has information been aggregated? Is this information being fed back in to decision makers to help make future decisions?

    EVALUATION PROCESS

    Conduct Post Implementation Reviews (PIRs) Using a Standard Methodology


  95. Does the organization have a defined, documented process for conducting post implementation reviews (PIR) of IM/IT projects?

    • Is the purpose(s) of the PIR process clearly explained and communicated?
    • Is the process clear about when PIRs are to be conducted?
    • Are PIRs required on a regular basis to ensure that completed projects are reviewed in a timely manner?
    • Does the process delineate roles, responsibilities and authorities for people and offices involved in conducting the PIRs?
    • Does the process help ensure that these assessments are objective?
    • Does the process stipulate how conclusions and recommendations resulting from PIRs are to be communicated to and reviewed by senior management?
  96. Does the organization have a standardized methodology for conducting PIRs? Does this methodology, at a minimum, include assessments of customer satisfaction, mission/programmatic impact and technical performance/capability? Is the methodology required at all levels of the organization?
  97. Does the process define how the outcomes of PIRs are to be addressed? Are roles and responsibilities defined for taking action to address any concerns or problems that are identified?
  98. What Steps does the organization require to ensure that PIRs are conducted independently and objectively? Are the results of the PIRs validated or verified?
  99. Are the causes of project and process problems identified as part of the PIR?

    Feeding Lessons Learned Back into the Planning and Tracking & Oversight Phases


  100. Does the organization process or methodology for conducting PIRs include provisions for:

    • changing or improving existing management decision making processes; and
    • strengthening project level management?
  101. Does the organization have a mechanism for tracking (over time and for different kinds of projects) and aggregating the results of PIRs that are conducted?

    • Are the results of PIRs collected and maintained (in a manual or automated database)?
    • Is the information reported in a timely manner?
    • Are the results easily accessible? Has the organization identified roles and responsibilities for collecting and analyzing PIR report information?
  102. Does the organization have procedures for regularly reporting PIR results to senior management? Is this information reliable, accurate and easily accessible?
  103. Does the organization have procedures for regularly assessing the PIR process for completeness, quality and contribution to project level and executive decision making?

    Measurements of Actual vs. Projected Performance


  104. Is the organization collecting projected versus actual cost, benefit and risk data as part of the post implementation reviews?

    • Has the cost, benefit and risk information that was used for initial project justification been preserved?
    • Have updates that have been made to costs, benefits, or risks been noted? Have these updated also been preserved?
    • Have project benefits that were obtained been quantified?
    • If not, are qualitative measures being used to determine impact?
    • Have the cost data been verified or validated?
    • Are these data contained in a recognized organization financial management/accounting database?
  105. Does the PIR include assessments of customer satisfaction (end users, business or program unit sponsor, etc.)?
  106. Does the PIR include assessments of technical capability (eg. conformance to recognized systems development methodology, architecture compliance, contractor performance and oversight)?

    Documented "Track Record" (Project and Process)


  107. Does the organization routinely record its evaluation activities?
  108. Has the organization conducted trend analyses (descriptive or statistical) using the results of PIRs that have been conducted?

    • Do these analyses attempt to correlate actual project performance results with factors that may have caused these results (positive and negative)?
    • Are the results of these analyses presented to management as a regular part of the investment decision making process?
    • Are special reports issued to executive management?
  109. Are recommendations for specific projects and senior management decision making processes presented in PIRs?

    • Do these recommendations cover changes to process, data or decision making procedures used for both the Planning and Tracking & Oversight phases?
    • Are these recommendations well documented and supported by existing analyses?

    Assessing Projects Impact on Mission Performance and Determining Future Prospects for the Project


  110. What decisions have been made by senior management (or investment review group) regarding whether implemented projects met defined criteria for successful outcomes?
  111. Were corrective actions for specific projects included in senior management's decisions?

    • Were timetables and steps for implementing these changes established as part of the decision?
    • Were follow-up management reviews established?
    • Has a clear purpose for these reviews been defined?
  112. Has a plan been developed detailing how future O&M and disposition costs will be addressed?
  113. Are decisions that are being made on specific projects cognizant of the potential (or actual) impact the decision may have on other related projects?
  114. Have decisions regarding the status of projects been finalized? Have expected changes been communicated to the project manager?

    Revising the Planning and Tracking & Oversight Based on Lessons


  115. What decisions have been made to modify existing organizational IM/IT investment management processes?

    • Have these decisions been communicated to staff?
    • Are changes to existing processes, operating procedures and data requirements aligned with conclusions and recommendations documented in PIRs?
    • Has the organization clearly established and communicated when these changes to existing management processes will take effect?