ARCHIVED - IM/IT Investment Evaluation Guide

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Planning Phase: Choosing the Best IM/IT Investments

1.1 Planning Processes

The IM/IT investment management process begins with the project planning process. Projects being proposed for funding are put through a "coarse-grained" screening process to:

1. eliminate proposals that fail to pass minimal acceptance criteria; and
2. ensure that projects are being reviewed at the most appropriate organizational level (Department, Branch, Sector, Region, Directorate, Unit etc.).

Proposals that pass this screening process have their costs, benefits, and risks analyzed in-depth. Once this is accomplished, all of the projects are compared against some common decision criteria and ranked based on their relative benefits, costs, and risks. Using this prioritized list as a guide, senior managers make decisions about which projects will be proposed for funding for the upcoming year. This post-prioritization decision-making on the appropriate mixture of projects is the essence of IM/IT portfolio analysis. Finally, after these funding decisions have been made, schedules for reviewing projects are established or updated.

1.1.1 Screening New Projects

The organization should have a process that outlines how to introduce projects for funding and how these projects will be screened for relevancy to business/program goals and objectives and technical soundness. Specifically, the organization should:

• define what constitutes an IM/IT project;
• identify initial requirements that projects must meet in order to be seriously considered for funding,
• explain how screening will be conducted, and
• establish roles and responsibilities for conducting the screening.

The screening process should be established in policy guidance (to ensure that it is conducted consistently) and used at all levels of the organization. As part of the initial screening process, there should be documented screening criteria (minimal requirements) that all projects are expected to meet.

The screening criteria should serve three functions. They should:

• a) Identify whether the project meets initial acceptance requirements;

• There should be some initial requirements that projects must meet before they are reviewed in greater detail. These initial requirements may include return-on-investment (ROI) thresholds (or minimum cost/benefit ratios), identification of the project's link to objectives in the business or strategic plans, evidence of compliance with the organization's information technology architecture, identification of business/programmatic sponsorship, and assurance that all necessary project proposal and justification steps have been performed.

• b) Ensure that the project is being reviewed at the most appropriate organizational level

• Not all projects need to be reviewed at the same organizational level. The criteria for screening projects should be structured so a determination can be made about where in the organization a decision would best be made. Certain cost and risk thresholds can be used to determine what requires centralized (Department or Branch level) versus decentralized approval (Sector, Region, Directorate, or Unit level).

• c) Identify what level of management scrutiny is appropriate given the project's type, size, and risks.

• There should be flexible but defined rules explaining how project review and approval may vary based on the project's relative costs, benefits, and risks. Low-cost, low-risk projects should not need to have the same justification provided for them as projects of greater cost, risk, and organizational impact.

On the basis of this screening process, projects will either move on for more in-depth analysis or will be sent back to the originating program group.

1.1.2 Analyzing and Ranking All Projects Based on Benefit, Cost, and Risk Criteria

The cost, risk, and benefit information of all projects (initial concept, proposed, under development, operational) should be analyzed and assessed in detail.

Each project should have a business case developed that provides the sponsor's justification for the project. The business case should identify the organizational needs that the project is meeting or proposes to meet; provide information on the benefits, costs, and risks of the project; and establish proposed project development time frames and delivery schedules. The information in the business case should be continuously updated to ensure that it always reflects the current situation.

The organization should have some established group or audit function that is responsible for verifying and validating the various analyses (cost/benefit analyses including feasibility studies, risk assessments, and alternatives analyses) and information that are submitted as part of a project's business case. This validation should include:

• reviewing assumptions that were made;
• assessing all of the alternatives that were analyzed and determining whether others should have been included;
• reviewing the cost and benefit estimates to ensure that they were accurate and realistic;
• evaluating the risks that were identified and determining whether others may be applicable;
• evaluating the sensitivity analyses that were conducted.

The organization should have a management information system (MIS) or some other mechanism where all project information is collected and maintained. Such a mechanism, if kept accurate and up-to-date, can make data verification and validation easier by allowing the organization to track costs, risks, etc. over time.

This mechanism for collecting and maintaining project information will also be essential during the Control and Evaluation phases to:

1. help assess whether projects are still aligned with mission needs and organizational objectives,
2. determine whether projects are meeting planned performance goals, and
3. identify possible revisions to the overall investment management process based on previous experiences and lessons learned.

After each project's cost, risk, and benefit information has been examined and validated, all of the projects should be compared against some common decision criteria in order to weigh the relative merits of the projects and develop a prioritized listing of projects.

The criteria used for assessing and ranking projects should consist of elements related to three essential areas - benefits, costs, and risks. Often organizations will establish broad categories related to these three areas and then develop more specific sub elements that come under each broad category. For example, an organization may establish risk as a categorical heading and then include schedule risks, cost sensitivity, technical risks, organizational risks, and risks if the project is not undertaken as sub elements under the risk heading.

Different organizations will break these broad categories and sub elements out in different ways. For instance, some organizations may include a project's costs as one of several factors under risk, while others break project costs out as a separate category.

Decisions should rarely be made based on one project factor, such as the project's estimated cost or a projection of reduced cycle time. Using an assortment of decision criteria to make decisions allows an organization to take into account and compare the different subtleties of a wide variety of projects.

The organization may assign weights to each of the broad categories, as well as any sub elements related to each category, in order to help prioritize those factors that the organization considers to be the most significant (e.g., a company that has limited experience developing systems may give technical risk a greater weight than projected cost). The mixture of weights among the ranking criteria will vary from organization to organization. The weights that are given should take into account the organization's unique mission, capabilities, and limitations. The weighting schema that the organization establishes should be defined and documented. Such documentation is even more important if different weighting approaches are used for different kinds of projects (operational, infrastructure, applications development projects, etc.).

To provide senior managers with an understanding of the relative costs, risks, and benefits of each project compared to the other projects, the organization may develop a scoring model or decision support tool. Such a tool compares the costs, benefits, and risks of each project against the cost, benefit, and risk criteria and assigns a score for each factor. The scores that the project receives for each factor are then added up to produce a cumulative score that establishes the project's relative worth and allows comparison against all other projects.

An important point for an organization in developing such a scoring model or decision support tool is to precisely define the scoring elements. The purpose behind these definitions is to ensure more consistent or uniform objectivity in the scoring process, which helps to eliminate widely varying interpretations and implementation.

The criteria for comparing and ranking projects should be used uniformly across the organization (i.e., unit, division, directorate, Region, Sector level decisions should be made using a set of criteria that are similar to criteria used for Branch or Department-level decisions). Although different levels of the organization may use additional criteria, the organization should have a set of minimum criteria that are used enterprise wide. Using some common decision criteria provides greater assurance that the organization is selecting projects consistently and helps to avoid "apples versus oranges" project comparison problems.

There should also be incentives to ensure compliance with the process and to dissuade gamesmanship. The organization should identify who is responsible for enforcing the process and there should be explicit consequences for noncompliance.

1.1.3 Selecting a Portfolio of Projects

The organization should have a senior management decision-making body, made up of program, and financial managers, that makes decisions about which projects to fund for the year based on its determination of where organizational needs are greatest. Such a determination will usually be made by analyzing the gap between the organization's goals and objectives (as highlighted in its strategic and annual performance plans) and the organization's existing capacity.

The roles and responsibilities of the IM/IT investment review group should be clearly identified and documented. The organization should also identify how this group will go about making decisions. This should include establishing how decisions will be reached, how conflicts will be handled, and how stakeholder input will be brought into the process.

The investment review group will make decisions about which projects to propose for funding, using the list of ranked projects as a key input. As the group goes about making these decisions, a number of trade offs will have to be made. For instance, the group will need to decide how much should be spent to continue operating and maintaining existing systems, versus funding enhancements to current systems, versus funding systems that are currently under development, versus funding new projects, versus funding research projects that assessing the applicability of emerging technologies. The group must also determine the proportions that will be spent on the various IM/IT types (i.e., research and development, administrative, mission critical, infrastructure, etc.). And, the group must take into account dependencies among projects.

The decision-making process should help address difficulties associated with using different units of measure for analyzing different kinds of IM/IT projects, as well as a balancing of "soft" versus "hard" quantitative data.

To aid the investment review group in making trade offs between various project types and phases, the organization may maintain a data repository that contains historical information on expenditures in different IM/IT investment categories (operations and maintenance, enhancements to current systems, new systems development, research into developing or applying emerging technologies, etc.). By maintaining this information, the organization can review how much was spent previously and factor this in to current spending decisions.

As part of the process of making trade offs and determining spending priorities, the organization may also conduct a review (in- house or via outside consultant/expert) of its current IM/IT spending portfolio to assess alignment with mission needs, priorities, strategic direction, major process re-engineering, etc. This review may include a trend analysis to show how patterns of investment and spending have changed, as well as an analysis to estimate how the spending pattern may change with the proposed IM/IT portfolio.

No matter how rigorous or structured the organization's decision- making process is, decisions about which projects to select for funding are ultimately managerial decisions. If senior managers select projects that score low when compared to other projects (e.g., high-risk, high-return projects) the justification for these decisions should be documented and the project's progress should be closely monitored during the Control phase. Making such exceptions should be kept as minimal as possible, however, to preserve the integrity of the decision-making process.

The process of reviewing and selecting IM/IT projects should be explicitly linked with other business processes (e.g., planning, budgeting, acquisition). Most investment decisions should mirror a planning decision or business objective and should be reflected in related budgeting documents and decisions.

The investment review group's responsibilities will usually not end once it has decided upon the mix of projects that will be proposed to comprise the current year's investment portfolio. Instead, the group should meet on a regular basis (often quarterly) to discuss the status of projects and to make further project decisions. The group may also be responsible for reviewing investment portfolio decisions that were made by lower-level organizational units.

1.1.4 Establishing Project Review Schedules

After making funding decisions, each project that was selected should have a review schedule established for it, or should have its current review schedule assessed and updated as needed. The time frames for these reviews will depend on various project-specific factors (amount of risk, investment size, mission importance, capability of the project team, etc.).

It is important that these reviews be conducted on a regular, scheduled basis. These reviews do not necessarily have to coincide with major project milestones. Moreover, "review triggers" should be established that automatically require a management review meeting. For example, a cost, schedule, or performance deviation of 10% or greater might require an immediate project review.

1.2 Planning Data

Good decisions require good data. Ensuring that each project meets established screening and ranking requirements and that the project's information is accurate and up-to-date is essential for ensuring that the most critical needs of the organization are being met by the projects and systems that are selected. In addition, the ex post information that is generated during this phase, such as project review schedules or risk mitigation plans, based on the planning decisions that are made, is critical for controlling and evaluating projects during the next two phases.

1.2.1 Evidence That Each Project Has Met Initial Project Submission Requirements

The efficiency of the investment management process depends initially upon how well the organization is ensuring that all projects meet initial project acceptance requirements and that necessary project proposal and justification steps have been performed. There should be evidence that each project that is submitted has been screened, analyzed, and evaluated according to processes and criteria established by the organization.

The information that is analyzed may include verification that all requisite planning data were submitted, that answers were received for all relevant questions, that projects met business/program goals and conformed to the agency's information technology architecture, and that projects that did not meet these requirements were not allowed to move on for further review and consideration. There should also be evidence demonstrating that all business units adhered to organizational policies and procedures regarding the screening and acceptance of projects.

Much of the evidence that will be reviewed will consist of cursory completeness and quality checks. For instance, if the organization has requirements that all projects over a certain cost threshold must

1. submit complete cost/benefit and risk analyses,
2. identify the business objectives that the project is meeting, and
3. provide assurance that the project conforms to the organization's technical architecture, then a review of projects that went on for further review should not identify any projects that did not meet these initial requirements. Evidence should also be available demonstrating that each project adhered to the documented process.

There should also be evidence that information that was submitted was validated by a quality assurance/control function. Such validation can be performed by in-house quality control/quality assurance staff, internal audit staff (e.g., inspector general), etc. The project information should also be verified to ensure that it is accurate and reflects the most up-to-date information.

All project information should be up-to-date, cost numbers should be accurate, benefits should be quantified to the extent possible, risks should be spelled out, alternatives should be identified, and sensitivity analyses should have been conducted.

1.2.2 Analysis of Each Project's Costs, Benefits, and Risks

Each project that is submitted should have a business case prepared that provides justification for the project. Included in the business case should be identification of the project's functional requirements and estimates of the project's life-cycle costs, benefits, and risks (to the extent possible), as well as the corresponding analyses that were conducted to develop the estimates. Making accurate cost savings estimates and benefit determinations requires having at least a rudimentary understanding of the baseline costs and benefits from existing IM/IT capabilities.

A key analysis that should almost always be submitted with project proposals is a cost/benefit analysis. A complete cost/benefit analysis should

• identify and quantify benefits and costs,
• identify assumptions and constraints that were used when developing these figures,
• evaluate alternatives using net present value, and
• perform risk and sensitivity analyses.

The amount of rigor and types of analyses that are conducted will depend, in part, on the size of the investment and the amount of risk. It may not be economical to conduct an in-depth cost-benefit analysis for a low-cost, low-risk project that only affects a specific division or office or a limited number of users. The organization should have a process that outlines what project data are required given each project's type, cost, and risks; variation in the quality or type of data should not be ad hoc.

Listed below are some of the cost, risk, and benefit elements that an organization should keep in mind as IM/IT develops project estimates.

Costs (recurring and non-recurring)

• Up-front costs, such as hardware and software purchases, costs to design and develop the project, transition costs, etc...
• Ongoing costs, such as salaries, software upgrades, training, supplies, operations and maintenance, etc...
• Indirect costs, such as initial productivity losses, computer support (network management, data administration, hotlines), etc...

Risks

• Project risks, such as the size of the investment, project size and longevity (is the project designed in modules, does the project rely on the implementation of other systems, do other systems rely on this project), has the project group managed other projects of similar risk and complexity.
• Organizational risks, such as mission risks if the project does not proceed, program management's commitment to the project, political expectations for the project, whether the project is legislatively mandated, etc.
• Technical risks, such as skills required, hardware and software dependencies, application software, etc.

Benefits (will usually consist of both tangible and intangible benefits)

• Tangible benefits include benefits that can be explicitly quantified. Such benefits may include reducing costs, increasing productivity (e.g., reducing errors, eliminating duplication or needless work steps, etc.), decreasing cycle time, or improving service quality (e.g., timeliness, convenience, access, reliability, etc.).
• Intangible benefits include benefits that may be easy to identify, but that are difficult to quantify. These benefits may include faster, more efficient decision-making; greater data accuracy; improved data security; reduced customer burden; or increased organizational knowledge.

In identifying and measuring IM/IT benefits, it is important to always remember the business function or process that is being supported by the technology. For instance, the benefits that are gained from implementing EDI technology are derived from the increased capability and efficiency that the technology provides to the organization and its customers.

All of the information in the business case should be as up-to-date and accurate as possible. If the analyses are to yield meaningful results, it is essential that the project team carefully formulate assumptions, identify feasible alternatives, and provide realistic cost and benefit estimates.

Most agencies have criteria or methodologies detailing how cost/benefit analyses are to be conducted and what should be included. In addition, OMB Circular A-94 provides guidance and discount rates for conducting cost/benefit analyses.

1.2.3 Data on the Existing Portfolio

Because all projects (ongoing, under development, etc.) go through the Planning process (usually on an annual basis), portfolio data from previous years should be available to assess and compare previously selected projects. The spending, cost, and obligation data in this portfolio should be up-to-date and categorized in ways that are most meaningful to organization management.

An agency's cost accounting system should be able to distinguish between what has been obligated to date and what is still available, as well as identify what the incurred costs to date were for. In addition, the system may be able to split spending into more specific categories, such as development, operations, maintenance, etc. (Activity-based cost tracking, for example, should provide this detail.)

1.2.4 Scoring and Prioritization Outcomes

There are several pieces of information that should arise out of the Planning phase, based on the actual decisions that are made. This information includes:

• the initial project scores and ranked list of projects,
• the investment review group's scores based on any additional decision-support tools,
• the investment group's final list of projects that will make up the investment portfolio,
• documented justification for selecting projects that scored below accepted thresholds (e.g., high-risk, high-return projects), and
• funding information, as well as acquisition and development schedules, for all projects that were selected.

The organization should also be maintaining net cost and benefit information on the complete portfolio of IM/IT investments.

Finally, all of the projects that were selected for funding should be included in the Organization Capital Plan that is submitted to OMB. Information that is submitted in this plan should include baseline cost, schedule, and performance goals for each project.

In addition to the Capital Plan, decisions that are made on the mix of existing and new projects should be clearly identified in the agency's annual performance plans. Actions described in the Capital Plan to implement the funding, procurement, and management of the IM/IT projects should also be articulated in these performance plans.

1.2.5 Project Review Schedules

All projects that are selected for funding should have project review schedules, risk management plans, and project-specific performance measures established. All of this information will be particularly critical for assessing performance, identifying risks, and making decisions during the Control and Evaluation phases.

The timing of reviews, as well as the number of reviews that will be conducted, will depend on the investment size of the project, the amount of risk, the capability of the project team, etc.

In addition, the investment review group may identify additional project management or investment review reporting requirements (data, information, analysis), beyond what is specified by existing processes, for projects that it determines are particularly high-risk. These additional requirements should be clearly documented and communicated to the responsible project team. The project team should also be given explanation detailing how this information ... and its assessment by senior management ... may influence project continuation, delay, or cancellation.

At some point the project team should develop an outline or strategy describing how any necessary acquisitions will be handled. Key tenants of a sound acquisition strategy are that it appropriately allocates risk between the organization and contractor, effectively uses competition, ties contract payment to accomplishments, and takes maximum advantage of commercial technology.

1.3 Planning Decisions

The purpose of the Planning phase is to put the organization in the best possible position to make decisions about which IM/IT proposals or projects to fund. Getting to this final decision requires that initial decisions be made about whether proposed projects should be moved on for further consideration. It then requires decisions to be made about the relative merits of each individual project. This is followed by the most important decisions, in which tradeoffs are made between the various projects and systems in order to develop the IM/IT investment portfolio that will be funded for the upcoming year.

1.3.1 Determining Whether Projects Met Process Stipulated Requirements

All new projects should have a decision made about whether the project meets all minimal project requirements, at what organizational level the project should be reviewed, and the level of analytical rigor necessary for decisions. While these screening decisions should be relatively straightforward, driven primarily by project-level data sufficiency, they should not be thought of as simply a cursory exercise. The overall efficiency and effectiveness of the entire Planning phase depends to a large extent upon these initial screening decisions.

The organization should also have a process for determining where in the organization a funding decision should be made. The efficiency of the investment management process is significantly affected by how well the organization identifies which projects should be reviewed where. Senior decision makers should not spend their time in lengthy, in-depth reviews of projects that could have been easily assessed and decided upon at lower organizational levels.

1.3.2 Deciding Upon the Mixture of Projects in the Overall IM/IT Investment Portfolio

Decisions made at this stage are the most important of all. The projects that are proposed to make up the investment portfolio for the year should represent the best match with organizational needs and business objectives or, in instances where exceptions were made, an explanation should be provided detailing reasons for the exception.

In making the planning decisions, senior managers should be taking into account tradeoffs between the various projects and systems that are going to be funded. Making these tough choices requires the organization to develop and maintain an understanding that not every project or system can be funded. Spending more for operational systems may mean that there is less money for research and development. The relative merits of each project should be rigorously assessed and analyzed in order to prioritize and select those projects that best match the most critical business needs of the organization.

In addition, projects selected for the portfolio should have decisions made about how often they will be reviewed and how associated risks are going to be managed.

• Previous Page
• Table of Contents
• Next Page