ARCHIVED - IM/IT Investment Evaluation Guide

• Previous Page
• Table of Contents
• Next Page

This page has been archived.

Tracking and Oversight: Manage the Investments by Monitoring for Results

2.1 Tracking and Oversight Processes

Achieving maximum benefits from a project, while minimizing risks, requires that the project be consistently monitored and managed for successful results. During the Tracking and Oversight phase, agency executives should be actively engaged in monitoring all of the projects in the investment portfolio, making decisions and taking actions to change the course of a project when necessary and incorporating their experiences back in to the Selection phase to further refine and improve the process.

2.1.1 Consistently Monitoring Projects

Each project should be reviewed at key milestones in its life cycle (a project review schedule should have been approved when the initial funding decision was made). The focus of these reviews should expand as projects move from initial design and pilot through full implementation and as the dollar amounts that are expended increase.

A low-cost, small-scale research and development project being conducted to determine the applicability of a systems technology to a business process requirement might receive limited review other than assessing whether the general approach is sound and feasible. However, projects that are preparing for limited field or full-scale implementation should be reviewed in depth including cost and performance to date - to ensure that the project delivers promised benefits within cost and risk limitations and to correct any problems before significant dollars are expended.

In addition, as the reviews are conducted, the context of the program that the system or project supports should be factored in. For instance, a project may exceed performance expectations, but if it is contributing to a program that is failing or is no longer needed, then little is gained for the organization.

The project reviews should assess several aspects of the project and its development. Below are examples of assessment categories that should be considered as part of the project reviews:

1. Deliverables - results achieved to date versus expected results.
2. Methodology - problems that have arisen concerning the systems development process (including contractor management issues).
3. Technical - technical issues or problems, concerning such factors as hardware, software development, or telecommunications.
4. Schedule - estimated time frames versus actual, including schedule slippages and/or compressions.
5. Costs - estimated costs versus costs spent or obligated to date, any changes in funding, and the impact of these changes.
6. Business/program alignment - evaluation of the project's mission improvement effectiveness and relationship to business objectives.
7. Risk - risks that were previously identified are being appropriately mitigated, new risks have been evaluated and steps taken to address them.

The organization should have some standard policies that help ensure that these different categories are assessed uniformly across the organization; however, the measures that are used to evaluate each project will be specific to that project. For instance, the organization may have a requirement that all projects have their schedules reviewed, but the schedules that are reviewed will be different for each project.

The problem with many progress reviews is that they focus almost exclusively on cost and schedule concerns. While these factors are important, the prime focus of progress reviews should be on ensuring that benefits are being accomplished, that risks are being managed, and that the project is still meeting strategic needs. As noted earlier, "review triggers," such as updated benefit/cost ratios or ROI thresholds, done in conjunction with schedule and spending checks, can help the organization determine when actions need to be taken.

The organization should have a documented process detailing how reviews will be conducted, what data and project information is required, and how decisions will be made based on the results of the project reviews. This process should include identifying roles and responsibilities for making decisions, as well as rules for how the project decisions will be made.

Some organizations use a traffic-light method to help make project decisions. Projects are given red, yellow, or green lights depending on how the project rated against expected performance measures. Yellow lights indicate that management action is necessary to avoid potential problems affecting project outcomes. Red lights indicate that major problems have already occurred. (As with all reporting and scoring mechanisms, it is critical that the organization define the conditions associated with element.) The following is an example of a traffic light tracking and oversight process:

The organization should also have an independent audit team, quality assurance group, or independent validation and verification (IV&V) contractor who is responsible for ensuring that project information is valid and verifying that corrective actions have been taken. In addition, the organization should have procedures in place to ensure that information from this quality assurance function is integrated in to the project review process.

Finally, the organization should have mechanisms in place to ensure that project teams are complying with the tracking and oversight process. This may include incentives for raising problems to senior managers and disincentives for noncompliance.

Project reviews, while helping to ensure accountability, should not be totally viewed as a "gotcha" opportunity, in which project managers are punished when problems are identified. Rather, the reviews should be considered opportunities for raising problems early, when they may be easier to address, rather than allowing the problems to be buried, creating a risk that they will arise later when costs are higher and the potential impact is greater.

2.1.2 Involving the Right People

Senior managers (particularly program managers) should be actively involved in the ongoing project reviews and are responsible for making decisions about whether to continue, accelerate, modify, or cancel a project. While members of the development team can, and should, be part of the decision-making process, they should not have unilateral responsibility or authority to make all project decisions. In addition, site executives and project managers should take part in devising and approving the solution to any problems that are identified.

2.1.3 Documenting All Actions and Decisions

All of the information in the business case, including the various analyses that were conducted to justify the project, should be updated to reflect the current state as project implementation continues and dollar amounts increase.

Some leading organizations estimate that often they cannot accurately estimate costs or quantify benefits until almost 40 percent of the way into the project.

The organization should have a uniform mechanism (e.g., management information system) for collecting, automating, and processing data on expected versus actual outcomes. Specifically, this mechanism should: provide the cost and performance data needed to monitor and evaluate investments individually and strategically, provide feedback on the project's adherence to strategic initiatives and plans, and allow for the review of unexpected costs or benefits that resulted from investment decisions.

Data in this system should be easily accessible to both the program team and senior managers.

Collecting and maintaining project information is important, not only from a project review standpoint, but also from the standpoint of establishing an organizational memory. Decisions in all three phases of the investment cycle (Planning, Tracking and Oversight, and Evaluate) will depend on this information being accessible and up-to-date.

2.1.4 Feeding Lessons Learned Back in to the Planning Phase

Information learned during the Tracking and Oversight phase should be fed back in to the Planning phase to help make future selection decisions and to modify and enhance the screening and selection decision criteria. To make this easier, there should be some mechanism in place for aggregating decisions and actions in order to identify patterns of problems or, conversely, patterns of excellence.

Document the warning signs that, with hindsight, preceded the problem, identify what remedial steps were taken, and what the outcome of this approach was. Such documentation will help to make future acquisition decisions and identify recurring problems on existing programs.

2.2 Tracking & Oversight Data

Because the Selection phase usually occurs only once a year during the annual budget process, project information for that phase tends to be collected and assessed on a periodic basis. In contrast, information in the Tracking and Oversight phase is continuously collected, updated, and fed to agency decision makers. The data in the Tracking and Oversight phase should consist of such items as comparisons of actual results achieved to date versus estimates and an assessment of benefits achieved as part of project pilots or prototypes. Data collected during this phase will also consist of ex post documentation such as executive decisions and changes made to projects to address risks or better meet business requirements. The type and depth of data that are collected and maintained in this phase should be commensurate with the type and size of the project.

2.2.1 Measures of Interim Results

As projects move from one phase of the project's life cycle to the next, and as the dollars that are expended increase, interim results should be compared against estimates to ensure that the project is progressing as expected and to indicate when actions should be taken as problems arise or requirements change.

The organization should have project-specific measures established to help analyze actuals versus estimates, ensure that the project is meeting business requirements, and identify where improvements may be needed. These measures will consist of items such as cost and schedule information, quantitative and qualitative benefit measures, status of deliverables, risk elements, etc. The measures should be updated as actual costs, risks, and benefits are identified.

Using these measures, the organization should identify and monitor interim results that are achieved. The following are examples of the kinds of data that should be analyzed:

Accumulation of actual cost data and comparisons to estimated cost levels.

Evidence that results for the phase (or results to date) have been compared against initial estimates for cost, schedule, performance, risk, and return. Documentation of the change between the current number and scope of requirements and the original requirements baseline established for the project. Documentation of the comparison between the current business conditions and assumptions and the projects' initial assumptions and context. After the release of each new increment, each project participating in the increment should be analyzed to determine what interim benefits have been achieved in comparison to the previous increment.

Documentation of differences between the actual performance of the software organization or contractor and their claims at the beginning of the project (e.g., schedule, costs, functionality, technical solutions, etc.).

Aggregate data covering costs, benefits, and project performance for all IM/IT projects in the investment portfolio.

2.2.2 Updated Analyses of Each Project's Costs, Benefits, Schedule, and Risks

The cost, benefit, schedule, and risk information that was included in the business case, including the various analyses that were conducted to justify the project, should be updated as project implementation continues and as dollar amounts increase. For instance, it may have been difficult to precisely estimate costs and benefits when the project was first proposed, but such quantification may be improved as prototype and pilot project results become available.

Information and analyses in the business case should also be updated to provide justification for adding additional functional requirements to the project. This justification should weigh the costs of adding the requirement late in the development process versus the anticipated benefits that are expected from the added functionality.

Older versions of these analyses should be maintained for later comparisons and to feed lessons learned back in to the Selection phase.

2.3 Tracking and Oversight Decisions

The primary focus of the Tracking and Oversight phase should be on making project management decisions. Actions should be taken quickly to address problems as they are identified and senior managers should be actively involved in making decisions about all of the projects in the investment portfolio. While many of these decisions will be implicit (the project is right on course, no problems have been identified, requirements have remained the same and, thus, the decision will usually be to continue the project as is), it is critical, nonetheless, that a conscious decision be made about the future of each project.

2.3.1 Deciding Whether to Cancel, Modify, Continue, or Accelerate a Project

As each project is reviewed at various stages in its life-cycle, decisions should be made about the future of the project. These decisions will be unique for each particular project and should be based on the particular merits of the project. In addition, some explanation or documentation of the decision should be included. Even implicit decisions should have some documentation to show that a conscious decision was made to continue the project.

Projects that have deficiencies or problems identified (actuals exceed estimated levels, risks are increasing, requirements have changed, etc.) should have a decision made by senior managers about what to do with the project. Decisions will usually involve one of four alternatives: modify the project cancel the project continue the project as is accelerate the project's development

Decisions may also be made to suspend funding or make future funding releases conditional on corrective actions being taken. These decisions should be documented, along with an explanation or criteria stating how funding can be reobtained. The decisions should also be reflected in budget information. For instance, if a project's development is halted while the feasibility of an alternative is assessed, budgetary spending information should reflect such a halt in funding. There should also be an explanation or documented criteria stating what must occur before funding is reinstated.

In addition, depending on what decisions are made about a project, future "cascading" actions resulting from these decisions should be clearly identified and delineated. For instance, halting the development of a project will impact a number of other areas, including project management, personnel and staffing decisions, budget decisions and spending priorities, etc. These cascading actions should also be reviewed and monitored to ensure that money is not continuing to be spent and that all development activities have ceased.

An independent review should be conducted prior to funding being reinstated to ensure that all corrective actions have been taken and to determine whether additional changes or modifications are still needed.

2.3.2 Aggregating Data and Reviewing Collective Actions Taken to Date

A review of past activities and decisions made about a particular project can influence both current and future managerial decisions. This is a primary reason why aggregating information is important. Aggregating allows trends to be more easily identified. Looking at projects across an agency or bureau, for example, can help pinpoint those divisions that have had repeated success at developing and managing IM/IT projects, and those that have had more trouble. This in turn can be used as inputs for decision makers when weighing organizational capability risks and determining project review schedules.

Aggregating can also help as the organization looks to refine and improve the screening and selection criteria and performance measures. Data can be aggregated by project, or can be grouped along unit, divisional, bureau, or agency lines.

Problems that are identified from this analysis may be serve as an indication of specific endemic weaknesses with project management, contractor oversight, or cost-estimation practices that need revision and corrective actions. In addition, positive trends that are identified can provide valuable lessons for highlighting and reinforcing organizational strengths.

• Previous Page
• Table of Contents
• Next Page