Evaluation Guidebook for Small Agencies

• Previous page
• Table of contents
• Complete text
• Next page

Section Five: Managing Evaluations

5.1  When to Utilize External Resources

This subsection outlines guidelines for determining when to use external resources. Following are the two basic questions to be answered when deciding whether to employ external resources in an evaluation project:

1. Are there staff members with the requisite background and knowledge available within the agency to conduct the evaluation?
2. Is there a need to employ an external consultant to maintain objectivity or the appearance of such?

The first factor relates to the availability of the internal evaluation resources. When an agency lacks specific experience or knowledge, it may wish to contract for that aspect of the evaluation. In such circumstances, it may be useful to set up the contract so that there is some knowledge transfer to the agency. An agency may also lack the resources to do things like telephone surveys. Finally, an agency simply may not have the resources available internally to conduct the evaluation project and will have to resort to employing an external resource.

For the second factor, the agency needs to consider the use and target audience of the report and the risk associated with using an internal versus an external resource. Staff involvement in some evaluation tasks may bias the results (e.g., focus groups, administering a survey on client satisfaction) and for these specific tasks, an agency may wish to employ an external resource. For controversial programs where the report will be used as a public accountability tool, employing external resources may mitigate risks.

Source :  Adapted from Who Should Conduct Your Evaluation? http://www.bja.evaluationwebsite.org/guide/documents/chapter_3_housing.htm

If you answered "yes" to Question 1, but "no" to all other questions, then you will likely need external assistance in conducting your evaluation. If you answered "no" to Question 1 but "yes" to most of the other resource questions, then an in-house evaluation may be a good choice.

5.2  Preparing for Evaluation

In this subsection, two important steps to preparing for an evaluation will be described. They are as follows:

• Establishing Roles and Responsibilities; and
• Preparing the Terms of Reference.

5.2.1  Roles and Responsibilities

Project Coordinator

You will need to decide who the project coordinator will be before proceeding with the evaluation. If the coordinator is to be managing an external consultant, he/she should be familiar with the project, understand the basics of evaluation, and have good project management skills.

Advisory Committee

You may also need to set up an advisory committee comprised of agency and other stakeholder representatives that have an interest in the project or organization being evaluated. It should be noted, however, that setting up an Advisory Committee on an as needed basis may be very resource intensive for a small agency. The need for an Advisory Committee must therefore be weighed very carefully. Where appropriate, a small agency may want to consider existing work teams to act in an advisory capacity.

Steering Committee

A Steering Committee is usually comprised of senior managers and might also include central agency representatives and, where appropriate, regional representatives. When you want the advisory committee to have decision-making powers (i.e., final word on evaluation deliverables), then you may consider setting up a Steering Committee.

5.2.2  Setting the Evaluation's Terms of Reference (TORs)

TORs provide an overview of the evaluation and they make explicit management's initial requirements and expectations for the evaluation. The TORs are a useful tool for senior management in that they can guide the process until the evaluation work plan becomes the primary document.

TORs are used for many purposes.

• To engage senior managers and ensure that the evaluation will address their requirements.
• To help manage the evaluation.
• To secure necessary stakeholder members.
• To develop the evaluation work plan.

To prepare the terms of reference, you may consider the following:

• reasons for the evaluation;
• issues to be addressed;
• resources available for conducting the evaluation;
• anticipated costs;
• expertise required to complete the evaluation; and
• time frame for completion.

The terms of reference provide the basis for the next step in the process – selecting the evaluator or evaluation team. Evaluators are selected through the established contracting process.

Contracting externally for an evaluation team involves the following four steps:

1.  determine the sourcing options;

2.  identify the best value from potential candidates;

3.  notify the successful candidate (posting on MERX for competitive process); and

4.  negotiate and sign the contract.

Bidders often have questions about the Terms of Reference. You should designate one contact person to co-ordinate the responses to these questions. You must ensure that you provide the same information to all proponents and that you do not identify the source of the question. You may want to consider faxing or e-mailing responses to all potential bidders.

Guidelines for Budgeting

Budgeting for evaluation should be part of the upfront planning. It is important to identify a budget amount or range in the TORs. It also allows you to judge which proposals offer the best value per dollar.

Typically, budgeting involves breaking down the evaluation into components or tasks and providing estimates for each component. You may consider costs with respect to evaluation staff, consultants, travel, communications, printing and photocopying, supplies and equipment, and translation and editing.

Leveraging Resources for Evaluation

A small agency can supplement its limited resources by leveraging its needs and resources. While not all strategies might be applicable to your agency, here are some suggestions:

• Group numerous external and internal data collection and analysis exercises as much as possible to avoid redundancy and achieve economies of scale in data collection.
• Search for extra funding from central agency functions desiring coverage of special issues or groups (e.g., Women's Issues or Service Improvement/Government On-Line pilot projects).
• Conduct evaluation work with other agencies involved in the same issues or initiatives.
• Utilize project teams of seconded resources including those from client groups or provincial partners.
• Collaborate with partner agencies to pre-qualify consultants to form a pool of readily available and experienced resources to reduce the administrative burdens of contracting and help ensure quality, useful products.

Preparing the Statement of Work

The statement of work is the main document between the manager and the consultant. The consultant responds to this document by developing a work plan. It outlines the study purpose and objectives, approach, data collection methods, and tasks. If an RMAF or evaluation framework has been prepared, this may be used as a guide when preparing the Statement of Work. As you may recall from the previous section, an RMAF outlines the sources of information and data collection methods for each evaluation question.

One key decision is the level of detail you may want to include in this section. You may choose a less detailed statement of work relying on the proponent to suggest specific approaches and methodologies. However, a detailed statement of work is useful as it gives the proponent a better idea of your expectations. It is also important to keep in mind that the proponent can suggest changes to approach and methods. It is helpful to request that the proponents include a discussion of possible methodological challenges and solutions. This may help you assess the proponent's expertise and may also serve to strengthen the proposed work plan.

5.3  Contracting Options

If you do decide to use external resources you then have to consider the contracting options. You can review the Contracting Policy on the TBS Web site at
http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/contracting/contractingpol_2_e.asp .

These options include the following:

TBS Standing Offer for Small Agency Evaluation Services. (This option is under development at time of printing.)

Proposal calls on MERX – Canada's Official Electronic Tendering Service.

Selective Tendering – The requirement for bids may be set aside when the total costs of the contract (including GST) do not exceed $25,000. In such cases where a fully open competitive process would not really be cost effective, a limited tendering approach may be used. This is done by inviting a minimum of three firms from a source list to bid. This approach is useful for obtaining the best value for money and providing a fair and competitive access to government procurement. Where can you locate sources? Consider the branch's corporate memory and contractor inventory along with the contracting authority's corporate memory, inventory and short lists obtained from other departments and agencies. In this situation, a formal Request For Proposals (RFP) is not required.

Sole Source – used in cases where only one person or firm is capable of performing the job.

Standing Offer – can be used for services of a repetitive nature.

It should be noted that the Government Contracts Regulations contain only four exceptions that permit the contracting authority to set aside the requirement to solicit bids.

Please Note: You may opt to secure the services of Consulting and Audit Canada (CAC). With respect to federal government agencies, there is a Memorandum of Understanding rather than a contract. Federal government contracting rules do not apply when this option is used.

5.4 Choosing Consultants

The terms of reference provides the basis for choosing the consultant. It is important to develop the criteria to be used for selecting a consultant during the planning stages (i.e., before sending out the RFP).

Mandatory and Rated Requirements

The criteria you use to select the proposal should reflect the requirements of the specific project. The RFP typically contains a set of mandatory and rated criteria. To be considered, a proposal must meet the mandatory requirements set out in the RFP. Mandatory requirements set the minimum requirements for the bid to be considered. They are assessed on a simple met/not met basis. These requirements are expressed by using clear and strong terms such as "shall," "must" and "will." When mandatory requirements are used, the RFP shall clearly indicate that failure to meet any of the mandatory criteria will render the bid non-compliant and that it will be given no further consideration.

Proposals that meet the mandatory requirements may go on to be rated against other criteria and may be required to achieve a minimum level for the point-rated requirements to be given further consideration (e.g., score at least 70 points out of 100 to move to the next step in the evaluation process). Each criterion must have associated scoring criteria (the basis for scoring). Contractors can be asked to provide evidence to demonstrate their capacity to respond to a criterion (e.g., résumé).

When you are deciding what rated criteria to include in the RFP, you may want to consider what the most important factors are that will affect the quality of the work to be done (e.g., subject matter expertise, methodological experience, academic qualifications).

Quantitative rating scales are a useful decision-making device. However, these scales have to be constructed and weighted very carefully to ensure you are selecting the right consultant. The rate assigned to each criterion should reflect its importance.

5.4.1  Methods of Selection

The following methods are commonly used to assess proposals. The method to be used should be articulated in the RFP.

1.  The lowest price proposal method. This may be used when cost is the most important factor. This method can be used when mandatory requirements are the main tool for evaluating the proposals and suppliers offer uniform services.

2.  The best technical proposal method. The proposal that receives the highest score within budget is the winner. This method may be used when technical merit is the most important factor. A maximum budget is identified in the RFP. The point-rated requirements are typically used to assess proposals.

3.  The best overall value for money method or cost-per-point method. The proposal that has the best ratio score/price is the winning proposal. It is used when technical merit and price are both important factors .

5.4.2  Criteria to Consider When Reviewing Proposals

When reviewing the proposal, you need to ask yourself the following question: Does the evaluator have a proven ability to deliver results and meet expectations? Other criteria are presented in the checklist below.

5.5  Best Practices for Managing Consultants

Establishing clear expectations at the beginning of a project is crucial to its success. This subsection considers best practices in terms of scope of work, managing the project, and follow-up.

The following checklist provides an overview of the management process from its inception to the reporting of findings.

5.5.1  Defining the Scope of Work

The first planning meeting or initial start-up meeting is critical for clarifying expectations and the evaluation work plan. The project manager should also ensure that the consultant has all the information needed to carry out the project – relevant documents and contact information.

For the start-up meeting you should

ensure that the evaluator(s) have full access to files, reports, publications and any other relevant information;

ensure there is adequate administrative and logistical support during the evaluation;

establish project management and reporting expectations (how often and in what format (written, oral); and

identify the major issues and priorities.

5.5.2  Preparation of the Work Plan

The evaluation work plan or methodology report will likely be the first deliverable produced by the consultant. The work plan should provide a clear description of what the evaluation team is expected to do, as well as where, when, how, and why. The work plan typically builds on the proposal and statement of work. Some changes to the original proposed work plan may be suggested given new information or other considerations.

Here are some key elements of the work plan:

• project background;
• detailed description of proposed methodology;
• specific work schedule; and
• data collection instruments (i.e., interview guides, survey questions).

5.5.3  Overseeing the Day-to-Day Operations with Consultants

The relationship between the consultant and evaluation manager is a partnership in the sense that you bring subject matter expertise and he/she brings evaluation expertise. However, the main role of the evaluation manager is to ensure the consultant follows the agreed upon work plan and provides a satisfactory level of quality. The evaluation manager must also address issues that the consultant might raise when preparing and implementing the work plan. While there is no single good strategy for managing consultants, the checklist below may offer some helpful considerations.

5.5.Managing Follow-ups

In the 3 to 6 months following the completion of the evaluation report, you may develop a better sense of the usefulness of the evaluation. You may want to consider providing follow-up feedback to the evaluator in terms of the usefulness of the evaluation. This information serves to build the evaluator's capacity and benefits the agency by improving the available resource pool.

Key References

Canadian Evaluation Society. Evaluation Methods Sourcebook , 1991.

Financial Management Board Secretariat. NWT. Working Well With Consultants : http://www.gov.nt.ca/FMBS/documents/dox/Consultant%20Guide.pdf .

HRDC. Evaluation Tool Kit. User Guide on Contracting HRDC Evaluation Studies , 1999.

Treasury Board of Canada Secretariat. TBS Contracting Policies , 2003.

UNFPA, Office of Oversight and Evaluation. Planning and Managing an Evaluation , http://www.unfpa.org/monitoring/toolkit/tool5_4.doc .

• Previous page
• Table of contents
• Complete text
• Next page