Evaluation Guidebook for Small Agencies

Evaluation Guidebook
for Small Agencies

Acknowledgements

Section One: Introduction

Section Two: Defining Evaluation

Section Three: Developing an Evaluation Function

Section Four: Evaluation Cycle

Section Five: Managing Evaluations

Section Six: Using Evaluations

Appendix A―Types of Small Agencies
Appendix B―Horizontal Initiatives
Appendix C―Seeking External Advice and Support 
Appendix D ―Expenditure Review Committee's 7 Tests
Appendix E―"How to" Information for Planning and Conducting Evaluations
Appendix F―Terms of Reference Template
Appendix G―Glossary 
Appendix H―Evaluation Web Sites

Section One: Introduction

This guidebook is a reference tool designed to assist small agencies to develop their capacity to plan, implement and manage evaluations. The guidebook will help small agencies meet the requirements of the Treasury Board of Canada (TB) Evaluation Policy .

This "how-to" guidebook presents effective practices for planning and managing evaluation projects, provides strategies for fostering the use of information from evaluations in strategic decision making within small agencies, and provides examples of key documents used in the evaluation process.

This guidebook builds on the work that was previously conducted in 2003–04 with respect to developing appropriate models of evaluation and performance measurement in small agencies. [1] The guidebook represents the next step in assisting small agencies to build their capacity to plan, implement, and manage evaluations.

1.1  Purpose of the Guidebook

This guidebook should help you with the following:

• to understand the relevance and role of evaluation in supporting effective control and performance measurement regimes within your agency;
• to plan, design, implement, and manage an evaluation appropriate for your agency; and
• to communicate and foster the use of evaluation results within your agency.

1.2  Who Should Use This Guidebook?

This guidebook is designed for small agencies that require assistance in implementing the Treasury Board of Canada (TB) Evaluation Policy requirements. The primary target audience is the personnel who are responsible for evaluation within small agencies. For those readers who have limited experience in the field of evaluation, the guidebook is designed to provide detailed information on most aspects of evaluation and includes helpful material such as context, definitions, checklists, management tips and references for more detailed information on certain subjects. For those readers who have more experience with evaluations, they may find various parts or sections of the guidebook more useful than others, such as the checklists. It is also recognized that small agencies' needs are varied with respect to evaluation and information needs. The guidebook attempts to address these diverse needs.

1.3  Small Agency Context

Most small agencies were created by the federal government to independently carry out specific mandates. For example, some agencies render impartial decisions (e.g., tribunals). Other agencies perform a facilitation or intermediary role. Others have regulatory and inspection mandates. See Appendix A for an overview of different types of small agencies.

With respect to evaluation, previous studies [2] have noted that many small agencies face challenges in developing evaluation and performance measurement functions for their agencies. These challenges include having adequate resources and capacity to develop these functions, having to adapt models from larger organizations, and integrating the function with the day-to-day business of the agencies.

Small agencies as a group also have unique challenges and characteristics when compared with medium and large federal departments. One difference is that small agencies often have one or two main business lines in comparison with medium and large departments that often have multiple business lines containing many programs, initiatives, and policies. Another difference is that, in comparison with medium and large departments, small agencies often have more limited flexibility with respect to financial resources. Given these and other differences, many aspects of the models in meeting accountability and performance requirements in medium and large departments are not applicable to small agencies.

When considering what type of evaluation and performance measurement models are required in the small agency community, it is important to realize that, although the community shares commonalities when compared with medium and large departments, there is also a great amount of diversity within the small agency community itself. The small agency community in the federal government is diverse on various dimensions such as organizational structure, relationship with larger departments, nature of work, and organization size. These dimensions, in addition to others, contribute to the type and nature of information that agencies need for decision making and ensuring accountability within their organizations.

A 2003 Treasury Board of Canada Secretariat (TBS) project entitled Models for Evaluation and Performance Measurement for Small Agencies [3] presented three models of evaluation and performance measurement functions within federal small agencies. Two key considerations in developing the models were the types of management decision making and the types of information needed to make decisions within agencies.

These models are as follows:

4  Model A – Straightforward Information Needs

4  Model B – Blend of Straightforward and Complex Information Needs

4  Model C – Complex Information Needs

For a more detailed description of the models and how to classify your own agency, please go to the TBS Web site at http://www.tbs-sct.gc.ca/cee/tools-outils/model-eng.asp .

1.4  Structure of the Guidebook

• Section 1 contains this introduction.
• Section 2 presents an overview of evaluation, relevant concepts and context.
• Section 3 describes how to establish an evaluation function and provides a brief review of evaluation policy requirements. It also presents strategies for building internal evaluation capacity.
• Section 4 presents an overview of preparing for and conducting evaluations. Topics include results-based management and accountability frameworks, evaluation design, data collection methods, analysis, and report writing.
• Section 5 contains an overview of managing external resources. This section includes topics such as preparing terms of reference, contracting options, methods for selecting contractors, and best practices for managing evaluation consultants.
• Section 6 provides an overview of strategies for communicating and using evaluation findings within your organization.

In addition, there are a number of appendices that contain more detailed information that can be used as references.

• Appendix A contains a description of the different types of small agencies.
• Appendix B contains information on horizontal initiatives and their evaluation.
• Appendix C contains information on seeking external advice and support on evaluation.
• Appendix D contains the Expenditure Review Committee's seven tests.
• Appendix E contains more detailed "how-to" information for planning and conducting evaluations.
• Appendix F contains a terms of reference template.
• Appendix G contains a glossary of relevant evaluation terms.
• Appendix H contains links to evaluation Web sites.

1.5  How to Use this Guidebook

The guidebook is designed to be used by readers in a number of different ways. For those readers who are relatively new to the field of evaluation, you may want to work your way systematically through the guide. For more experienced readers, it may be more useful to go directly to the specific section that you need. Regardless of how you choose to use the guide, you should be aware of the symbols used throughout the text as indicated below.

If you require a more detailed guide with respect to planning or implementing an evaluation, please refer to Appendix E.

Key References

Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies: Summary Report , 2003.

Treasury Board of Canada Secretariat. Independence vs. Partnering – Finding the Right Balance: A Dialogue on Values and Ethical Decision-Making in Small Agencies , 2003.

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Section Two: Defining Evaluation

2.1  What is Evaluation?

Note that evaluation

• is periodic (has a "cycle");
• can cover policies, programs or initiatives within a small agency;
• involves judgment about a policy, program or initiative's merit or worth (based on systematic and high quality data);
• focusses on how and why results are achieved;
• looks at intended and unintended effects; and
• attempts to address future options and strategies for improvement.

2.2  Why Evaluation?

Following the release of the document Results for Canadians: A Management Framework for the Government of Canada in April 2001, TB released an Evaluation Policy "to ensure that the government has timely, strategically focussed, objective and evidence-based information on the performance of its policies, programs, and initiatives to produce better results for Canadians."

1.  Improvement: to help managers design or improve policies, programs, and initiatives; and

2.  Accountability: to provide, where appropriate, periodic assessments of policy, program, or initiative effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.

Within the small agency context, both improvement and accountability objectives are relevant. Evaluations can assist small agencies

• to become more focussed on results in their decision making;
• by providing valuable performance information for Departmental Performance Reports and as required by the Management Resources and Results Structure (e.g., a program's relevance and achievements with respect to strategic objectives); and
• by satisfying elements of the Management and Accountability Framework (e.g., results and performance, accountability, risk management, policy and programs).

2.3  Evaluation and the Program Development Cycle

Ideally, planning and evaluation are interdependent processes. Evaluation can and should be built into the planning process for agencies. Combined with the feedback from performance measurement activities, information from evaluations can help to guide planning within the agency.

2.4  Overview of Tasks in Conducting Evaluations

The illustration below provides an overview of the tasks required to conduct an evaluation. This guide examines these tasks in more detail in Sections 4, 5, and 6.

2.5  Types of Evaluation

There are two main types of evaluations: formative and summative.

Another type of evaluation often referred to within the federal context is a Horizontal Evaluation . These are evaluations of initiatives that involve the co-ordinated activities of several federal departments or agencies. See Appendix B for more details.

2.6  Other Accountability Activities

As previously mentioned, evaluation is one type of activity or tool that can be used by managers to demonstrate accountability. This section contains brief descriptions of other tools (e.g., performance measurement, audit), a discussion of how to determine when each is most appropriate, and how they can be used in conjunction with one another.

2.6.1  Performance Measurement

Performance measurement is the regular collection of information for monitoring how a policy, program, or initiative is doing at any point in time. It generally focuses on providing operational performance information to program managers. For many small agencies, one of the first tasks in developing an evaluation function is to develop a performance measurement strategy that encompasses periodic evaluation.

Evaluation and performance measurement are two complementary activities. Evaluation provides accountability by ensuring performance measurement is on track and indicators are appropriate. In turn, performance measurement data are often important data sources for periodic evaluations.

2.6.2  Internal Audit

Internal audit is a function that provides assurances on a department or agency's risk management strategy, management control framework and information, both financial and non-financial, used for decision making and reporting.

The internal audit function does the following:

• assesses the adequacy of internal control;
• verifies compliance with established rules, regulations or procedures;
• assesses the risk of each of the above;
• systematically reviews operations to ensure conformance with policies, strategies and plans;
• systematically reviews management practices and controls;
• reviews information for validity (e.g., financial, operational and management information); and
• defines policies, projects, initiatives that are managed by an entity (audit domain).

2.6.3  Review

Reviews are often conducted in response to a pressing or immediate need of management. As such, the emphasis is usually on quick generation of sufficient information to inform decision making or reassure senior management of the dimensions of a problem or situation. The methodology used to gather information is usually secondary to developing an adequate answer in a timely fashion (i.e., evaluation or audit protocols and approaches are not adhered to). Although they are useful to address targeted issues, reviews or special studies do not conform to external reporting requirements, project control processes, or standards which delineate a discipline such as audit or evaluation.

2.7  Evaluation in Context

Evaluation has a long history in the federal government. More recently, evaluation is perceived as a key tool for public service managers as they address the requirements of recent approaches to management in the public sector. Some of these are briefly described below to situate evaluation in its current context. More detailed information on each of these approaches or initiatives is available directly from TBS. [5] All of the initiatives described below will continue to have an impact on managers in small agencies as they develop capacity in areas such as evaluation, performance measurement, and internal audit.

2.7.1  Results-based Management

Results-based management is a comprehensive approach to management aimed at improving performance through achieving better results.

Shortly after the federal government introduced Results for Canadians in 2000, the Results-based Management and Accountability Framework (RMAF) was introduced. The RMAF is a tool used to plan, monitor, evaluate, and report on the results of a program. The RMAF integrates the evaluation function within the context of results-based management. It is also a link to the Management Resources and Results Structure (see below). RMAFs will be discussed in more detail in Section 4 of this guidebook.

2.7.2  Management Accountability Framework (MAF)

The MAF defines and clarifies management expectations. It is a set of ten statements that summarizes TBS's expectations for modern public service management.

The MAF aims to

• improve management practices and stewardship of resources across government;
• align management expectations to the vision of Results for Canadians ; and
• represent management as a broader integrative function.

The MAF will be used in the following ways:

• as a basis of dialogue between TBS and departments or agencies;
• as an assessment tool of organizational health;
• as input for assessing deputy minister performance; and,
• for framing future reporting on management.

A graphical representation of the Management Accountability Framework follows.

Management Accountability Framework: Ten Essential Elements of Modern Management

2.7.3  Management Resources and Results Structure (MRRS)

The MRRS replaces the Planning, Reporting and Accountability Structure (PRAS) policy. In accordance with MAF expectations, MRRS supports governance and strategic direction, accountabilities and results and performance. The new policy is directed to the organizational level and encourages the alignment of programs, resources, and management practices with expected results.

The Program Activity Architecture (PAA) is an element of the MRRS and comprises

• clearly defined and appropriate Strategic Outcomes; and
• a complete program inventory that links all agency programs and program activities so that they roll up to these strategic outcomes.

Over time, an integrated MRRS should also include

• performance measures for each level of the agency's architecture; and
• a governance structure that defines decision making and accountability by outcome and by program.

The Expenditure Management Information System (EMIS) will provide a common framework that aligns information on priorities, plans, actual expenditures, and results.

2.7.4  Where Does Evaluation Fit In?

With its focus on the "how" and "why" of results, evaluation is a key tool in managing for results. As a part of management practices, evaluation can help to design and improve programs. Evaluation should be used in conjunction with other management tools to improve accountability and decision making.

Evaluations are a critical tool for demonstrating results and performance and as such help to support the other elements contained in the MAF. Moreover, through the inclusion of results and performance as a key management expectation, the MAF makes an explicit commitment to evaluation.

Evaluation can provide timely information on impacts and relevance with respect to strategic objectives of an agency. Hence, evaluation will address some of the information needs as required by MRRS. Annual evaluation plans (organizational level) and RMAFs (program level) provide links to the MRRS.

The graphic below places evaluation within the context of other government initiatives.

Evaluation in Context

Key References

Canadian Evaluation Society. Project In Support of Advocacy and Professional Development: Evaluation Benefits, Outputs, and Knowledge Elements , September 16, 2002.

Goss Gilroy Inc. Review of Approaches in Other Jurisdictions. Lessons Learned from International Experience with Performance Measurement/Evaluation in Small Agencies , 2003.

National Science Foundation. User-Friendly Handbook for Project Evaluation.

Treasury Board Canada Secretariat Web site RBM E-Learning Tool .

Treasury Board Canada Secretariat Web site. Management Accountability Framework

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Treasury Board of Canada Secretariat. Preparing and Using Results-based Management and Accountability Frameworks , April 2004.

Treasury Board of Canada Secretariat. Results for Canadians , 2000.

Section Three: Developing an Evaluation Function

3.1  Challenges in Developing an Evaluation Function

As mentioned previously, many small agencies are currently facing or have faced considerable challenges in developing evaluation and performance measurement capacity in their organizations. Despite these challenges, some small agencies have made considerable progress with respect to implementing evaluation and performance activities in their organizations in a meaningful way. In these agencies, information and findings from performance measurement and evaluations is valued and is actively used to support decision making and planning within the organization.

Some factors that have been found to be associated with enhanced evaluation capacity in smaller organizations include the following:

• Agencies that have a regulatory or research mandate may be inherently more evaluative in nature because they are used to the analytic and evaluative process required for performance measurement and evaluation.
• There is a commitment to evaluation and performance measurement activities by the political and senior management levels of the organization.
• Managers have a good understanding of the role of evaluation in the management cycle.
• The existing culture promotes the use of information for decision making.
• There is an internal group that has the capacity to market performance measurement and evaluation services within the organization and to make the effort to communicate evaluation results to managers and external stakeholders.
• There is an identifiable "champion" in the agency who understands and works consistently at explaining the benefits of performance measurement and evaluation to other members of the organization.
• There is senior management support to produce the overall cultural shifts required in an organization as it integrates the concepts and process of performance measurement and evaluation within the day-to-day activities of the organization. [6]

Small agencies face many challenges with respect to enhancing evaluation capacity within their organization. Examples include the following:

• The political appointee (Head of Agency) often does not have extensive experience within the Public Service. One possible outcome of this is that there is a lack of support or understanding from the Head of the Agency with regard to issues of performance reporting and evaluation within a public service context.
• Resource limitations of small agencies are notable. Consequently, there is often little flexibility in the allocation of resources for the development of new internal processes that are not directly part of the agency's mandate.
• With regard to human resources considerations, there is difficulty in attracting internal capacity, even where positions exist.
• The workload may not justify the need for a full-time evaluation function.

As illustrated, the small agency environment creates unique circumstances and challenges for building capacity in evaluation and performance measurement. Suggestions for building capacity are outlined in later subsections.

3.2  Checklists for Implementing the Evaluation Policy

In this section, there are a number of checklists available that will assist the reader in becoming familiar with various sections of the TB Evaluation Policy . For readers already familiar with the policy, it may be a good exercise to go through the various lists to identify potential gaps or areas covered by your agency in implementing the policy.

*  For small agencies a departmental evaluation committee or a combined audit and evaluation committee may serve the same role. In the smaller agencies (say less than 50 FTEs) the Evaluation Committee, Senior Departmental or Management Committee is often comprised of the same people.

3.2.1  Establishing an Agency Evaluation Plan

The Evaluation Plan is strategically focussed and designed to balance the evaluation workload between meeting internal agency information needs and serving external reporting against federal government priorities. It should be developed or reviewed annually, although it may span several years. Many small agencies may conduct one or possibly two evaluations every five years. As a result, the annual evaluation plan is more likely to be reviewed rather than developed each year. During the review, the plan should be updated and modified to address any changes or shifts that have occurred within the previous twelve months.

The evaluation plan incorporates measures against the MRRS strategic outcomes, covers the program inventory, utilizes the ongoing performance measures, and feeds the products into the governance structure to support the MRRS decision-making needs.

See the TBS Web site at http://www.tbs-sct.gc.ca/cee/pubs/guide/dep-epe-eng.asp for a detailed guide to good practices for developing and tabling annual evaluation plans. The following checklist outlines an abbreviated version of these emerging or best practices. Please note that not all of the points are required elements. (See previous checklist for required elements of the Evaluation Policy .)

*  Evaluation Policy requires evaluation reports to be made public.

3.3  Building Internal Evaluation Capacity

The concept of capacity building is similar to the concepts of organizational development, organizational effectiveness and/or organizational performance.

Capacity building involves a variety of activities such as the following:

• addressing gaps in infrastructure by identifying and providing tools and training;
• providing incentives to recognize and reinforce new behaviours;
• focussing on people and ensuring that strategies are in place during the transition period;
• supporting collaboration with other agencies and partners; and
• other methods of organizational performance management including the balanced scorecard approach, principles of organizational change, cultural change, and organizational learning.

3.3.1  Steps for Building Evaluation Capacity

Building capacity within an organization usually follows three basic steps.

Step One – Identify Gaps

Identify gaps and issues with respect to evaluation capacity.

a.  Conduct inventory of existing data sources and information (i.e., operational, financial, administrative, strategic management, and accountability data).

b.  Undertake assessment with managers to identify current information needs.

c.  Identify information gaps and needs.

Step Two – Develop Strategies

Identify the changes or strategies needed to address these gaps.

d.  Consider strategies related to organizational change, human resources, infrastructure, networking. (See examples and suggested strategies below.)

Step Three – Action Plan for Change

e.  Develop an action plan for change and assign roles and responsibilities.

3.3.2  Strategies for Building Evaluation Capacity

Some examples of how small agencies have developed performance measurement and evaluation capacity within their agency are presented below.

Some approaches to enhancing capacity that were suggested during the exercise to develop models for performance measurement and evaluation in small agencies include the following:

• building commitment and supporting cultural change within an organization;
• identifying an evaluation "champion;"
• providing relevant training and staff development (e.g., mentoring);
• working with a central agency or larger department to identify lessons learned that can be adapted to a small agency setting;
• identifying the importance of evaluation planning, clarifying objectives of the evaluation function, and linking it to other functions within the agency;
• seeking external advice and support (e.g., TBS, Centre of Excellence, Small Agency Administrator's Network, Canadian Evaluation Society); [7]
• identifying and/or leveraging additional resources from external and internal sources; and
• developing or enhancing leadership training within the agency so there is a clear understanding of the role of evaluation within a result-based management environment.
  

General project management skills are required for the effective planning, implementation, and management of evaluations. In this manner, evaluation projects do not differ substantially from many other projects that public service managers will plan and implement in a results-based environment. The checklist below is a generic guide to be used for those who wish to develop and monitor capacity in the area of project management.

Key References

Community of Small Organizations Web site: .

London School of Economics and Political Science. Juggling On a Unicycle: A Short Guide to Organizing A Small Agency , 1999.

Performance and Planning Exchange. Making Change Personal: Why we don't know what we know , November 2003.

Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies. Summary Report , 2003.

Treasury Board of Canada Secretariat. Changing Management Culture: Models and Strategies to Make It Happen , 2003.

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Treasury Board of Canada Secretariat. Modern Comptrollership Practices: Toward Management Excellence , 2003.

Treasury Board of Canada Secretariat. Review of Approaches in Other Jurisdictions. Lessons Learned from International Experience with Performance Measurement/Evaluation in Small Agencies , 2003.

Section Four: Evaluation Cycle

Overview of Evaluation Life Cycle

This section provides an overview of evaluation from preparing an RMAF to collecting the information and writing the report. The graphic below illustrates the evaluation cycle.

The focus of the section will be on the first three steps: (1) planning; (2) implementing; and (3) reporting. More specifically, this section is presented as follows:

Drafting the plan at the program level (or RMAF)

• describing the project
• preparing a logic model
• preparing the performance measurement strategy
• developing an evaluation strategy

Carrying out the evaluation

• collecting information
• analyzing information

Writing the report

The fourth step in the overall evaluation life cycle, using the evaluation results, will be discussed in Section 6.

4.1  Planning the Evaluation

In order to ensure that the evaluation will be a useful product, the details need to be worked out early in the evaluation life cycle. In the beginning it is important to establish a basic understanding of why the evaluation is to be carried out. Below is a checklist for planning your evaluation at the program level.

4.2  RMAF: Tool for Planning

Since the Results-based Management and Accountability Framework (RMAF) is a common tool in the federal government for evaluation planning, this subsection will describe the RMAF and its components. This section will focus on the profile, logic model and the evaluation strategy but will only provide a high level description of the ongoing performance measurement strategy.

While the previous section dealt with the overall agency evaluation plan, this subsection focusses on planning an evaluation for a specific program . The evaluation framework component of the RMAF should, however, link to the Agency Evaluation Plan.

RMAFs are a management tool, somewhat like a guidepost or compass for an organization, program, policy or initiative.

4.2.1  Frequently Asked Questions on RMAFs

An RMAF should be concise and focussed. This will help to support its implementation.

4.2.2  Profiling the Program

A clear understanding of the organization and the program is needed to guide monitoring and evaluation activities. The profile typically includes a summary of the context, objectives, key stakeholders and beneficiaries, organization and governance structures, and resources. The profile should provide a clear understanding of what the program aims to achieve and how.

*  In the past, profiles have typically included planned results and the delivery strategy. These components may now be included in the Planned Results and Program Theory (Logic Model) section.

4.2.3  Developing the Logic Model

What is a Logic Model? A logic model is a diagram or picture that shows the causal links from the activities to the results. Logic models illustrate the cause-effect relationship between activities and outputs through to the final results. It is a visual way of expressing the rationale, thought process or theory behind an organization, program or initiative. It is a representation of how the organization or initiative is expected to lead to the results. A logic model can be applied to an organization, policy, program or initiative. It can be used for the purposes of planning, project management, evaluation and communication. Logic models can help clarify objectives and focus the evaluation on results

Components of a Logic Model

While logic models can vary considerably in terms of how they look, they typically have three main components – activities, outputs, and results.

Some logic models also include other features, such as:

• Reach – To which target groups/clients are the activities directed?
• Inputs – What resources are used?
• Internal/External Factors – The identification of factors within and outside control or influence.

TYPE 1: Flow Chart or Classic Logic Model

The flow chart or classic logic model illustrates the sequence of results that flow (or result) from activities and outputs. It is a very flexible logic model as long as the three core components of the logic model are presented: activities, outputs, and results. You can have any number of result levels to ensure that your logic model accurately depicts the sequence of outcome results.

The cause-effect linkages can be explained by using "if-then" statements. For example, if the activity is implemented, then these outputs will be produced. If the immediate result is achieved, then this leads to the intermediate result, and so on.

The flow chart logic model makes you think carefully about the linkages between specific activities, outputs and outcomes. What outputs result from each activity? What outcome resulted from the output?

This type of model is also referred to as a performance chain. While it is similar to the flow chart model, it does not isolate the specific activities, outputs or results. The results chain, therefore, does not show the same detail with respect to the causal sequence of outputs and results.

Both types of logic models, however, are used as a structure for describing the expectations of a program and as a basis for reporting on performance. Like the flow chart model it is based on the rationale or theory of the program.

Source :  Six Easy Steps to Managing For Results: A Guide for Managers , April 2003, Evaluation Division, Department of Foreign Affairs and International Trade.

Other Considerations

• The results chain is less time-consuming to develop.
• The flow chart logic model enhances understanding of how specific activities might lead to results.
• You may develop one, two, or three result levels, depending on the relevance to your program or organization.

4.2.4  Developing the Performance Measurement Monitoring Plan for an RMAF

While this is an evaluation guidebook, RMAFs also include a monitoring plan. This strategy should generate a timely flow of information to support decision making on an ongoing basis. It is important to note that some data required for evaluation purposes can be collected on an ongoing basis as part of the performance measurement system.

The indicators for the performance measurement strategy are developed from the logic model's outputs and results. For each indicator, the data source, collection method, timing and frequency of the data collection, and responsibility for measurement must be identified.

4.2.5  Developing the Evaluation Strategy

Overview of an Evaluation Strategy

• The evaluation strategy includes the following components:
• evaluation issues and questions;
• corresponding indicators;
• sources of data (including performance measurement reports);
• data collection method;
• timing; and
• estimated costs for evaluation activities.

The evaluation strategy may be presented in matrix format similar to the example below.

Articulating the Strategy Efficiently and Effectively to Meet the Agency's Needs

An evaluation strategy should balance the need for timely and credible information with the need for practicality. Good linkages between evaluation and performance measurement will help to ensure that performance information is used as a source of information for evaluations. Linkages between planning and evaluation will help to ensure that the evaluation strategy is appropriately focussed and directed towards information needs.

Some things to think about when developing an evaluation strategy:

• Consider a balanced, mixed methods approach to evaluation design. This helps to strengthen the evaluation design and enhance the credibility of the findings.
• Place appropriate emphasis on information needs (i.e., process issues) and practicality to guide your development of an evaluation strategy.
• Target evaluation questions to the most pressing evaluation concerns.
• Only collect information relevant to those questions.
• Where practicable, consider strategies for integrating information from performance measurement and evaluation activities with other management information.
• Where applicable, consider using existing data as a possible source of information for the evaluation.

Developing Evaluation Issues and Questions

Evaluation issues are the broad areas which need to be explored within an evaluation while evaluation questions are the more specific research questions that need to be answered in order to be able to address each evaluation issue.

The identification of the evaluation issues and questions provides a guide for the development of the strategy that ensures all essential issues will be addressed during later evaluation. The issues are used to elaborate a set of indicators and data collection strategies, which, on implementation, helps to ensure that information necessary for evaluation is available when it is needed. As such, the evaluation strategy needs to be linked to the ongoing performance measurement strategy, as some evaluation data will be collected through ongoing performance measurement activities.

For information on how to develop a specific list of issues and questions, see Appendix E. For the main evaluation issue areas please refer to the following diagram.

Determining Appropriate Indicators

An example of various indicators is illustrated in the table below. See Appendix E for a review of how to develop indicators for your agency.

Source :  Adapted from First Nation Self-Evaluation of Community Projects: A Guidebook on Performance Measurement .

Overview of Evaluation Designs

Evaluation design is the process of thinking about what you want to do and how you want to go about doing it. 

The most practical approach to determining evaluation design is to consider your information needs (i.e., evaluation questions) and use this to guide your design. Key considerations for selecting an appropriate design are feasibility and practicality .

Evaluation designs are typically placed into the following three categories:

• Experimental designs involving comparisons of clients and a control group (these are randomly assigned and rarely used in federal evaluations);
• Quasi-experimental designs involving comparisons of clients and the control group, but do not use randomization; and,
• Implicit designs involving measuring the effects of a program after it has been implemented. Control or comparison groups are not used.

Implicit designs are the most frequently used evaluation design. In the public service context, it is often the only design that can be used, when no pre-program measures exist and there is no obvious control group available, or it is not reasonable to assign interventions on a random basis. This design type is flexible and practical to implement.

It should be noted, however, that there are considerably more challenges in attributing impacts to specific interventions as we move away from experimental designs (the strongest for attributing impacts) to implicit designs.

Overview of Data Collection Methods

The table below provides an overview of various data collection methods available to evaluators. Note that these data collection methods involve either primary or secondary data. The investigator collects primary data directly. Secondary data have been collected and recorded by another person or organization, sometimes for different purposes.

In choosing appropriate data collection methods you can consider the following:

• information and decision-making needs;
• appropriate uses, pros and cons of the data collection methods;
• costs and practicality of each method; and
• balanced approach, a mix of quantitative and qualitative methods.

More information on choosing appropriate data collection methods is located in Appendix E.

4.3  Collecting the Information

4.3.1  Gathering Data

Data collection should follow the plans developed in the previous step. The individuals assigned to the various data collection tasks need to be thoroughly trained in the data collection requirements and procedure.

Appropriate quality control procedures should be implemented and maintained during the evaluation study. If you are managing an evaluation you need to be aware of the progress of the data collection and any other issues of concern.

To facilitate analysis, information collected during an evaluation should

4  use appropriate methods to organize and record the information collected (e.g., frequency distributions, categories, tables); and

4  implement effective quality control procedures to ensure recorded information is accurate and original information is labelled and secure.

4.3.2  Analyzing Data

Once data are collected, they need to be analyzed and interpreted. Data analysis may take many forms from basic description to complex statistical analysis depending on the type of data and the complexity of the issues. For more detail as to how to analyze data, please refer to Appendix E.

Cause and Effect Inferences

The choice of analysis techniques is influenced by the evaluation questions and the evaluation design (i.e., experimental or implicit). For example, drawing inferences about causality is dependent upon the evaluation design rather than the analysis technique.

Generalizing the Findings

The only valid way of generalizing findings to an entire or target population--where you cannot survey or study everyone--is to use findings from a statistically representative random sample of the population you wish to study. Caution must therefore be exercised when analyzing data from non-randomized samples.

Qualitative and Quantitative Analysis

Analyzing qualitative data requires effective synthesis and interpretative skills. Qualitative information can be used, for example, to provide contextual information, explain how a program works, and to identify barriers to implementation. Qualitative data can be analyzed for patterns and themes that may be relevant to the evaluation questions. Qualitative material can be organized using categories and/or tables making it easier to find patterns, discrepancies, and themes.

Quantitative data analysis assigns numerical values to information. It can range from simple descriptive statistics (e.g., frequency, range, percentile, mean or average) to more complicated statistical analysis (e.g., t-test, analysis of variance). Computer software packages such as Statistical Package for Social Sciences (SPSS), Minitab, and Mystat can be used for more complicated analysis. Quantitative data analysis also requires interpretation skills. Quantitative findings should be considered within the context of the program.

*  In advance of gathering data, the evaluator needs to calculate the probability that the findings are not "accidental." With tests of significance an evaluator can decide how strong the results must be in order to be reasonably confident that the results are not due to chance.

4.4  Writing the Evaluation Report

A good evaluation report responds effectively to the evaluation questions. Recommendations and lessons should be conclusive, concise, and practical. The executive summary should be a summary of the overall report. Often the executive summary is the most widely read section of the report so it should be detailed enough to give the reader a good sense of the highlights of the evaluation.

4.4.1  Table of Contents

An evaluation report typically contains the following sections:

• Executive Summary;
• Introduction and Background;
• Scope and Objectives of Evaluation;
• Approach and Methodology;
• Findings;
• Conclusions; and
• Recommendations.

4.4.2  Linking Findings, Conclusions, and Recommendations

There needs to be a clear link between findings, analysis, conclusions and recommendations. Practically speaking, the findings may not answer specific evaluation questions conclusively. Conclusions are formulated by combining the best evidence. Gathering different types of evidence relating to the same evaluation question can enhance credibility. Recommendations should link to the analysis and the conclusions.

Key References

CIDA, Performance Review Branch. CIDA Evaluation Guide , 2004.

Canadian Evaluation Society. Evaluation Methods Sourcebook , 1991.

National Science Foundation. A User-friendly Guide to Mixed Method Evaluations , 1997.

Forest Research Extension Project. Conducting Project and Project Evaluations: A Primer for Natural Resource Project Managers in British Columbia , 2003.

Gray & Guppy. Successful Surveys: Research Methods and Practice , 2003.

McLaughlin and Jordan. Logic Models: A Tool for Telling Your Program's Performance Story , 1999.

Stufflebeam, D.L. A Checklist Organizer. Guidelines for Choosing and Applying Evaluation Checklists:

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

Treasury Board of Canada Secretariat. The Art and Architecture of Writing Evaluation Reports , 2004.

Treasury Board of Canada Secretariat. Guide for the Review of Evaluation Reports , January, 2004.

Treasury Board of Canada Secretariat. Principles for the Evaluation of Programs by Federal Departments and Agencies , 1984.

Treasury Board of Canada Secretariat. Preparing and Using Results-based Management and Accountability Frameworks , April 2004.

Treasury Board of Canada Secretariat. Program Evaluation Methods: Measurement and Attribution of Program Results.

Treasury Board of Canada Secretariat. RBM E-Learning Tool.

Section Five: Managing Evaluations

5.1  When to Utilize External Resources

This subsection outlines guidelines for determining when to use external resources. Following are the two basic questions to be answered when deciding whether to employ external resources in an evaluation project:

1. Are there staff members with the requisite background and knowledge available within the agency to conduct the evaluation?
2. Is there a need to employ an external consultant to maintain objectivity or the appearance of such?

The first factor relates to the availability of the internal evaluation resources. When an agency lacks specific experience or knowledge, it may wish to contract for that aspect of the evaluation. In such circumstances, it may be useful to set up the contract so that there is some knowledge transfer to the agency. An agency may also lack the resources to do things like telephone surveys. Finally, an agency simply may not have the resources available internally to conduct the evaluation project and will have to resort to employing an external resource.

For the second factor, the agency needs to consider the use and target audience of the report and the risk associated with using an internal versus an external resource. Staff involvement in some evaluation tasks may bias the results (e.g., focus groups, administering a survey on client satisfaction) and for these specific tasks, an agency may wish to employ an external resource. For controversial programs where the report will be used as a public accountability tool, employing external resources may mitigate risks.

Source :  Adapted from Who Should Conduct Your Evaluation? http://www.bja.evaluationwebsite.org/guide/documents/chapter_3_housing.htm

If you answered "yes" to Question 1, but "no" to all other questions, then you will likely need external assistance in conducting your evaluation. If you answered "no" to Question 1 but "yes" to most of the other resource questions, then an in-house evaluation may be a good choice.

5.2  Preparing for Evaluation

In this subsection, two important steps to preparing for an evaluation will be described. They are as follows:

• Establishing Roles and Responsibilities; and
• Preparing the Terms of Reference.

5.2.1  Roles and Responsibilities

Project Coordinator

You will need to decide who the project coordinator will be before proceeding with the evaluation. If the coordinator is to be managing an external consultant, he/she should be familiar with the project, understand the basics of evaluation, and have good project management skills.

Advisory Committee

You may also need to set up an advisory committee comprised of agency and other stakeholder representatives that have an interest in the project or organization being evaluated. It should be noted, however, that setting up an Advisory Committee on an as needed basis may be very resource intensive for a small agency. The need for an Advisory Committee must therefore be weighed very carefully. Where appropriate, a small agency may want to consider existing work teams to act in an advisory capacity.

Steering Committee

A Steering Committee is usually comprised of senior managers and might also include central agency representatives and, where appropriate, regional representatives. When you want the advisory committee to have decision-making powers (i.e., final word on evaluation deliverables), then you may consider setting up a Steering Committee.

5.2.2  Setting the Evaluation's Terms of Reference (TORs)

TORs provide an overview of the evaluation and they make explicit management's initial requirements and expectations for the evaluation. The TORs are a useful tool for senior management in that they can guide the process until the evaluation work plan becomes the primary document.

TORs are used for many purposes.

• To engage senior managers and ensure that the evaluation will address their requirements.
• To help manage the evaluation.
• To secure necessary stakeholder members.
• To develop the evaluation work plan.

To prepare the terms of reference, you may consider the following:

• reasons for the evaluation;
• issues to be addressed;
• resources available for conducting the evaluation;
• anticipated costs;
• expertise required to complete the evaluation; and
• time frame for completion.

The terms of reference provide the basis for the next step in the process – selecting the evaluator or evaluation team. Evaluators are selected through the established contracting process.

Contracting externally for an evaluation team involves the following four steps:

1.  determine the sourcing options;

2.  identify the best value from potential candidates;

3.  notify the successful candidate (posting on MERX for competitive process); and

4.  negotiate and sign the contract.

Bidders often have questions about the Terms of Reference. You should designate one contact person to co-ordinate the responses to these questions. You must ensure that you provide the same information to all proponents and that you do not identify the source of the question. You may want to consider faxing or e-mailing responses to all potential bidders.

Guidelines for Budgeting

Budgeting for evaluation should be part of the upfront planning. It is important to identify a budget amount or range in the TORs. It also allows you to judge which proposals offer the best value per dollar.

Typically, budgeting involves breaking down the evaluation into components or tasks and providing estimates for each component. You may consider costs with respect to evaluation staff, consultants, travel, communications, printing and photocopying, supplies and equipment, and translation and editing.

Leveraging Resources for Evaluation

A small agency can supplement its limited resources by leveraging its needs and resources. While not all strategies might be applicable to your agency, here are some suggestions:

• Group numerous external and internal data collection and analysis exercises as much as possible to avoid redundancy and achieve economies of scale in data collection.
• Search for extra funding from central agency functions desiring coverage of special issues or groups (e.g., Women's Issues or Service Improvement/Government On-Line pilot projects).
• Conduct evaluation work with other agencies involved in the same issues or initiatives.
• Utilize project teams of seconded resources including those from client groups or provincial partners.
• Collaborate with partner agencies to pre-qualify consultants to form a pool of readily available and experienced resources to reduce the administrative burdens of contracting and help ensure quality, useful products.

Preparing the Statement of Work

The statement of work is the main document between the manager and the consultant. The consultant responds to this document by developing a work plan. It outlines the study purpose and objectives, approach, data collection methods, and tasks. If an RMAF or evaluation framework has been prepared, this may be used as a guide when preparing the Statement of Work. As you may recall from the previous section, an RMAF outlines the sources of information and data collection methods for each evaluation question.

One key decision is the level of detail you may want to include in this section. You may choose a less detailed statement of work relying on the proponent to suggest specific approaches and methodologies. However, a detailed statement of work is useful as it gives the proponent a better idea of your expectations. It is also important to keep in mind that the proponent can suggest changes to approach and methods. It is helpful to request that the proponents include a discussion of possible methodological challenges and solutions. This may help you assess the proponent's expertise and may also serve to strengthen the proposed work plan.

5.3  Contracting Options

If you do decide to use external resources you then have to consider the contracting options. You can review the Contracting Policy on the TBS Web site at
http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/contracting/contractingpol_2_e.asp .

These options include the following:

TBS Standing Offer for Small Agency Evaluation Services. (This option is under development at time of printing.)

Proposal calls on MERX – Canada's Official Electronic Tendering Service.

Selective Tendering – The requirement for bids may be set aside when the total costs of the contract (including GST) do not exceed $25,000. In such cases where a fully open competitive process would not really be cost effective, a limited tendering approach may be used. This is done by inviting a minimum of three firms from a source list to bid. This approach is useful for obtaining the best value for money and providing a fair and competitive access to government procurement. Where can you locate sources? Consider the branch's corporate memory and contractor inventory along with the contracting authority's corporate memory, inventory and short lists obtained from other departments and agencies. In this situation, a formal Request For Proposals (RFP) is not required.

Sole Source – used in cases where only one person or firm is capable of performing the job.

Standing Offer – can be used for services of a repetitive nature.

It should be noted that the Government Contracts Regulations contain only four exceptions that permit the contracting authority to set aside the requirement to solicit bids.

Please Note: You may opt to secure the services of Consulting and Audit Canada (CAC). With respect to federal government agencies, there is a Memorandum of Understanding rather than a contract. Federal government contracting rules do not apply when this option is used.

5.4 Choosing Consultants

The terms of reference provides the basis for choosing the consultant. It is important to develop the criteria to be used for selecting a consultant during the planning stages (i.e., before sending out the RFP).

Mandatory and Rated Requirements

The criteria you use to select the proposal should reflect the requirements of the specific project. The RFP typically contains a set of mandatory and rated criteria. To be considered, a proposal must meet the mandatory requirements set out in the RFP. Mandatory requirements set the minimum requirements for the bid to be considered. They are assessed on a simple met/not met basis. These requirements are expressed by using clear and strong terms such as "shall," "must" and "will." When mandatory requirements are used, the RFP shall clearly indicate that failure to meet any of the mandatory criteria will render the bid non-compliant and that it will be given no further consideration.

Proposals that meet the mandatory requirements may go on to be rated against other criteria and may be required to achieve a minimum level for the point-rated requirements to be given further consideration (e.g., score at least 70 points out of 100 to move to the next step in the evaluation process). Each criterion must have associated scoring criteria (the basis for scoring). Contractors can be asked to provide evidence to demonstrate their capacity to respond to a criterion (e.g., résumé).

When you are deciding what rated criteria to include in the RFP, you may want to consider what the most important factors are that will affect the quality of the work to be done (e.g., subject matter expertise, methodological experience, academic qualifications).

Quantitative rating scales are a useful decision-making device. However, these scales have to be constructed and weighted very carefully to ensure you are selecting the right consultant. The rate assigned to each criterion should reflect its importance.

5.4.1  Methods of Selection

The following methods are commonly used to assess proposals. The method to be used should be articulated in the RFP.

1.  The lowest price proposal method. This may be used when cost is the most important factor. This method can be used when mandatory requirements are the main tool for evaluating the proposals and suppliers offer uniform services.

2.  The best technical proposal method. The proposal that receives the highest score within budget is the winner. This method may be used when technical merit is the most important factor. A maximum budget is identified in the RFP. The point-rated requirements are typically used to assess proposals.

3.  The best overall value for money method or cost-per-point method. The proposal that has the best ratio score/price is the winning proposal. It is used when technical merit and price are both important factors .

5.4.2  Criteria to Consider When Reviewing Proposals

When reviewing the proposal, you need to ask yourself the following question: Does the evaluator have a proven ability to deliver results and meet expectations? Other criteria are presented in the checklist below.

5.5  Best Practices for Managing Consultants

Establishing clear expectations at the beginning of a project is crucial to its success. This subsection considers best practices in terms of scope of work, managing the project, and follow-up.

The following checklist provides an overview of the management process from its inception to the reporting of findings.

5.5.1  Defining the Scope of Work

The first planning meeting or initial start-up meeting is critical for clarifying expectations and the evaluation work plan. The project manager should also ensure that the consultant has all the information needed to carry out the project – relevant documents and contact information.

For the start-up meeting you should

ensure that the evaluator(s) have full access to files, reports, publications and any other relevant information;

ensure there is adequate administrative and logistical support during the evaluation;

establish project management and reporting expectations (how often and in what format (written, oral); and

identify the major issues and priorities.

5.5.2  Preparation of the Work Plan

The evaluation work plan or methodology report will likely be the first deliverable produced by the consultant. The work plan should provide a clear description of what the evaluation team is expected to do, as well as where, when, how, and why. The work plan typically builds on the proposal and statement of work. Some changes to the original proposed work plan may be suggested given new information or other considerations.

Here are some key elements of the work plan:

• project background;
• detailed description of proposed methodology;
• specific work schedule; and
• data collection instruments (i.e., interview guides, survey questions).

5.5.3  Overseeing the Day-to-Day Operations with Consultants

The relationship between the consultant and evaluation manager is a partnership in the sense that you bring subject matter expertise and he/she brings evaluation expertise. However, the main role of the evaluation manager is to ensure the consultant follows the agreed upon work plan and provides a satisfactory level of quality. The evaluation manager must also address issues that the consultant might raise when preparing and implementing the work plan. While there is no single good strategy for managing consultants, the checklist below may offer some helpful considerations.

5.5.Managing Follow-ups

In the 3 to 6 months following the completion of the evaluation report, you may develop a better sense of the usefulness of the evaluation. You may want to consider providing follow-up feedback to the evaluator in terms of the usefulness of the evaluation. This information serves to build the evaluator's capacity and benefits the agency by improving the available resource pool.

Key References

Canadian Evaluation Society. Evaluation Methods Sourcebook , 1991.

Financial Management Board Secretariat. NWT. Working Well With Consultants : http://www.gov.nt.ca/FMBS/documents/dox/Consultant%20Guide.pdf .

HRDC. Evaluation Tool Kit. User Guide on Contracting HRDC Evaluation Studies , 1999.

Treasury Board of Canada Secretariat. TBS Contracting Policies , 2003.

UNFPA, Office of Oversight and Evaluation. Planning and Managing an Evaluation , http://www.unfpa.org/monitoring/toolkit/tool5_4.doc .

Section Six: Using Evaluations

A successful evaluation is one that is used. Communications is an essential tool that can directly influence the application of results generated from evaluations. The adaptation of lessons learned and recommendations from evaluation findings increase with an efficient and systematic communication strategy.

In this section, strategies for communicating and using evaluation findings will be highlighted. Strategies for enhancing utilization of findings will also be discussed.

6.1  Communicating Evaluation Findings

 
Developing a communications strategy is a good first step in communicating evaluation findings. You may want to consider the following steps when developing a communications strategy:

• What is the purpose of the communication about the evaluation?
• Who are the target audiences? What are the key messages of the evaluation? Who needs to know what?
• How can each audience best be reached?
  • What are their information needs?
  • What will the audience relate to and understand?
• How will you share sensitive or negative results?
  • Present the positives with the negatives.
  • Foster a problem-solving approach.

6.1.1  Communication to Managers

Communication of results to managers should consider their unique information and decision-making needs. Communications may focus on

• the overall performance of the program;
• reasons why program is achieving or not achieving results;
• managerial performance;
• levels of employee and/or client and stakeholder satisfaction; and,
• missed targets or achievements including reasons for these findings.

Managers need a clear idea of the perspectives and opinions of different stakeholder groups. It is therefore important to report findings according to these various groups.

It may also be advisable to provide preliminary findings to program managers via a presentation. A presentation can offer the evaluator with additional information to more effectively communicate the findings and can also help prepare the management response to the evaluation.

6.1.2  Communication to Stakeholders

Stakeholder groups can include the public, special interest groups, partners, third party deliverers, other federal government departments and agencies, and provincial or municipal governments.

Stakeholders may want information with respect to the program or project's broader impact (e.g., on society) and value for money. They may also be interested in understanding the relationship of the project to overall government goals and activities (e.g., are there duplications?)

Appropriate strategies for widely disseminating results to stakeholders can include general meetings, seminars, conferences, Web sites, and annual reports.

6.1.3  Communication to Senior Managers [8]

Generally, senior managers require more "high-level" strategic information. Senior managers require an understanding of how the findings relate to strategic objectives. Any concerns with respect to legislation, regulations, and policy should be promptly communicated to this group. Findings relating to values and ethics of the organization may also be given emphasis when communicating to this group.

Remember that evaluations are generally submitted to the evaluation committee for approval.

6.2  Using Evaluation Findings

Remember the evaluation and program management cycle presented in Section Two? An effective feedback system needs to be incorporated within the program or project management cycle. Ideally, evaluation findings should be used to make informed decisions. In practice this may not happen. Why?

Sometimes lack of use relates to poor communication of the evaluation findings. At other times, the evaluation report may lack relevance to the manager's information and decision-making needs.

Effective strategies for use of evaluation reports begin in the planning stage. A good evaluation design needs to correspond with information needs. It will be easier to communicate and use results within the Agency if the evaluation questions are of relevance to decision makers.

Management Action Plan

An effective management response to the evaluation report is also a critical step in ensuring that the evaluation is used. The action plan should adequately address the findings and recommendations of the report. The plan should identify the required action, timelines and who is responsible for carrying out the action. The head of evaluation may play a role in monitoring the implementation of an action plan.

Key References

Annie E. Casey Foundation, Baltimore, MD. Getting Smart, Getting Real: Using Research and Evaluation Information to Improve Projects and Policies , September 1995. http://www.aecf.org/publications/data/getsmartgetreal.pdf

National Institute of Justice, Washington, D.C. Using Evaluation Findings for Decision Making , 1989. http://www.bja.evaluationwebsite.org/guide/documents/chapter_5_nij_guide.htm

National Institute of Law Enforcement and Criminal Justice, Washington, D.C. Achieving Utilization of Evaluation Findings , 1975. http://www.bja.evaluationwebsite.org

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

UN-HABITAT. Monitoring and Evaluation Guide , 2003.

Appendix A―Types of Small Agencies

Regulatory: Agencies that grant approvals or licences based on criteria set out in legislation or regulation.

Judicial: Courts presided over by federally appointed judges. (Please note that the Federal Court and Tax Court recently amalgamated to form the Courts Administrative Service, which leaves only the Supreme Court.)

Quasi-judicial Tribunal: Agencies that hear evidence under oath and render decisions based on that evidence alone in conjunction with the applicable statutes and precedents but independent of government policy.

Investigative Agency: Agencies that investigate a complaint or inquiry and report or make recommendations on their findings.

Parliamentary Agency (Agents of Parliament): Agencies that report directly to Parliament (i.e., Information Commissioner and Privacy Commissioner, Commissioner of Official Languages, Auditor General and Chief Electoral Officer).

Policy Development and Advisory: Agencies that develop policy and make recommendations to the government on issues such as health, the economy, or environment.

Other: Small agencies that do not fall under any of the above categories.

Appendix B―Horizontal Initiatives

Horizontal initiatives are efforts involving the co-ordinated activities of several federal departments and/or agencies focussed on specific objectives of national interest. They must have an RMAF and associated evaluation plans and strategies to which all partners are expected to contribute information and possibly resources.

These types of initiatives may provide opportunities for small agencies to share and/or develop internal evaluation capacity (e.g., through case studies or peer review steps in an evaluation design, through existing internal information, and benefiting from evaluation expertise or consulting expertise provided or paid for by other partners). Horizontal initiatives also provide opportunities for an appropriate grouping of agencies to develop common indicators and share experience in performance measurement and evaluation.

For some general guidance on developing Results-based Management and Accountability Frameworks, please refer to the TBS Web site at http://www.tbs-sct.gc.ca/cee/pubs/guide/sarmaf-ascgrr-fra.as .

Types of horizontal initiatives include:

• partnerships with other jurisdictions;
• health promotion;
• public safety and anti-terrorism;
• climate change;
• youth employment;
• Government-wide initiatives such as
  • infrastructure;
  • Government On-Line (GOL);
  • official languages; and
  • aboriginal procurement.

Some challenges for evaluating horizontal initiatives include:

• the need to minimize the number of performance indicators;
• the difficulty of collecting information across different databases, agencies, and/ or departments; and
• co-ordination.

Appendix C―Seeking External Advice and Support

It is important to have adequate advice and support when attempting to build evaluation capacity. Consider the following possible sources of support.

TBS/CEE Support for Small Agencies

According to the TB Evaluation Policy , the Treasury Board of Canada Secretariat must provide central direction for evaluation; use evaluation results where appropriate in decision-making at the centre; and set standards and monitor capacity in the government.

At TBS, the Centre for Excellence in Evaluation (CEE) was established to

• provide leadership for the evaluation function within the federal government;
• take initiative on shared challenges within the community, such as devising a human resources framework for long-term recruiting, training and development needs; and
• provide support for capacity building, improved practices, and a stronger evaluation community within the Public Service of Canada.
• The CEE has also established a Small Agency Portfolio Team. The function of this team is to
  • provide feedback to TBS Program Sectors on evaluations and RMAFs submitted as part of TB Submissions;
  • monitor the evaluation function in small agencies;
  • undertake projects to support the small agency community in its evaluation functions and activities; and
  • provide advice and guidance to small agencies on
    • evaluation function and capacity
    • evaluation plans
    • evaluation studies
    • RMAFs
    • performance measurement activities

Other particularly relevant TBS areas include those involved in the change functions associated with Modern Comptrollership, Internal Audit, Results-based Management, Horizontal Reporting, and Expenditure Reporting. For more information, see the following Web site http://www.tbs-sct.gc.ca/cee/about-apropos-eng.asp .

Canadian Evaluation Society (CES)

In the field of evaluation, the CES promotes leadership, knowledge, advocacy, and professional development. The CES provides access to a community of evaluators, annual conferences, the Essential Skills Series of courses in evaluation, and reserved resources on the CES Web site ( http://www.evaluationcanada.ca/ ). The CES has various provincial chapters as well as a National Capital Chapter.

Small Agency Administrator's Network (SAAN)

SAAN's mission is to provide opportunities for small agencies to share information and practices as well as to discuss issues of common concern and to provide a common voice to Central and Common Service Agencies with respect to small agency issues. For more information, see the following Web site: http://www.cso-cpo.gc.ca/saan-rapo/charter_e.html .

Appendix D ―Expenditure Review Committee's 
7 Tests

Program spending will be assessed against the following specific tests:

1.  Public Interest Test – Does the program area or activity continue to serve the public interest?

• What public policy objectives is the initiative designed to achieve?
• How does it align with current government priorities and the core mandate of the organization?

2.  Role of Government Test – Is there a legitimate and necessary role for government in this program area or activity?

• Governance: Who else is involved? Is there overlap or duplication?

3.  Federalism Test – Is the current role of the federal government appropriate, or is the program a candidate for realignment with the provinces?

• What are the initiative's impacts on other levels of governments? Could they play a greater role?

4.   Partnership Test – What activities or programs should or could be transferred in whole or in part to the private or voluntary sector?

• What are the initiative's impacts on the private and/or voluntary sectors and/or other key stakeholders? Could they play a greater role?

5.  Value for Money Test – Are Canadians getting value for their tax dollars?

• Results: What is the evidence that the initiative is achieving the stated policy objectives?
• Is the program citizen-centred?

6.  Efficiency Test – If the program or activity continues, how could its efficiency be improved?

• Efficiency and Effectiveness: Does the program exploit all options for achieving lower delivery costs through intelligent use of technology, public-private partnership, third-party delivery mechanisms, or non-spending instruments?

7.  Affordability Test – Is the resultant package of programs and activities affordable? If not, which programs or activities could be abandoned?

• Relativity and Performance: How do program delivery costs compare to those in other jurisdictions and the private sector for similar activities?
• Sustainability and Stewardship: What actions have been taken to manage future spending pressures? What more can be done?

Appendix E―"How to" Information for Planning and Conducting Evaluations

1.0  How do you build a logic model?

The following graphic presents an overview of the three steps for logic model development.

Step 1a:  Preparing for Logic Model Development
Determining Internal Capacity

Step 1b:  Preparing for Logic Model Development
Collecting Relevant Information

Note that the extent to which you consult with all of the above groups depends on your information needs and resource constraints. However, you should note that perspectives from a variety of stakeholders provide you with a better understanding of the program.

Step 2:  Building the Logic Model

There are different strategies for building a logic model. Two options include

• developing a draft model first and presenting it for discussion at a working session;
• developing the draft logic model during the working session.

The key advantage of the first option is that sometimes the session proceeds more efficiently if the core elements of the logic model are developed beforehand. If there is limited time to conduct a working session, you may want to consider drafting the logic model prior to the working session.

There are also advantages of the second option in that stakeholders develop the logic model. It helps to build internal capacity with respect to logic model development and group processes. It can also lead to an enhanced understanding of the initiative. Finally, this approach may also strengthen commitment of stakeholders to the process.

When making the decision, consider stakeholder and facilitator preferences and timeframe.

Step 3:  Validation of Logic Model

Consult with working groups and stakeholders. It is often helpful to solicit the feedback of individuals who are familiar with the program but who were not part of the working session to verify that all necessary elements are represented in the model.

Build awareness of the logic model. The working group or individual can create awareness on an informal basis by referring to the logic model in conversations with staff and stakeholders. Management and program teams can use the model as a consistent reference for all aspects of the management cycle: planning, monitoring, and reporting. Increased awareness will lead to feedback and insights on how to improve the model.

The model will never be prefect. Use feedback from consultations and from using it as a management reference to update it. Remember that as the context of the program changes over time, so will the underlying logic.

Key References

Treasury Board of Canada Secretariat. RBM E-Learning Tool.

2.0  How do you develop evaluation questions?

3.0  How do you identify the right performance indicators?

Step 1:  Review Logic Model

Go through each row of the logic model (except activities) and determine what specific piece of information or particular data would be required to assess whether each output has been produced or result achieved. A working session is an effective method for brainstorming indicators.

Step 2:  Prioritize

Identify the "need to have" versus the "nice to have" for each component.

Once a comprehensive set of performance indicators and associated measurement strategies have been identified, a smaller set of the best indicators needs to be identified. Check the top ranked indicators against the selection criteria described.

Step 3:  Check Against Criteria

4.0  How do you choose an appropriate evaluation design?

Evaluation design is the process of thinking about what you want to do and how you want to go about doing it.

Considerations of threats to validity in choosing evaluation design

When developing an evaluation design, you have to consider whether other factors are affecting the results of the program. It is particularly important when you are trying to determine the impacts or effectiveness that these factors or threats to validity are considered. These factors can be due to real changes in the environment or changes in participants involved in the program.

• Changes in the environment that occur at the same time as the program and will change the program results (e.g., the state of the economy could influence the results of a program)
• Changes within individuals participating in program (e.g., changes due to aging or psychological changes that are not the result of the intervention)
• The evaluation itself may influence the results (e.g., effects of taking a pre-test on subsequent post-tests, inconsistencies in observers, interviewers, scores or measuring instruments).

Overview of Evaluation Designs

Both quasi-experimental and experimental designs involve some type of pre-test followed by a post-test. Both design types are appropriate for conducting summative evaluations. However, practicality and costs must also be considered.

Key References

Treasury Board of Canada Secretariat. Program Evaluation Methods: Measurement and Attribution of Program Results , 1998.
http://www.tbs-sct.gc.ca/cee/pubs/meth/pem-mep00-eng.asp

5.0  How do you choose appropriate data collection methods?

To choose an appropriate data collection method, you may consider the following:

• information and decision-making needs;
• appropriate uses, pros and cons of the data collection methods;
• costs and practicality of each method; and
• taking a balanced approach, including a mix of quantitative and qualitative methods.

For more detailed information, see the two tables below that compare the quantitative and qualitative methods, as well as describe the specific data collection methods available for evaluations.

A Comparison of Quantitative and Qualitative Methods

Source:  Adapted from the Program Manager's Monitoring and Evaluation Toolkit Number 5, Part III: Planning and Managing the Evaluation – the Data Collection Process . May 2001 ( www.unfpa.org United Nations Population Fund, Office of Oversight and Evaluation).

Overview of Data Collection Methods

Source :  Adapted from TBS, RBM E-Learning Tool

 Key References

Treasury Board of Canada Secretariat. RBM E-Learning Tool .

6.0  How do you design a survey questionnaire?

Step 1:  Sampling Procedures

Clearly define purpose of survey

In order to develop sampling procedures, you need to clearly define the purpose of the survey. The sampling strategy must be designed to answer the evaluation questions.

• What are the key evaluation questions the survey can answer?
• What are the priorities for the survey?
• What are the characteristics of the general population you wish to survey (e.g., gender, age)?

Sample Size

Determining the sample size will help determine which type of survey to use.

The following are considerations for determining sample size:

• budget;
• the level of precision required and extent of sub-population comparisons;
• a smaller sample size is used if you have reason to expect a strong effect;
• Expected non-response rate – you can increase the sample size by that factor; and
• to track respondents over time, a much larger sample is required (to account for people who no longer wish to participate in the survey).

If you wish to draw conclusions about the entire population, you will require a certain sample size that is based on statistical parameters. For example, when polling firms say such a sample is accurate with a 5 per cent margin of error, 90 per cent of the time, this is based on a certain size of sample. The required sample size is related to the size of your population, the confidence level required, and the allowable margin of error. For example, if you have a population of 1,000 people from which you need a representative sample and you require a 95 per cent confidence interval with a 5 per cent margin of error, then you will require a maximum sample of 278. In social science research, a 95 per cent confidence level and a 5 per cent allowable margin of error is typically specified. You may need to consult with a statistical expert regarding appropriate sample size.

Sampling Techniques

There are a number of different methods that can be used to select a sample. Ideally you want the sample to represent the whole group so that you can generalize the findings to the program. Types of sampling include simple random sampling, stratified random sampling, systematic sampling, and cluster sampling.

With simple random sampling a list of all people (i.e., the survey population) is made and then individuals are selected randomly for inclusion in the sample. Random sampling means that everyone in the target group has an equal chance of being included in the study. One challenge with this approach is obtaining a complete list of the group.

The steps for sampling procedures are as follows:

1.  obtain a sampling frame;

2.  check for bias;

3.  assess the potential sampling source in advance; and

4.  apply sampling procedure.

Sampling Frames

Sampling frames are listings of people that represent or approximate the population of interest. Sampling frames should be comprehensive and representative. The list should be unbiased.

Check for Biases

Some common biases include:

• lists of only approved applicants and not rejected applicants (where the sampling frame is a list of applicants); and
• lists are out of date.

Assess the potential sampling sources in advance

• Verify if contact information is available (e.g., for all provinces/cities/services).
• Verify whether there is information about when the service was received.
• Minimize recall bias.
• Conduct the survey as close to the service event as possible.
• If measuring satisfaction with a product, you need to allow enough time for client(s) to use it.

What if contact information doesn't exist? Options include

• on-site surveys; and
• asking clients at the end of a transaction if they would be willing to participate in a client satisfaction survey.

What if information about potential respondents is limited?

• Sample during the survey.
• Ask what services they received from where and when.
• Ask them to target their answers to a specific time period, or channel, etc.

Examples of Random Sampling Technique

• Systematic sampling – Select every "nth" number. Make sure there are no hidden patterns in population list.
• Random digit dialling – used for telephone surveys – enables interviewers to call unlisted, new, and recently changed numbers.

Step 2:  Determining the Survey Format

• Considerations for determining the appropriate survey format (on-line, mail, telephone, face-to-face) includes the following:
  • type of information the respondent is expected to provide;
  • budget – Telephone interviewing can be more expensive. Mail surveys are more economical if you have a large sample or if your sample spans a large geographical area.
  • sample size;
  • speed – online and telephone surveys are most timely.
  • length of survey – for lengthy surveys (over one hour), consider in-person;
  • subject matter – if questions are personal or require thought, consider self-administered survey (mail, on-line).

Step 3:  Develop Survey Questionnaire

A key consideration in developing your questionnaire is determining what types of questions to use. While open-ended questions can provide detailed information, they are time consuming to record and analyze. If you have a large group to survey, the questionnaire should be largely comprised of closed-ended questions.

You should also prepare a script and instructions for the interviewers (if by telephone or in-person). The script should include how to greet the respondent, how to invite them to participate, how to respond to their answers, how to keep the respondent on the line, how to thank the respondent, how to code each survey (completed, no answer).

Step 4:  Pre-testing the Questionnaire

Surveys can be pre-tested and then adjusted. The purpose of pilot testing is to identify and resolve problems and deficiencies in the information collection methods or in the form and usefulness of the information gathered. Based on the results of the pre-test you can modify the survey accordingly. Conducting a small number of about 10 interviews can provide higher quality survey results.

You should determine what questions your pre-test will answer. For example, your pre-test might want to determine the following:

• Can the respondents answer the questions?
• How long did the interview last? Is this within your budget?
• Did the respondent have any problems interpreting or understanding the questions?
• When you analyzed the preliminary findings, did the results make sense?

Step 5:  Implement the Survey

During implementation

• track progress periodically
• monitor your response rate (A higher response rate returns higher quality results, since self-selection and other biases are minimized by tracking down the original clients who were sampled.)

If you expect your results to be comparable with existing results for benchmarking purposes, the research design, questionnaire, and administration need to be very similar, if not identical. It is okay to make some improvements over the previous iteration, but the bigger the change, the less comparable the results.

You should ensure there are procedures in place for following up non-response and compensating for lower response rates. Changes in the original composition of a sample are usually inevitable during the course of an evaluation study. Individuals may drop out from the sample and others may provide incomplete information. These changes may bias the study if they are not addressed.

Key References

Goss Gilroy Inc., Designing Effective Client Satisfaction Surveys , Strategic Management Conference, Montreal, 2003.

SPSS BI Survey Tips, A Handy Guide to Help You Save Time and Money as You Plan, Develop and Execute Your Surveys . http://www.spss.com/uk/SurveyTips booklet.pdf

7.0  How do you analyze data?

Step 1: Start with the evaluation objectives

When analyzing any type of data, review the purpose of the evaluation. This will help you organize your data and focus your analysis.

Step 2: Review for accuracy, completeness and consistency

Step 3: Summarize and organize data

• Describing and counting – These are two of the most common analytic techniques and are often required as the basis or context for further data analysis. All types of qualitative and quantitative data at the input, output, and result stages can be described and counted. Data are gathered from various data sources using previously described data collection methods. Qualitative data can be described in narrative form or counted and analyzed using a variety of statistical techniques. Quantitative data can also be used to describe a program or purpose, and are easily counted and coded for analysis.
• Aggregating and disaggregating – Aggregating is the process of grouping (or clustering) data by identifying characteristics or patterns that seem to link them. Disaggregating means breaking down (or factoring) information into smaller units. The reason for aggregating data is to determine whether relationships exist among different variables based on a pre-existing theory (hypothesis) or patterns seen in the data. Disaggregated data can be examined in different ways (e.g., over time, across different populations, between two comparison groups).
• Comparison – Comparison covers a range of methods that can be used to draw conclusions about the relationship among data and make generalizations to a larger population. Comparison involves contrasting a person or population against itself, another comparison group, or a standard typically after an event or the implementation of a program.

Generalizing the Findings

The only valid way of generalizing findings to an entire or target population (where you cannot survey or study everyone) is to use findings from a random sample of the population you wish to study. Caution must therefore be exercised when analyzing data from non-randomized samples.

Qualitative and Quantitative Analysis

Analyzing qualitative data requires effective synthesis and interpretative skills. Qualitative information can be used to provide contextual information, explain how a program works, or to identify barriers to implementation. Qualitative data can be analyzed for patterns and themes that may be relevant to the evaluation questions. Qualitative material can be organized using categories and/or tables making it easier to find patterns, discrepancies and themes.

Quantitative data analysis provides numerical values to information. It can range from simple descriptive statistics (e.g., frequency, range, percentile, mean or average) to more complicated statistical analysis (e.g., t-test, analysis of variance). Computer software packages such as Statistical Package for Social Sciences (SPSS), Minitab, and Mystat can be used for more complicated analysis. Quantitative data analysis also requires interpretation skills. Quantitative findings should be considered within the context of the program.

About quantitative data

Frequencies, range, percentile, and standard deviation are used for descriptive statistics. Measures of central tendencies – mean, median, or mode – are also calculated.

Examples of inferential statistics:

• Correlation coefficients are used to determine the strength of the relationship between two variables.
• T-tests are used to determine differences in average scores between two groups.
• ANOVA (analysis of variance) determines differences in average scores of three or more groups.

Key References

Treasury Board of Canada Secretariat. RBM E-Learning Tool.

Appendix F―Terms of Reference Template

Toolkit: Templates
Agency Audit and Evaluation Plan Template [9]

Introduction and Context

• Identify management client and stakeholder information needs
• Outline how audit (where applicable), risk management and evaluation will be used in the agency
• Link evaluation to strategic concerns (PAA strategic outcomes, program inventory and performance measures)
• Refer to TB Evaluation Policy and Policy on Internal Audit

Methodology/Approach

• Methodology used for determining projects
• Take into account priority setting and risk management approach
• Link to agency service, business lines and strategic priorities

Rationale

• Indicate scope and coverage for evaluation, audit, risk management plan
• Rationale for including study in the plan – factors considered in selecting audit and evaluation projects
• Give appreciation of proportion of the agency's evaluation universe the current year's projects represent
• If applicable consider cross-jurisdictional evaluations

Evaluation and Audit Plan Summary

• Identify planned projects for fiscal year
• Estimate costs for completing each project and/or planned expenditure in current fiscal year
• Total expenditure on evaluation, funding received in addition to A-base funding for evaluation and audit

• Indicate project title, objective, client, status of project (e.g., planned, in progress)
• Identify project teams and schedules
• Identify key assumptions in order to achieve deliverables as per plan
• Consider TBS standards during development of plan

Appendices

• May include draft TORs, statements of work for proposed projects or expenditures

Appendix G―Glossary

Accountability ( Responsabilisation) – The obligation to demonstrate and take responsibility for performance in light of agreed expectations. There is a difference between responsibility and accountability: responsibility includes the obligation to act whereas accountability includes the obligation to answer for an action

Activity ( Activit̩ ) РAn operation or work process internal to an organization, intended to produce specific outputs (e.g., products or services). Activities are the primary link in the chain through which outcomes are achieved.

Attribution ( Attribution ) – The assertion that certain events or conditions were, to some extent, caused or influenced by other events or conditions. This means a reasonable connection can be made between a specific outcome and the actions and outputs of a government policy, program, or initiative.

Departmental Performance Reports (DPR) ( Rapport minist̩riel sur le rendement (RMR) ) РDepartmental Performance Reports, tabled in the fall of each year by the President of the Treasury Board on behalf of all federal departments and agencies named in Schedule I, I.1 and II of the Financial Administration Act , are part of the Estimates and Supply process. The reports explain what the government has accomplished with the resources and authorities provided by Parliament. The performance information in the reports is intended to help members of Parliament advise the government on resource allocation in advance of the annual budget and Supply process in the spring.

Effectiveness ( Efficacit̩ ) РThe extent to which an organization, policy, program, or initiative is meeting its planned results. (A related term is Cost Effectiveness РThe extent to which an organization, policy, program, or initiative is producing its planned outcomes in relation to expenditure of resources.)

Efficiency ( Efficience ) – The extent to which an organization, policy, program, or initiative is producing its planned outputs in relation to expenditure of resources.

Evaluation ( Évaluation ) – The systematic collection and analysis of information on the performance of a policy, program, or initiative to make judgements about relevance, progress, or success and cost-effectiveness and/or to inform future programming decisions about design and implementation.

• Annual Reference Level Update (ARLU)
• Estimates
• Reports on Plans and Priorities (RPP)
• Departmental Performance Reports (DPR)

Final Outcome ( R̩sultat final ) РThese are generally outcomes that take a longer period to be realized, are subject to influences beyond the policy, program, or initiative, and can also be at a more strategic level.

Goal ( But ) – A general statement of desired outcome to be achieved over a specified period of time. The term goal is roughly equivalent to Strategic Outcome. For technical precision, the Treasury Board of Canada Secretariat recommends that Strategic Outcome be used instead of goal. See also Objective.

Horizontal Result ( R̩sultat horizontal ) РAn outcome that is produced through the contributions of two or more departments or agencies, jurisdictions, or non-governmental organizations.

Impact ( Impact ) – Impact is a synonym for outcome, although an impact is somewhat more direct than an effect. Both terms are commonly used, but neither is a technical term. The Treasury Board of Canada Secretariat recommends that result be used instead of impact.

Indicator ( Indicateur ) – A statistic or parameter that provides information on trends in the condition of a phenomenon and has significance extending beyond that associated with the properties of the statistic itself.

Input ( Intrant ) – Resources (e.g., human, material, financial) used to carry out activities, produce outputs, and/or accomplish results.

Logic Model ( Mod̬le logique ) Р(also referred to as Results-based Logic Model) An illustration of the results chain or how the activities of a policy, program, or initiative are expected to lead to the achievement of the final results. Usually displayed as a flow chart. See also Results Chain.

Management Accountability Framework (MAF) ( Cadre de responsabilisation de la gestion (CRG) ) – It is a set of expectations for modern public service management. Its purpose is to provide a clear list of management expectations within an overall framework for high organizational performance.

Management of Resources and Results Structure (MRRS) ( Structure des ressources et des r̩sultats de gestion (SRRG) ) РThe MRRS replaces the Planning, Reporting, and Accountability Structure (PRAS) policy as the new reporting regime. The MRRS

• clearly defines appropriate strategic outcomes;
• is a complete program inventory that links all departmental programs and program activities so that they are aligned with strategic outcomes;
• sets performance measures for each level of the department's architecture; and
• ensures that a departmental governance structure that defines decision-making and accountability by strategic outcome and by program is in place.

Mission Statement ( Énoncé de mission ) – A formal, public statement of an organization's purpose. It is used by departmental management to set direction and values.

Objective ( Objectif ) – The high-level, enduring benefit towards which effort is directed.

Outcome ( R̩sultat ) РAn external consequence attributed to an organization, policy, program, or initiative that is considered significant in relation to its commitments. Outcomes may be described as immediate or intermediate; final, direct or indirect; intended or unintended. See also Result.

Output ( Extrant ) – Direct products or services stemming from the activities of a policy, program, or initiative, and delivered to a target group or population.

Performance (Rendement) – How well an organization, policy, program, or initiative is achieving its planned results measured against targets, standards, or criteria. In results-based management, performance is measured, assessed, reported, and used as a basis for management decision-making.

Performance Measurement Strategy ( Strat̩gie de mesure du rendement ) РSelection, development, and ongoing use of performance measures to guide corporate decision-making. The range of information in a performance measurement strategy could include reach; outputs and results; performance indicators; data sources; methodology; and costs.

Performance Measures ( Mesures du rendement ) – An indicator that provides information (either qualitative or quantitative) on the extent to which a policy, program, or initiative is achieving its results.

Performance Monitoring ( Suivi du rendement ) – The ongoing process of collecting information in order to assess progress in meeting Strategic Outcomes, and, if necessary, provide warning if progress is not meeting expectations.

Performance Reporting ( Rapport sur le rendement ) – The process of communicating evidence-based performance information. Performance reporting supports decision-making, serves to meet accountability requirements, and provides a basis for citizen engagement and a performance dialogue with parliamentarians.

Planned Results (Targets) ( R̩sultats pr̩vus (Cibles) ) РClear and concrete statement of results (including outputs and results) to be achieved within the time frame of parliamentary and departmental planning and reporting (1 to 3 years), against which actual results can be compared.

Reach ( Port̩e ) РThe individuals and organizations targeted and directly affected by a policy, program, or initiative.

Reliability ( Fiabilit̩ ) РRefers to the consistency or dependability of the data. The idea is simple: if the same test, questionnaire, or evaluation procedure is used a second time, or by a different research team, would it obtain the same results? If so, the test is reliable. In any evaluation or research design, the data collected are useful only if the measures used are reliable.

Reports on Plans and Priorities (RPP) ( Rapport sur les plans et les priorit̩s (RPP) ) РAs part of the Main Estimates, the RPPs provide information on departmental plans and expected performance over a three-year period. These reports are tabled in Parliament each spring, after resource allocation deliberations. They generally include information such as mission or mandate, strategies, as well as Strategic Outcomes and performance targets.

Result ( R̩sultat ) РThe consequence attributed to the activities of an organization, policy, program, or initiative. Results is a general term that often includes both outputs produced and outcomes achieved by a given organization, policy, program, or initiative. In the government's agenda for results-based management and in the document Results for Canadians: A Management Framework for the Government of Canada , the term result refers exclusively to outcomes.

Results Chain (also results-based logic model, results sequence) ( Enchąnement des r̩sultats (mod̬le logique ax̩ sur les r̩sultats, s̩quence de r̩sultats) ) РThe causal or logical relationship between activities and outputs and the outcomes of a given policy, program, or initiative, that they are intended to produce. Usually displayed as a flow chart.

Results for Canadians: A Management Framework for the Government of Canada ( Des résultats pour les Canadiens et les Canadiennes : un cadre de gestion pour le gouvernement du Canada ) – A document published in early 2000 that describes the management framework for the Government of Canada. This key document outlines the four management commitments for the federal government: citizen focus, values, results, and responsible spending.

Results-based Management ( Gestion ax̩e sur les r̩sultats ) РA comprehensive, life-cycle approach to management that integrates business strategy, people, processes, and measurements to improve decision-making and drive change. The approach focuses on getting the right design early in a process, implementing performance measurement, learning and changing, and reporting performance.

Results-based Management and Accountability Framework (RMAF) (Cadre de gestion et de responsabilisation ax̩ sur les r̩sultats (CGRR)) РA document that sets out the performance monitoring, evaluation, and reporting strategies for a policy, program, or initiative.

Service Commitment ( Engagement en mati̬re de service ) РService commitments or standards generally set performance objectives for the delivery of government products or services to the public, specifying the quality or level of service to which a department or agency commits, or can be expected to deliver to clients.

Strategic Outcome ( R̩sultat strat̩gique ) РA Strategic Outcome is a long-term and enduring benefit to Canadians that stems from a department's mandate, vision, and efforts. This Outcome represents the difference a department wants to make for Canadians and should be measurable. The achievement of or progress towards a strategic outcome will require, and Canadians will expect, the sustained leadership of a federal department or agency, especially in developing partnerships and alliances with other stakeholders and organizations.

Canadians also expect that departments will strive for excellence by establishing challenging outcomes that are within their sphere of control or influence. These outcomes will form the standards by which a department's performance is assessed through departmentally derived measures.

Target Group (Target Population) ( Groupe cible (Population cible) ) – The set of individuals that an activity is intended to influence.

Validity ( Validit̩ ) РThe extent to which the questions or procedures actually measure what they claim to measure. In other words, valid data are not only reliable, but are also true and accurate. Measures used to collect data about a variable in an evaluation study must be both reliable and valid if the overall evaluation is to produce useful data.

Source: TBS Guide to RMAFs .
http://www.tbs-sct.gc.ca/cee/tools-outils/rmaf-cgrr/guide00-eng.asp

Appendix H―Evaluation Web Sites

1.  http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024

This is the Treasury Board of Canada Secretariat (TBS) Evaluation Policy .

2.  http://www.oecd.org/dataoecd/29/21/2754804.pdf

This link is a comprehensive glossary of key terms in evaluation and results-based management.

3.  http://www.wkkf.org/Pubs/Tools/Evaluation/Pub770.pdf

This is an evaluation handbook by Kellogg.

4.  http://www.evaluationcanada.ca

This is the Canadian Evaluation Society homepage. It contains information on courses and special events, various resources on evaluation, and unpublished documents for evaluators.

5.  http://www.phac-aspc.gc.ca/ncfv-cnivf/familyviolence/html/fvprojevaluation_e.html

This document is entitled Guide to Project Evaluation: A Participatory Approach . It was developed by the Population Health Directorate at Health Canada in 1996. The Guide provides an easy-to-use, comprehensive framework for project evaluation. This framework can be used to strengthen evaluation skills and knowledge to assist in the development and implementation of effective project evaluations.

6.  http://www.mapnp.org/library/evaluatn/fnl_eval.htm#anchor1585345

This link contains a Basic Guide to Program Evaluation. This document provides guidance toward planning and implementing an evaluation process for non-profit or for-profit organizations.

7.  http://www11.hrdc-drhc.gc.ca/pls/edd/toolkit.list

This link contains HRDC's Evaluation Tool Kit series developed by Evaluation and Data Development (EDD). This is a series of publications that provides pertinent information about designing, planning, and conducting an evaluation. Publications include the following:

• Evaluation Tool Kit Focus Group – A guide to understanding the use of focus groups as an information gathering tool
• Quasi-Experimental Evaluation – Summarizes the basics of evaluation research focussing on the "quasi-experimental" design
• User Guide on Contracting HRDC Evaluation Studies – Summarizes the provisions of the Treasury Board of Canada Secretariat's (TBS) Contracting Policy as well as HRDC 's contracting guidelines and administrative practices as they apply to services related to evaluation.

Logic Models

8.  http://www.ed.gov/teachtech/logicmodels.doc

Logic Models: A Tool for Telling Your Program's Performance Story , describes the Logic Model process in detail and how logic models can be used to develop and tell the performance story for a program.

9.  http://national.unitedway.org/outcomes/resources/mpo/

The manual Measuring Program Outcomes: A Practical Approach , is a good source for information on logic models and performance indicator development.

10.  http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf

See the W.K. Kellogg Foundation Logic Model Development Guide for a good overview on logic model development. It also provides information on variations and types of logic models.

11.  http://www.impactalliance.org/file_download.php/prevent+1.pdf?URL_ID=2744& filename=10196046740prevent_1.pdf&filetype=application%2Fpdf&filesize=646378&name=prevent+1.pdf&location=user-S/

Prevention Works! A Practitioner's Guide to Achieving Outcomes

12.  http://www.insites.org/documents/logmod.htm

Everything you wanted to know about logic models but were afraid to ask.

13.  http://www.calib.com/home/work_samples/files/logicmdl.pdf

This paper provides a description of logic models and discusses their uses in treatment services planning and evaluation.

14.  http://www.uwex.edu/ces/lmcourse/

Enhancing Program Performance with Logic Models , University of Wisconsin, an online self-study course and excellent resource.

15.  http://www.gse.harvard.edu/hfrp/content/pubs/onlinepubs/rrb/learning.pdf

Learning from logic models. An example of a family/school partnership

RMAF Links

16.  http://www.tbs-sct.gc.ca/cee/tools-outils/rmaf-cgrr/guide00-eng.asp

This is the August 2001 Guide for the Development of Results-based Management and Accountability Frameworks . It contains guidelines for developing the Profile, Logic Model, Performance Measurement Strategy, Evaluation Strategy, and Reporting Strategy.

17.  http://www.tbs-sct.gc.ca/cee/tools-outils/comp-acc00-eng.asp

This is a companion guide for the development of RMAFs for horizontal initiatives. Horizontal initiatives often need to integrate vertical and horizontal accountabilities, various resource pools, as well as a variety of departmental mandates, performance measurement strategies, and reporting structures. This guide is designed to complement the Guide for the Development of Results-based Management and Accountability Frameworks by addressing the unique challenges encountered when diverse organizations work together to achieve common objectives. While it does not provide answers to every question, it does provide guidance based on the most important lessons learned to date.

18.  http://www.tbs-sct.gc.ca/cee/pubs/guide/sarmaf-ascgrr-eng.asp

This document Guidance for Strategic Approach to RMAFs complements the August 2001 Guide for the Development of Results-based Management and Accountability Frameworks . The purpose of this document is to help managers tailor the development of the RMAF to specific circumstances, taking into account such factors as overall risk, program complexity, and reporting requirements so as to ensure that RMAFs remain responsive to evolving needs.

19.  http://www.tbs-sct.gc.ca/res_can/rc_e.html

This document is Results for Canadians: A Management Framework for the Government of Canada . It outlines what public service managers are expected to do to improve the efficiency and effectiveness of their programs. The RMAF is an important management tool in meeting the four main objectives of Results for Canadians : a citizen focus in all government activities; emphasis on values; achievement of results; and responsible use of public funds.

20.  http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=12257

This is the Treasury Board of Canada (TB) Policy on Transfer Payments , which formalizes the requirement of the RMAF as part of a TB submission involving transfer payments.

[1] .  See the TBS Web site at http://www.tbs-sct.gc.ca/cee/tools-outils/model-eng.asp .

[2] .  Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies , 2003.

Treasury Board of Canada Secretariat. Interim Evaluation of the Evaluation Policy , 2002.

[3] .  Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies , 2003.

[4] .  Treasury Board of Canada Secretariat. Preparing and Using Results-based Management and Accountability Frameworks. April 2004.

[5] .  Refer to the following Web site: http://www.tbs-sct.gc.ca/cee/ .

[6] .  Treasury Board of Canada Secretariat, Models for Evaluation and Performance Measurement for Small Agencies , 2003.

[7] .  For more information on the organizations mentioned, please refer to Appendix C.

[8] .  Smaller agencies may have only one level of management.

[9] .  Many small agencies submit an Annual Audit and Evaluation Plan. The template incorporates both audit and evaluation. Internal audit plans are required where internal audit priorities have been identified.

Acknowledgements

An Inter-Agency Steering Committee provided direction and context for the development of the Evaluation Guidebook for Small Agencies. We would like to thank the following participants:

Sylvie Charbonneau , Office of the Commissioner of Official Languages

Pierre Couturier , National Parole Board

Annette Ducharme , Canadian Forces Grievance Board

Yolaine Gauthier , Canadian Forces Grievance Board

Michael Glynn , Canadian Human Rights Tribunal

Janine Sherman , Canada School of Public Service

Robert Sauvé , Patented Medicine Prices Review Board

Greg Smith , Canadian Human Rights Tribunal

 Sharon Watts, Vice-President, Corporate Services & Adjudication, Hazardous Materials Information Review Commission

They reviewed drafts, and provided guidance and feedback to assist the consulting team to understand the context of small agencies.

We are most grateful to Glenn Crone, Kim Cronkwright and Zeljka Spasojevic of the Centre of Excellence for Evaluation, Treasury Board of Canada Secretariat, for their ongoing support.

Members of the Inter-Agency Steering Committee worked in collaboration with Zelda Yule, Sandy Moir, Celine Pinsent and Simon Roy of Goss Gilroy Inc.