Evaluation Guidebook for Small Agencies

• Previous page
• Table of contents
• Complete text
• Next page

Section One: Introduction

This guidebook is a reference tool designed to assist small agencies to develop their capacity to plan, implement and manage evaluations. The guidebook will help small agencies meet the requirements of the Treasury Board of Canada (TB) Evaluation Policy .

This "how-to" guidebook presents effective practices for planning and managing evaluation projects, provides strategies for fostering the use of information from evaluations in strategic decision making within small agencies, and provides examples of key documents used in the evaluation process.

This guidebook builds on the work that was previously conducted in 2003–04 with respect to developing appropriate models of evaluation and performance measurement in small agencies. [1] The guidebook represents the next step in assisting small agencies to build their capacity to plan, implement, and manage evaluations.

1.1  Purpose of the Guidebook

This guidebook should help you with the following:

• to understand the relevance and role of evaluation in supporting effective control and performance measurement regimes within your agency;
• to plan, design, implement, and manage an evaluation appropriate for your agency; and
• to communicate and foster the use of evaluation results within your agency.

1.2  Who Should Use This Guidebook?

This guidebook is designed for small agencies that require assistance in implementing the Treasury Board of Canada (TB) Evaluation Policy requirements. The primary target audience is the personnel who are responsible for evaluation within small agencies. For those readers who have limited experience in the field of evaluation, the guidebook is designed to provide detailed information on most aspects of evaluation and includes helpful material such as context, definitions, checklists, management tips and references for more detailed information on certain subjects. For those readers who have more experience with evaluations, they may find various parts or sections of the guidebook more useful than others, such as the checklists. It is also recognized that small agencies' needs are varied with respect to evaluation and information needs. The guidebook attempts to address these diverse needs.

1.3  Small Agency Context

Most small agencies were created by the federal government to independently carry out specific mandates. For example, some agencies render impartial decisions (e.g., tribunals). Other agencies perform a facilitation or intermediary role. Others have regulatory and inspection mandates. See Appendix A for an overview of different types of small agencies.

With respect to evaluation, previous studies [2] have noted that many small agencies face challenges in developing evaluation and performance measurement functions for their agencies. These challenges include having adequate resources and capacity to develop these functions, having to adapt models from larger organizations, and integrating the function with the day-to-day business of the agencies.

Small agencies as a group also have unique challenges and characteristics when compared with medium and large federal departments. One difference is that small agencies often have one or two main business lines in comparison with medium and large departments that often have multiple business lines containing many programs, initiatives, and policies. Another difference is that, in comparison with medium and large departments, small agencies often have more limited flexibility with respect to financial resources. Given these and other differences, many aspects of the models in meeting accountability and performance requirements in medium and large departments are not applicable to small agencies.

When considering what type of evaluation and performance measurement models are required in the small agency community, it is important to realize that, although the community shares commonalities when compared with medium and large departments, there is also a great amount of diversity within the small agency community itself. The small agency community in the federal government is diverse on various dimensions such as organizational structure, relationship with larger departments, nature of work, and organization size. These dimensions, in addition to others, contribute to the type and nature of information that agencies need for decision making and ensuring accountability within their organizations.

A 2003 Treasury Board of Canada Secretariat (TBS) project entitled Models for Evaluation and Performance Measurement for Small Agencies [3] presented three models of evaluation and performance measurement functions within federal small agencies. Two key considerations in developing the models were the types of management decision making and the types of information needed to make decisions within agencies.

These models are as follows:

4  Model A – Straightforward Information Needs

4  Model B – Blend of Straightforward and Complex Information Needs

4  Model C – Complex Information Needs

For a more detailed description of the models and how to classify your own agency, please go to the TBS Web site at http://www.tbs-sct.gc.ca/cee/tools-outils/model-eng.asp .

1.4  Structure of the Guidebook

• Section 1 contains this introduction.
• Section 2 presents an overview of evaluation, relevant concepts and context.
• Section 3 describes how to establish an evaluation function and provides a brief review of evaluation policy requirements. It also presents strategies for building internal evaluation capacity.
• Section 4 presents an overview of preparing for and conducting evaluations. Topics include results-based management and accountability frameworks, evaluation design, data collection methods, analysis, and report writing.
• Section 5 contains an overview of managing external resources. This section includes topics such as preparing terms of reference, contracting options, methods for selecting contractors, and best practices for managing evaluation consultants.
• Section 6 provides an overview of strategies for communicating and using evaluation findings within your organization.

In addition, there are a number of appendices that contain more detailed information that can be used as references.

• Appendix A contains a description of the different types of small agencies.
• Appendix B contains information on horizontal initiatives and their evaluation.
• Appendix C contains information on seeking external advice and support on evaluation.
• Appendix D contains the Expenditure Review Committee's seven tests.
• Appendix E contains more detailed "how-to" information for planning and conducting evaluations.
• Appendix F contains a terms of reference template.
• Appendix G contains a glossary of relevant evaluation terms.
• Appendix H contains links to evaluation Web sites.

1.5  How to Use this Guidebook

The guidebook is designed to be used by readers in a number of different ways. For those readers who are relatively new to the field of evaluation, you may want to work your way systematically through the guide. For more experienced readers, it may be more useful to go directly to the specific section that you need. Regardless of how you choose to use the guide, you should be aware of the symbols used throughout the text as indicated below.

If you require a more detailed guide with respect to planning or implementing an evaluation, please refer to Appendix E.

Key References

Treasury Board of Canada Secretariat. Models for Evaluation and Performance Measurement for Small Agencies: Summary Report , 2003.

Treasury Board of Canada Secretariat. Independence vs. Partnering – Finding the Right Balance: A Dialogue on Values and Ethical Decision-Making in Small Agencies , 2003.

Treasury Board of Canada Secretariat. Evaluation Policy and Standards , 2001.

• Previous page
• Table of contents
• Complete text
• Next page