Evaluation of the 2009 Policy on Evaluation

Performance of the Policy and Status of Policy Outcomes

Regarding the performance of the policy, the evaluation found the following:

1. In general, the evaluation needs of deputy heads and senior managers were well served under the 2009 Policy on Evaluation. Senior management was able to draw strategic insights to support higher-level decision making. At the same time, efforts to meet the policy's coverage requirements sometimes made evaluation units less able to respond to senior management's emerging needs.
2. The policy had an overall positive influence on meeting program managers' needs, and first-time evaluations of some programs were useful. However, program managers whose programs were evaluated as part of a broad program cluster or high-level Program Alignment Architecture entity sometimes found that their needs were not met as well as before 2009, when the program was evaluated on its own.
3. Central agencies found that evaluations were increasingly available, and they and departments increasingly used evaluations to inform expenditure management activities such as spending proposals (in particular program renewals) and spending reviews. At the same time, evaluations often did not meet central agencies' needs for information about program efficiency and economy.
4. Evaluation use under the 2009 policy was extensive, but use and impact could be improved by ensuring that the evaluations undertaken, and their timing, scope^{Footnote 1} and focus,^{Footnote 2} closely align with the needs of users.
5. The main use of evaluations was to support policy and program improvement.
6. The increased use of evaluations to support decision making was enabled by an observed government-wide culture shift toward valuing and using evaluations.
7. The factors that had the most evident positive influence on evaluation use in departments were policy elements related to governance and leadership of the evaluation function, whereas the factors that most evidently hindered evaluation use were those related to resources and timelines. 
8. Despite concerns about their capacity to meet all policy requirements, departments generally planned and expected to meet all requirements in the current five-year period.

Relevance and Impact of the Three Major Policy Requirements

Regarding the relevance and impact of the three major policy requirements (comprehensive coverage of direct program spending, five-year frequency for evaluations, and examination of the five core issues^{Footnote 3}), the evaluation found the following:

9. Challenges in implementing comprehensive coverage stemmed from the combined demands of the three key policy requirements (comprehensive coverage of direct program spending, five-year frequency for evaluations, and examination of the five core issues), along with the context of limited resources for conducting evaluations. The five-year frequency requirement appeared to be central to the implementation challenges in most departments.
10. Stakeholders at all levels recognized the benefits of comprehensive coverage for encompassing the needs of all evaluation users and for serving all purposes targeted by the policy. Nevertheless, there were clear situations where individual evaluations had low utility.
11. The five-year frequency for evaluations demonstrated benefits and drawbacks that varied according to the nature of programs and the needs of users. To optimize evaluation utility for a given program, a longer, shorter or adjustable frequency may be required.
12. In combination with the comprehensive coverage requirement, the five-year frequency limited the flexibility of evaluation units to respond to emerging or higher-priority information needs.
13. In general, the five core issues covered the appropriate range of issues and provided a consistent framework that allowed for comparability and analysis of evaluations within and across departments, as well as across time. However, the perceived pertinence of some core issues varied by evaluation and by type of evaluation user.
14. Longstanding inadequacies in the availability and quality of program performance measurement data and incompatibly structured financial data continued to limit evaluators in providing assessments of program effectiveness, efficiency (including cost-effectiveness) and economy. Central agencies and senior managers desired, in particular, more and better information on program efficiency and economy.

Approaches Used to Measure Policy Performance

Regarding the approaches used to measure policy performance, the evaluation found the following:

15. Mechanisms for measuring policy performance tracked the obvious uses of evaluations—those that were direct and more immediate—but did not capture the range of indirect, long-term or more strategic uses, and may not have given a robust perspective on the usefulness of evaluations.

Other Findings

The evaluation also found the following:

16. The requirements of the Policy on Evaluation and those of other forms of oversight and review, such as internal audit, created some overlap and burden.

Conclusions

The 2009 Policy on Evaluation helped the government-wide evaluation function play a more prominent role in supporting the Expenditure Management System. The policy also supported uses such as program and policy improvement, accountability and public reporting. Strong engagement from deputy heads and senior managers in the governance of the evaluation function promoted the utility of evaluations, and the evaluation needs of deputy heads, senior managers and central agencies were well served. In some cases, but not systematically across departments, evaluation functions produced horizontal analyses that contributed to useful cross-program learning, informing improvements both to the program evaluated and to other programs and to the organization as a whole. However, in assessing program effectiveness, efficiency and economy in evaluations, departmental functions were often limited by inadequacies in the availability and quality of performance measurement data and by incompatibly structured financial data.

The findings showed that while there was a general belief that all government spending should be evaluated periodically, there was also a widely held view that the potential for individual evaluations to be used should influence their conduct. Further, the policy requirements for evaluation timing and focus did not leave sufficient flexibility for departmental evaluation functions to fully reflect the needs of users in evaluation planning or to respond to emerging priorities. Evaluation needs were found to vary among different user groups (in particular, the needs of central agencies and departments were somewhat different). However, to fulfill coverage requirements within their resource constraints, departments sometimes chose evaluation strategies (for example, clustering programs for evaluation purposes) that were economical but that ultimately served a narrower range of users' needs. The lack of flexibility in the coverage and frequency requirements also made it challenging for departments to coordinate evaluation planning with other oversight functions in order to maximize the usefulness of evaluations and minimize program burden.

Recommendations

The evaluation recommends that when developing a renewed Policy on Evaluation for approval by the Treasury Board, the Treasury Board of Canada Secretariat should:

1. Reaffirm and build on the 2009 Policy on Evaluation's requirements for the governance and leadership of departmental evaluation functions, which demonstrated positive influences on evaluation use in departments.
2. Add flexibility to the core requirements of the 2009 Policy on Evaluation and require departments to identify and consider the needs of the range of evaluation user groups when determining how to periodically evaluate organizational spending (including the scope of programming or spending examined in individual evaluations), the timing of individual evaluations, and the issues to examine in individual evaluations.
3. Work with stakeholders in departments and central agencies to establish criteria to guide departmental planning processes so that all organizational spending is considered for evaluation according to the core issues; that the needs of the range of key evaluation users, both within and outside the department, are understood and used to drive planning decisions; that the planned activities of other oversight functions are taken into account; and that the rationale for choices related to evaluation coverage and to the scope, timing and issues addressed in individual evaluations is transparent in departmental evaluation plans.
4. Engage the Secretariat's policy centres that guide departments in the collection and structuring of performance measurement data and financial management data in order to develop an integrated approach to better support departmental evaluation functions in assessing program effectiveness, efficiency and economy.
5. Promote practices, within the Secretariat and departments, for undertaking regular, systematic cross-cutting analyses on a broad range of completed evaluations and using these analyses to support organizational learning and strategic decision making across programs and organizations. In this regard, the Treasury Board of Canada Secretariat should facilitate government-wide sharing of good practices for conducting and using cross-cutting analyses.

1.2.1 Evolution of the Federal Policy on Evaluation and the Context for Policy Renewal in 2009

The federal government has had central evaluation policies in place since 1977. Before the 2009 Policy on Evaluation, federal policies positioned the evaluation function to inform the management and improvement of programs, primarily from a program manager's perspective. In response to the increasing need for neutral, credible evidence on the value for money of government spending, the 2009 policy broadened the policy focus to include a more prominent role for the evaluation function in supporting the Expenditure Management System. Further, the policy situated the head of evaluation as a strategic advisor to the deputy head on the relevance and performance of departmental programs. Factors that led to refocusing the evaluation function included:

• The 2006 legislated requirement (Financial Administration Act, section 42.1) for all ongoing programs of grants and contributions to be reviewed for relevance and effectiveness every five years;
• The 2007 renewal of the Expenditure Management System, which was aligned with the Auditor General's recommendations of November 2006,^{Footnote 4} Budget 2006 commitments, and recommendations of the Standing Committee on Public Accounts^{Footnote 5} (adopted by the Standing Committee in February 2008^{Footnote 6}) on positioning evaluation to better support expenditure management decision making; and
• The advent of strategic and other spending reviews, which increased the demand for evaluations to provide information about program relevance and performance.

Coverage requirements existed in all previous federal evaluation policies and ranged from ensuring that all programs were evaluated periodically to considering, but not requiring, evaluation of all programs. The 2009 policy requires evaluations of all direct program spending.^{Footnote 7} Similarly, a frequency of evaluation was consistently specified in federal evaluation policies; however, this frequency varied from every three years to every six years. It is now every five years under the 2009 policy. Further, all federal evaluation policies included a set of issues to be addressed in evaluations. Since 1992 these issues have been consistent, requiring evaluations to examine the relevance, effectiveness and efficiency of programs. However, a notable change in 2009 was that core evaluation issues were no longer discretionary; the 2009 policy makes it mandatory for evaluations to address five core issues in order to meet coverage requirements.

For more information on the evolution of evaluation in the federal government, see Appendix A.

1.2.2 The International Context for Evaluation

Internationally, as governments undertook cost-cutting and cost-containment exercises in recent years, several countries expanded evaluation coverage and took steps to improve evaluation quality and to emphasize the use of evaluation in decision making. For example, the United Kingdom and the United States took steps to bolster the use of evaluation evidence in determining whether program spending is effective and provides value for money. The United Kingdom's guidance on evaluation for federal departments and agencies^{Footnote 8} indicates that with specific exceptions all policies, programs and projects should be comprehensively evaluated and that the risk of not evaluating is not knowing whether interventions are effective or delivering value for money. In the United States, evaluations are promoted as a means to “help the Administration determine how to spend taxpayer dollars effectively and efficiently—investing more in what works and less in what does not.”^{Footnote 9}

1.2.3 Introduction of the 2009 Policy on Evaluation

The current federal Policy on Evaluation was introduced on April 1, 2009, replacing the 2001 Evaluation Policy. The objective of the 2009 policy is to create a comprehensive and reliable base of evaluation evidence that is used to support policy and program improvement, expenditure management, Cabinet decision making and public reporting. To meet this objective, the policy strengthened requirements for evaluation coverage; for the assessment of the value for money of programs; for the quality and timeliness of evaluations and the neutrality of the function; and for evaluation capacity in departments. In its September 2010 report on Chapter 1, “Evaluating the Effectiveness of Programs,” of the Fall 2009 Report of the Auditor General of Canada, the Standing Committee on Public Accounts expressed support for the direction of the new policy by stating, “Effectiveness evaluations are very important for making good, informed decisions about program design and where to allocate resources. The Committee has long encouraged the development of effectiveness evaluation within the federal government and is pleased that the government has strengthened the requirements for evaluation.”

The 2009 policy and the associated directive and standard do the following:

• Establish evaluation as a function led by the deputy head, with a neutral departmental governance structure;
• Require comprehensive coverage of direct program spending^{Footnote 10} every five years;
• Articulate core issues of program relevance and performance that must be addressed in all evaluations (see Appendix A, Table 2);
• Introduce requirements for program managers to develop and implement ongoing performance measurement strategies;
• Set competency requirements for departmental heads of evaluation;
• Set quality standards for individual evaluations; and
• Require that evaluation reports be made easily available to Canadians in a timely manner.

The 2009 Policy on Evaluation and Directive on the Evaluation Function introduced flexibilities to help departments achieve comprehensive coverage.

Because of the significant changes introduced by the policy, and on the advice of an advisory committee of deputy heads^{Footnote 11} in 2008, a four-year phased implementation was adopted to give departments time to build their capacity for achieving comprehensive evaluation coverage. During this transition period, departments could use a risk-based approach to choose which components of direct program spending to evaluate. The transition period did not apply to ongoing programs of grants and contributions, which had to be evaluated every five years in accordance with the 2006 legal requirement.

Following the transition period, which ended on March 31, 2013, all direct program spending became subject to evaluation, and by March 31, 2018, the requirement for comprehensive coverage will need to be met for the first time. As stipulated in Annex A of the Directive on the Evaluation Function, departments could consider risk, program characteristics and other factors^{Footnote 12} when choosing evaluation approaches and when calibrating the methods and the level of effort applied to each evaluation. For example, calibrating an evaluation to expend less effort could entail:

• Selecting fewer and more targeted evaluation questions to examine the core value-for-money issues, or to focus on known problem areas of the program;
• Choosing a streamlined evaluation approach and a design with a shortened timeline;
• Calibrating the methods used and the level of effort by leveraging existing data instead of collecting new data whenever possible; by using smaller sample sizes; by using lower-cost interviewing methods (such as online or telephone instead of in-person, or clusters of in-person interviews to limit travel costs); or by conducting fewer case studies.

Departments could also adjust the scope of evaluations by grouping programs rather than evaluating programs individually.

3.1.1 Baseline Results for Policy Outcomes (evaluation questions 5 and 6)

1. Finding: In general, the evaluation needs of deputy heads and senior managers were well served under the 2009 Policy on Evaluation. Senior management was able to draw strategic insights to support higher-level decision making. At the same time, efforts to meet the policy's coverage requirements sometimes made evaluation units less able to respond to senior management's emerging needs.

Deputy heads who were consulted indicated that under the 2009 policy, their departments produced a good base of evaluations and had the capacity to use them. The performance case studies showed that evaluations met a range of deputy head needs, such as:

• Providing evidence of program effectiveness to support renewal decisions;
• Showing where program outcomes were not likely to be achieved; and
• Revealing related findings across a set of evaluations to support strategic decision making—for example, to identify an area of generalized concern.

The performance case studies showed that evaluations supported strategic decision making by delivering a more comprehensive perspective on the performance of departmental programming than that produced under the 2001 Evaluation Policy. The trend toward evaluating clusters of programs or larger entities, along with the convergence of all evaluations at departmental evaluation committees (or executive committees), enabled senior managers to recognize patterns across multiple evaluations and programs. Consultation evidence showed that some departments produced cross-cutting analyses from multiple evaluations of programs targeting common outcomes. In one case study, the insights drawn from across several evaluations led one deputy head to request a special review of a type of funding arrangement; in another case study, such insights influenced resource reallocation among a set of high-priority, horizontal activities. Senior executives on departmental evaluation committees also applied evaluation lessons from another branch to programs in their own branch.

Survey evidence showed that program managers felt that senior managers were well served under the 2009 policy. Three quarters of program managers surveyed (75%) reported that it was somewhat useful (38%) or very useful (37%)^{Footnote 17} for senior management (deputy ministers, associate deputy ministers and assistant deputy ministers) to have evaluations of their programs every five years, as required by the policy. Further, a majority of program managers (ranging from 68% to 87%) felt that each of the five required core issues was somewhat useful or very useful to senior management.

At the same time, the performance case studies showed that in some departments efforts to meet the policy's coverage requirements made evaluation units less able to respond to senior management's needs for special studies, reviews or specific evaluations on emerging issues. As shown in Figure 2, most evaluators surveyed reported that the proportion of time spent on evaluation activities directly related to the policy increased after the introduction of the 2009 policy, while the proportion of time spent on other evaluations, reviews, studies or research activities decreased.

Figure 2 - Text version

The figure shows the change in the proportion of time that evaluators, on average, have spent on various activities since the introduction of the 2009 Policy on Evaluation. The mean values for each of six activities are plotted on a vertical scale. The values from zero to one represent activities where an increased proportion of time was spent, and the values from zero to negative one represent activities where a decreased proportion of time was spent. The mean values are based on evaluators' survey responses, using a three-point scale where −1.00 means a decrease in the proportion of time spent, zero means the proportion of time spent stayed the same, and +1.00 means an increase in the proportion of time spent.

Two activities had mean values showing an increase, on average across evaluators, in the proportion of time spent on evaluation activities directly related to the policy (mean value of 0.59) and on corporate administrative activities (mean value of 0.25). Two activities had mean values that showed only a slight decrease, on average across evaluators, in the proportion of time spent on other activities (mean value of −0.04) and on developing or supporting the development of performance measurement strategies (mean value of −0.06). The remaining two activities had mean values that showed a decrease, on average across evaluators, in the proportion of time spent on other evaluations (mean value of −0.19) and on reviews, other studies, and other research activities (mean value of −0.22). The sample size (number of evaluators reporting) ranged from 41 to 82 depending on the activity.

2. Finding: The policy had an overall positive influence on meeting program managers' needs, and first-time evaluations of some programs were useful. However, program managers whose programs were evaluated as part of a broad program cluster or high-level Program Alignment Architecture entity sometimes found that their needs were not met as well as before 2009, when the program was evaluated on its own.

Program managers surveyed felt that evaluations were useful for a variety of purposes. In particular, 81% of program managers rated evaluations as somewhat useful (25%) or very useful (56%) for supporting program improvement, and 79% of program managers rated evaluations as somewhat useful (33%) or very useful (46%) for program and policy development. Performance case studies showed that some managers of programs that were evaluated for the first time gained insights that led to improvements. Further, evidence from case studies suggested that these programs may never have been evaluated were it not for the policy's comprehensive coverage requirement.

At the same time, other evidence from the Implementation Review showed that program managers did not always find their programs reflected in the findings of evaluations whose scopes aligned with Program Alignment Architecture entities (a common scope for evaluations)^{Footnote 18} or encompassed clusters of programs. In these cases, the evaluations did not equip them with sufficiently detailed information to make program improvements. Performance case studies illustrated that some departments addressed this issue by designing these evaluations to produce findings and conclusions at multiple levels.

In terms of assisting program managers as they developed performance measurement strategies, the policy's influence was mixed. Based on the findings from the Implementation Review, the demands of the policy's comprehensive coverage and frequency requirements may have made some evaluation units too busy to support program managers in developing their strategies to the same extent that they once had.^{Footnote 19} However, some evaluation units emphasized their support to program managers in this regard, to ensure that performance measurement would support future evaluations. The survey of evaluators showed that following the introduction of the 2009 policy, a slightly larger proportion (38%) of evaluators decreased the time spent supporting the development of performance measurement strategies compared with the proportion (32%) that increased the time spent. The balance of evaluators indicated that the time spent stayed the same. Despite some evaluators spending less time supporting the development of performance measurement strategies, however, 90% of program managers surveyed in 2014 indicated that their programs had a performance measurement strategy in place. Among those programs with a performance measurement strategy in place, 93% of program managers had consulted their departmental evaluation function during its development.

3. Finding: Central agencies found that evaluations were increasingly available, and they and departments increasingly used evaluations to inform expenditure management activities such as spending proposals (in particular program renewals) and spending reviews. At the same time, evaluations often did not meet central agencies' needs for information on program efficiency and economy.

Central agency analysts generally viewed evaluations as a key source of program information and often consulted them first in their analysis of spending proposals.^{Footnote 20} The performance case studies and stakeholder consultations showed that Secretariat analysts generally encouraged departmental use of evaluation findings in Treasury Board submissions, consistently required evaluation information for funding renewals in particular, and had recommended that departments not seek funding approval without a recent evaluation. Secretariat analysts reported that before the 2009 policy, evaluations were not always available to support submissions, but that today, if draft submissions do not provide evaluation information, they often seek such information from departments. In addition, when evaluation findings are negative, analysts seek departments' confirmation of corrective actions.

Several lines of evidence^{Footnote 21} indicated that evaluations were more widely used as a source of supporting information for Treasury Board submissions and, to a lesser extent, for Memoranda to Cabinet. Through the Capacity Assessment Survey, 96% of large organizations reported in 2012–13 that they used all or almost all relevant evaluations to inform Treasury Board submissions, and 78% reported that they used all or almost all relevant evaluations to inform Memoranda to Cabinet. These findings compare with those of the survey in 2008–09, prior to the 2009 policy, where 74% of large organizations reported that they almost always^{Footnote 22} considered evaluation results in Treasury Board submissions and 51% reported that they almost always considered them in Memoranda to Cabinet. Most large organizations established a formal process to include evaluation information in submissions (79%) and Memoranda to Cabinet (65%) in 2013–14. Evaluations were commonly used to support renewals of existing spending—notably, for ongoing programs of grants and contributions.^{Footnote 23} Central agency analysts typically used evaluation information to inform their advice to Treasury Board ministers, and some noted that they periodically received questions from Cabinet about evaluation results.

Based on performance and application case studies and on stakeholder consultations, the use and utility of evaluations, in particular at central agencies, was affected by how well evaluation timing aligned with the timing of spending decisions. Central agencies sometimes noted that evaluations arrived too late to meaningfully inform renewal decisions. For example, it was noted that key discussions on renewal are often held a year or more before a Treasury Board submission is developed. In those cases, an evaluation that is finished only in time to be appended to the submission can be seen as too late to support central agency analysts. It should also be noted, however, that within departments the draft evaluation reports are often available to program managers much earlier, allowing them to take advantage of the findings and knowledge generated, even if the report has not been fully approved.

Based on the Implementation Review and on case study consultations with central agency representatives, evaluation utility was also affected by how well the evaluation scope matched the unit of expenditure that was subject to a decision. When analyzing and advising on Memoranda to Cabinet or Treasury Board submissions, central agencies' information needs tended to be project-specific or program-specific—that is, specific to the unit of funding being renewed. When evaluations had a broad scope, such as a Program Alignment Architecture Program, they may not have provided sufficiently granular information. Case studies and stakeholder consultations showed that evaluations often did not meet central agencies' needs for information on program efficiency and economy—for example, because evaluators' analysis of program cost-effectiveness was limited by the incompatible structure of financial information. Central agencies also wanted better evidence on program alternatives in government-wide and cross-jurisdictional comparisons. A key risk associated with evaluations not meeting the needs of central agencies for this information is that their analysis and advice to ministers regarding departmental proposals may not be as well supported by neutral evidence as they could be.

Medium to high use^{Footnote 24} of evaluations in spending reviews (for example, strategic reviews) was enabled by the increased availability and relevance of evaluations,^{Footnote 25} and most departmental evaluation committee members and senior managers who were consulted, including deputy heads consulted in 2014, reported a high degree of evaluation utility for this purpose. Almost two thirds of evaluators surveyed reported positive impacts on the utility of evaluations for spending reviews because of the policy's comprehensive coverage requirement (63%) and core issues requirement (62%). Deputy heads consulted by the Treasury Board of Canada Secretariat in 2010 reported that strategic reviews raised the profile of the evaluation function by requiring departments to systematically address fundamental issues of program relevance. The majority of program managers surveyed (63%) reported that evaluations were somewhat useful (44%) or very useful (19%) for spending reviews. Most Secretariat program analysts^{Footnote 26} reported that evaluations supported their analysis during spending reviews and noted that in many departments these reviews increased the demand for evaluations and that the attention paid by senior executives helped evaluation demonstrate its value.

4. Finding: Evaluation use under the 2009 policy was extensive, but use and impact could be improved by ensuring that the evaluations undertaken, and their timing, scope and focus, closely align with the needs of users.

Before the introduction of the 2009 policy, weaknesses in evaluation use had been documented.^{Footnote 27} After 2009, the Secretariat's monitoring and reporting showed extensive evaluation use during the policy's transition period. In the 2012–13 Capacity Assessment Survey, large departments reported high implementation rates of management responses and action plans; of the 901 management action plan items that were scheduled for completion in 2012–13, 53% were fully implemented by the end of the fiscal year and 21% were partially implemented. In addition, Management Accountability Framework assessment ratings documented extensive evaluation use; more than 96% of large departments were rated acceptable or strong for evaluation use in 2013–14, compared with 77% of large departments in 2007–08 and 78% of large departments in 2008–09.

When consulted in fall 2010, many deputy heads^{Footnote 28} stated that evaluation was making a solid contribution to decision making, but a number of deputy heads felt that more could be achieved. When consulted in 2014, deputy heads acknowledged the usefulness of evaluations for program and policy improvement and development, strategic reviews and as a means of capturing corporate memory, while also noting that sometimes there were issues with the timing, focus and scale (level of intensity) of evaluations.

In case studies, evaluations were seen as most useful when they were timely, provided new information, and did not merely re-identify problems in program delivery that users already knew about. Evaluations were seen as less useful when they could not lead to organizational learning, when there was no decision to inform, or when no action could be taken. Central agencies as well as program managers, heads of evaluation, and evaluators noted situations where evaluations were less useful, including when their timing, scope, focus, report length and level of analytical rigour did not align with decision makers' needs or interests. Key risks associated with producing evaluations of low utility include spending evaluation resources inefficiently, rather than allocating the resources to evaluations that would be more useful and, more broadly, undermining the perceived value of the evaluation function as a whole.

5. Finding: The main use of evaluations was to support policy and program improvement.

Analysis by the Centre of Excellence for Evaluation showed that 75% of evaluation reports^{Footnote 29} completed in 2010–11 included recommendations to improve program processes. Similarly, across the evaluations examined in the performance case studies, most recommendations pertained to program improvement, and all were actually used for this purpose. Performance case studies also showed that almost all of the evaluations examined were used for program improvement and that some evaluations resulted in improvements to a larger suite of programs than the one evaluated.^{Footnote 30} However, performance case studies also showed that some evaluations were not used for improvement purposes when internal decisions left no opportunity for recommendations to be implemented—for example, when the program was eliminated or completely reorganized. The evidence showed that in the course of examining the relevance and performance of programs, evaluations sometimes contributed to operational efficiencies, but that these efficiencies were rarely in the form of direct cost savings.

Among a list of possible evaluation uses, program managers rated program and policy improvement as the one for which evaluations had been the most useful; 81% of program managers rated evaluations as somewhat useful (25%) or very useful (56%) for this purpose. Overall, both evaluators and program managers reported that the policy had a positive or neutral impact on the utility of evaluations for informing program and policy improvement; 56% of evaluators and 35% of program managers reported that evaluation utility increased, whereas only a small proportion (9% and 5% respectively) reported that utility had decreased. The balance (35% of evaluators and 60% of program managers) stated that utility had remained the same.

The evidence showed that the use of evaluation for accountability and public reporting increased, and program managers and evaluators reported that the policy had a positive impact on the utility of evaluations for these purposes.^{Footnote 31} The 2000 December Report of the Auditor General of Canada noted that performance reports to Parliament made too little use of evaluation findings; by 2011, the annual Capacity Assessment Survey showed that a high proportion of large organizations (89%) considered 80% or more of their evaluations when preparing their annual Departmental Performance Reports. In 2013–14, the annual Capacity Assessment Survey showed that 91% of large organizations had formal processes to ensure that evaluation inputs were considered in parliamentary reporting.

Performance case studies showed that organizations usually posted evaluation reports, including management responses and action plans, on their websites, although in some cases posting occurred long after the evaluation was completed. In the performance case studies, a small number of stakeholders suggested that part of the lag between functional completion of evaluation work and the approval and posting of reports was due to internal discussion when preparing reports for public posting, which led in some cases to less critical reporting.

6. Finding: The increased use of evaluations to support decision making was enabled by an observed government-wide culture shift toward valuing and using evaluations.

Key conditions had to be established for the policy to achieve its intended outcomes for evaluation use.^{Footnote 32} According to the theory of change developed for the Policy on Evaluation (see Appendix D), the policy was intended to drive a cultural shift in departments to increase the perceived value of, confidence in, and use of evaluation. This culture shift was evidenced by:

• The shift in stakeholder^{Footnote 33} perceptions, from viewing evaluations as an oversight burden for programs, to perceiving their value and the skills available in the evaluation unit;^{Footnote 34}
• Increased departmental dialogue about evaluation since 2009, as reported by 71% of surveyed evaluators and 46% of surveyed program managers;^{Footnote 35} and
• High rates of implementing recommendations, encouraged by the establishment of systems for tracking the implementation of evaluation recommendations, which 97% of all large departments reported having in place by 2013–14.^{Footnote 36}

Deputy heads' more general interest in evaluation likely contributed strongly to the observed outcomes.^{Footnote 37} Management Accountability Framework assessment ratings of departmental evaluation functions ensured management attention and were partially responsible for bringing a higher profile to evaluation. An analysis of Management Accountability Framework assessment ratings from  2006–07 to 2011–12 showed an  increasing trend in evaluation use, as well as coverage, governance and support, and quality of evaluation reports.

3.1.2 Factors Influencing Outcome Achievement (evaluation question 8)

7. Finding: The factors that had the most evident positive influence on evaluation use in departments were policy elements related to governance and leadership of the evaluation function, whereas the factors that most evidently hindered evaluation use were those related to resources and timelines.

Across all lines of evidence, the engagement of senior leaders in departmental evaluation functions appeared to have the clearest positive influence on evaluation use. This influence was attributed, at least in part, to policy requirements related to governance and leadership (for example, the defined roles and responsibilities of deputy heads, departmental evaluation committees and heads of evaluation, and the head of evaluation's unencumbered access to the deputy head), and to a government-wide climate that emphasized results-based management and evidence-informed decision making. Increased senior management engagement led to greater implementation of action plans and enhanced the overall visibility of the evaluation function. The presence of deputy heads on most departmental evaluation committees ensured that evaluation findings were taken seriously, and scrutiny from a more senior executive level may have increased evaluation quality.

As shown in Figure 3, evaluators^{Footnote 38} reported that the engagement of departmental evaluation committees, senior management and program managers, as well as the availability of qualified internal evaluation staff, had positive influences on achieving policy outcomes. These factors were more often reported by evaluators to have had a positive impact on use and utility than the policy requirements for comprehensive coverage of direct program spending, five-year frequency of evaluations, and examination of the five core issues.

Evaluators surveyed reported that the most evident negative influences on evaluation utility came from the timelines for evaluation projects, from the budgets for evaluation projects, and from spending reviews. Although the Treasury Board of Canada Secretariat did not track changes in the budgets of individual evaluation projects, government-wide financial resources for evaluation were 8% lower in 2012–13 than in 2008–09, before the policy's introduction, despite an initial increase in the first year of implementing the 2009 policy. The performance case studies provided further insights on the influence that spending reviews sometimes had as an external factor on evaluation utility: in some cases, evaluations could not be used for program improvement purposes because the program spending was significantly changed or discontinued. Program managers and senior managers also noted that the contribution of evaluations to decision making could be affected by the availability of other forms of oversight and review, such as spending reviews, especially when some of the input information was common.

Figure 3 - Text version

The figure shows the extent to which evaluators, on average, felt that various factors had positive or negative impacts on the use of evaluations. The mean values are based on evaluators' survey responses using a three-point scale, where −1.00 means the factor had a negative impact on evaluation use, zero means the factor had no impact on evaluation use, and +1.00 means the factor had a positive impact on evaluation use.

Seven factors had mean values indicating positive impacts on evaluation use: (1) engagement of departmental evaluation committee (mean value of 0.68), (2) support from senior management (mean value of 0.60), (3) engagement of program managers (mean value of 0.53), (4) availability of qualified internal staff (mean value of 0.49), (5) five core issues requirement (mean value of 0.27),  (6) comprehensive coverage requirement (mean value of 0.26), and (7) frequency requirement (mean value of 0.08). Three factors had mean values indicating negative impacts on evaluation use: (1) budgets (mean value of −0.10), (2) spending review (mean value of −0.12), and (3) timelines (mean value of −0.28). The sample size (number of evaluators reporting) ranged from 98 to 141 depending on the factor.

Management Accountability Framework assessments and ratings had a significant influence on policy implementation and results. On the positive side, they drew senior management's attention to evaluation and helped raise the profile of the function. On the negative side, they promoted risk-averse behaviour that may have limited departments' use of the policy's flexibilities. As noted in the Implementation Review, although flexibilities existed to calibrate evaluation effort when addressing core issues, they were not fully exploited owing to concerns that Management Accountability Framework assessment ratings would be adversely affected. This finding was corroborated by the performance case studies and stakeholder consultations, including consultations with deputy heads, who noted that while further policy flexibilities may be needed, existing flexibilities had not been fully exploited.

Another factor that influenced the policy's impact on the conduct and use of evaluations was the amount of grants and contributions spending administered by individual departments. In departments where the amount was large, the impact of the policy was small because of the pre-existing Financial Administration Act requirement (section 42.1) for comprehensive five-year coverage of this spending. Stakeholders noted that organizations with a large amount of grants and contributions spending had evaluation functions that were well established and producing useful evaluations before 2009.

3.1.3 Sustainability of Outcomes (evaluation question 7)

8. Finding: Despite concerns about their capacity to meet all policy requirements, departments generally planned and expected to meet all requirements in the current five-year period.

A comparison of Capacity Assessment Survey data collected before 2009 and in 2012–13 showed that on average, large organizations increased the human resources they devoted to their evaluation functions by 10% but decreased financial resources by 8%. As mentioned earlier, to expand evaluation coverage with these resources, departments used various strategies. Evaluators^{Footnote 39} reported that the most effective strategies were calibrating evaluation scope and approach according to program risks, aligning an evaluation's scope with Program Alignment Architecture units, clustering related programs, and increasing the use of internal staff to conduct evaluations.^{Footnote 40} The trend toward conducting evaluations with broad scopes, however, affected the utility of evaluations; for example, some program managers found that the information available to inform program improvements was less detailed.

When consulted for the Implementation Review, heads of evaluation highlighted resources as the main factor constraining them in meeting the coverage requirements, and they were concerned about their ability to meet the requirements in a meaningful manner with the available resources. Although heads of evaluation and other stakeholders^{Footnote 41} expressed concerns about the function's capacity to achieve and maintain comprehensive coverage over five years, in most cases it appeared that departments could manage their capacity to meet the requirements. Three quarters of evaluators^{Footnote 42} (74%) reported that the utility of the evaluation function could be maintained with current resources, and in a subsequent question, more than one third (36%) stated that utility could be increased.

Despite the potential for achieving full evaluation coverage within current capacity, several lines of evidence^{Footnote 43} showed that greater flexibility is needed in applying policy requirements related to coverage, timing, scope and focus for evaluations to be more responsive to the information needs of various users. Flexibility is further discussed in subsequent sections of this report.

3.2.1 Comprehensive Coverage (evaluation question 3)

10. Finding: Stakeholders at all levels recognized the benefits of comprehensive coverage for encompassing the needs of all evaluation users and for serving all purposes targeted by the policy. Nevertheless, there were clear situations where individual evaluations had low utility.

A literature review showed that six^{Footnote 47} of nine countries, as well as the Development Assistance Committee of the Organisation for Economic Co-operation and Development, recommended comprehensive evaluation coverage. Alternative approaches used in other jurisdictions involved targeting evaluation coverage by considering a variety of factors such as decision-making needs, priorities, program maturity, program type, self-assessment results and important government-wide themes.

Deputy heads who were consulted in 2014 held mixed opinions about the appropriateness of the comprehensive coverage requirement; many were in favour, some emphatically, whereas a smaller number favoured a more risk-based model. However, those who favoured the comprehensive coverage model often stated that the five-year period for achieving comprehensive coverage posed challenges. The reasons that deputy heads supported comprehensive coverage included the following:

• It makes sense to evaluate all programming.
• It ensures disciplined oversight, ensures accountability and keeps issues from being “swept under the carpet.”
• It leads to evaluation scopes that are often at a higher level and that support decision making on important “units of account.”

Central agency respondents generally favoured comprehensive coverage because it demonstrates good governance to scrutinize government's use of all taxpayers' contributions. Case studies showed that without the comprehensive coverage requirement some low-priority or low-risk programming would have been excluded from evaluation. However, central agency respondents felt it appropriate to periodically evaluate both low-risk and long-horizon programs, as these evaluations could be important in formulating advice to Treasury Board ministers on program renewals and for ensuring public accountability. The performance case studies demonstrated that there was value in evaluating some low-risk programs. At the same time, central agency respondents also recognized that evaluating certain programs was impractical. When central agencies had concerns about the comprehensive coverage requirement, the concerns often related to conducting evaluations of low utility—for example, evaluations that could not lead to actionable recommendations.

Heads of evaluation who were consulted generally agreed on several benefits that they observed from the comprehensive coverage requirement since 2009. In particular, they reported that it made evaluations available to inform decision making (notably, for programs where no past evaluations existed) and to inform processes such as departmental performance reporting and adjustments to Program Alignment Architectures. They also reported that comprehensive coverage increased the profile of the function, validating and empowering the function within departments, while increasing its workload and sometimes its resources.^{Footnote 48}

The views of other stakeholder groups, notably program managers and evaluators,^{Footnote 49} were more divided. Among those stakeholders that supported comprehensive coverage, there was a general view that in principle, all spending should be evaluated periodically. These stakeholders reported that benefits from the comprehensive coverage requirement included insights on programs never-before or long-ago evaluated, and a strategic view of performance that cut across related departmental programs—for example, to identify redundancies and synergies. Evaluators, in particular, generally reported that comprehensive coverage had increased the utility of evaluations for all major uses targeted by the policy.^{Footnote 50} In addition, case studies showed that the requirement had a profound effect in some departments, especially those with little grants and contributions spending, because many evaluations were conducted on entities that had never been evaluated before 2009. These evaluations sometimes produced valuable findings that led to program improvements. In some cases, however, stakeholders indicated that if an evaluation had not been required, the organization would likely have conducted a different type of study to address its needs.

Across all stakeholder groups, those that did not support comprehensive coverage generally questioned using resources to evaluate programs where there was little perceived need for the information (for example, for low-risk programs, or where other sources of information existed), where the evaluation might have no utility or where recommendations would be non-actionable. For example, the case studies showed that some stakeholders questioned the merits of applying the comprehensive coverage requirement to assessed contributions because evaluations would have no impact on what Canada is required to spend on these programs. However, case studies showed that existing evaluations of assessed contributions, which focused on the effectiveness of Canada's membership effort (for example, in deriving benefits for Canada or in influencing organizational policies) or on the coordination between the various departments and agencies engaged with the international organization) had demonstrated value. Assessed contributions are also subject to the evaluation requirements of the Financial Administration Act (section 42.1) and the Policy on Transfer Payments.

A small number of deputy heads^{Footnote 51} and heads of evaluation and a minority of program managers and evaluators suggested returning to the former risk-based model for evaluation planning—that is, using risk considerations to decide whether to evaluate programs. Several caveats accompanied this suggestion, including that risk-based approaches are not a panacea; that clear guidance from the centre would be needed to ensure consistency in the assessment of risk; that materiality alone should not be used to define risk; and that at least one full round of comprehensive coverage could be needed to provide assurance that program risk levels are accurately assessed. 

A number of alternative approaches to coverage were suggested in case studies and consultations, including:

• Comprehensive coverage over a longer time frame (longer frequency);
• Risk-based coverage, with calibration of individual evaluations;
• Targeted evaluations (evaluations that have narrower scopes or that address specific issues or themes);
• Evaluations focused on departmental priorities and interests; and
• Evaluations where there is the most potential for information gain.

In contrast, some central agency respondents and heads of evaluation recommended that the policy be expanded to require the evaluation of types of program spending that it currently does not evaluate. Specific program types mentioned were sunsetting or time-limited programs, internal services and statutory programs (beyond the administrative aspects alone). Central agencies in particular noted that sunsetting and time-limited programs are sometimes renewed and that evaluations are helpful for informing the renewal process.

3.2.2 The Five-year Frequency for Evaluations (evaluation question 2)

11. Finding: The five-year frequency for evaluations demonstrated benefits and drawbacks that varied according to the nature of programs and the needs of users. To optimize evaluation utility for a given program, a longer, shorter or adjustable frequency may be required.

12. Finding: In combination with the comprehensive coverage requirement, the five-year frequency limited the flexibility of evaluation units to respond to emerging or higher priority information needs.

Within and across stakeholder groups, there were diverse perspectives on the appropriateness and utility of the policy's five-year frequency requirement. Although the five-year frequency for evaluations was perceived to be mechanistic and insensitive to management needs and preferences, all stakeholder groups noted both positive and negative aspects.

Compared with other stakeholder groups, central agencies more strongly supported the five-year frequency requirement. This support may be explained by the nature of central agency work, which often involves advising the Treasury Board on funding renewals that usually occur every five years and that can be informed by evaluations.

Heads of evaluation viewed the positives and negatives associated with the five-year frequency requirement with equal emphasis, whereas program managers^{Footnote 52} and evaluators^{Footnote 53} were more likely to favour alternative frequencies. For example, 63% of evaluators surveyed thought that a different frequency than “every five years” should be used for evaluating programs, excluding ongoing programs of grants and contributions. It should also be noted that the experiences of departments in implementing the 2009 policy may have varied widely owing to the nature of their programs, their evaluation history and the experiences of their deputy heads and heads of evaluation.^{Footnote 54}

Positive aspects of the five-year frequency requirement expressed by a majority of stakeholder groups included the following:

Availability and broad support to decision making:

The requirement created and ensured a constant foundation of information about all direct program spending to support all areas of decision making—for example, expenditure management needs, spending reviews and accountability.

Recent information:

The requirement ensured that evaluation information would be no older than five years. Central agencies ^{Footnote 55} and senior executives said that they discounted evaluations older than five years and considered that those conducted in the past two or three years were the most useful—for example, for informing Treasury Board submissions.

Overall strengthening of the evaluation function:

The requirement supported a culture of evaluation, performance measurement and learning; increased the profile of evaluation functions; enabled evaluation planning; and increased engagement of program managers, including in performance measurement.

Negative aspects of the five-year frequency requirement that were identified by most stakeholder groups included the following:

Evaluation timing not always aligned with needs:

The requirement led to mechanistic timing that did not always produce an evaluation at the time it would be useful—for example, when program context was stable; when there was no specific decision to inform; when recent information was available from other sources or review activities; when recent significant program changes made evaluation premature; or when an earlier timing would provide useful formative information on a new program.

Strained evaluation capacity and reduced responsiveness:

The requirement lessened the responsiveness of some evaluation functions to new and emerging information needs, or limited their capacity to focus on the real problems, higher-value evaluation projects or strategic activities, as resources were fully committed to delivering on departmental evaluation plans. In some cases, this requirement led to prioritizing evaluations of low risk or low interest to deputy heads. It was also noted that the length of time to complete evaluations affects how easily evaluations can fit into a five-year cycle. ^{Footnote 56}

Effects of broad evaluation scopes:

As described earlier, the requirement resulted in strategies to evaluate programs in clusters or as larger entities. This approach reduced the utility of evaluations for some users but resulted in increased utility for others.

Poor use of resources:

As reported by the Auditor General in spring 2013, ^{Footnote 57} the requirement limited the ability of departments “…to put their evaluation resources to the best use.” The majority of stakeholder groups expressed similar concerns.

Lower perceived value of the evaluation function:

Some reported a combined negative effect of the above (timing not aligned with needs, low responsiveness and poor use of resources) on the overall perceived value of the evaluation function.

In addition, heads of evaluation and evaluators reported a negative aspect of the five-year requirement to be the burden on programs from undergoing repeated evaluations. Despite this burden, three quarters (77%) of program managers surveyed whose programs had been evaluated since 2009 felt that it was somewhat useful (41%) or very useful (36%) to have an evaluation of their program every five years, to support their decision making and program management needs.

Although a regular, cyclical evaluation approach was often preferred over an ad hoc approach, the dominant view was that the five-year frequency was not appropriate in some situations. Many favoured the notion of having greater flexibility in evaluation frequency so that evaluations could be timed to ensure maximum utility, including delaying one evaluation in order to facilitate another. Where the five-year frequency was inappropriate, stakeholders proposed many alternative triggers and considerations for evaluation timing. Based on evidence from multiple sources,^{Footnote 58} alternative approaches fell into three broad categories:

• Adopting a longer evaluation frequency, such as seven, eight or even 10 years;
• Maintaining a five-year frequency while allowing departments to target evaluations on specific programs areas or smaller program components, rather than on full programs; or
• Determining evaluation timing based on needs.

Internationally, evaluation policies most commonly link evaluation timing to decision making and reporting processes or requirements. This evaluation found that the increased alignment of evaluation timing with Treasury Board submissions contributed to the use and utility of evaluations.

Across all stakeholder groups, a proportion of stakeholders indicated that the importance of the program and the level of risk should be the main factors for determining evaluation frequency and that the life cycle or maturity of the program should also be considered. To help align evaluation timing with needs, several factors were identified across most stakeholder groups, including:

• Risk;
• Program factors—for example, materiality or importance, and type of funding;
• Program maturity or program life cycle;
• Decision-making needs;
• Known program issues;
• Planned restructuring; and
• Availability of performance measurement data or information from other sources.

Based on case study evidence, departments favoured the idea of choosing the appropriate evaluation frequency for their programs themselves, as this would help them meet ongoing and emerging needs (including their own, and those of central agencies); coordinate evaluation timing with other oversight activities (for example, audits and reviews); and minimize program burden. Some evidence suggested that central agencies (notably, the Treasury Board of Canada Secretariat) could play a role in informing departmental evaluation planning decisions on frequency or timing to ensure that expenditure management needs are met; however, there were no suggestions on how to put this into practice.

3.2.3 Core issues (evaluation question 1)

13. Finding: In general, the five core issues covered the appropriate range of issues and provided a consistent framework that allowed for comparability and analysis of evaluations within and across departments, as well as across time. However, the perceived pertinence of some core issues varied by evaluation and by type of evaluation user.

Specifying core issues is consistent with practices in the other jurisdictions examined. Further, the core issues identified in the 2009 Directive on the Evaluation Function are largely consistent with the categories of issues addressed in other jurisdictions (relevance, effectiveness and efficiency), although the breakdown of issues into components (for example, the breakdown of relevance into program need, government priority and government role) is less common.

Each of the policy's five core issues has one or more interested users within departments and central agencies and at the political and public levels. In terms of evaluation utility, central agencies generally benefited more than departments from the consistent application of all core issues.^{Footnote 59} Documents from the Treasury Board of Canada Secretariat indicated that the five core issues were intended to focus evaluations on examining program value for money. In the analysis of case studies, evaluators, program managers and senior management (including departmental evaluation committee members) perceived the core issues to be appropriate and useful. Program managers, in particular, perceived the core issues to be the “right ones.” Central agencies generally favoured consistent application of all five core issues across evaluations because they aligned with the types of questions ministers ask and the questions central agencies ask in playing their challenge function role. Moreover, the consistent framework of evaluation issues could support central agencies' emerging uses for performance information—for example, horizontal analytics or syntheses of evaluation information. No issue was consistently identified as missing from the set of core issues.

The core issues requirement had both positive and negative impacts on the utility of evaluations.^{Footnote 60}

The following are examples of the positive impacts:

• It helped support program improvement.
• It provided a consistent framework to focus evaluations and allow program comparisons—for example, systematic issues, overlaps and opportunities for synergy. This consistency supported the work of Secretariat analysts and prevented information gaps.^{Footnote 61}
• It focused evaluations on providing the information needed to support expenditure management (for example, supporting Treasury Board submissions and Memoranda to Cabinet), spending and program reviews, needs of decision makers and other evaluation users, and good management.
• It helped ensure that programs were not avoiding suspected performance issues.
• It highlighted problems with performance measurement data.

As shown in Figure 4, program managers surveyed generally viewed all core issues as somewhat useful or very useful.^{Footnote 62}

Figure 4 - Text version

The figure shows the percentage of the 115 program managers surveyed that rated each of the five core issues as somewhat useful or very useful in supporting the decision-making needs of senior managers and the proportion that rated each core issue as somewhat useful or very useful in supporting their own decision-making needs. For core issue 1, “continued need,” 73% of program managers rated the issue as somewhat useful or very useful in supporting the decision-making needs of senior managers and 80% rated the issue as somewhat useful or very useful in supporting their own decision-making needs. For core issue 2, “alignment with priorities,” 75% of program managers rated the issue as somewhat useful or very useful in supporting the decision-making needs of senior managers and 79% rated the issue as somewhat useful or very useful in supporting their own decision-making needs. For core issue 3, “alignment with federal roles,” 73% of program managers rated the issue as somewhat useful or very useful in supporting the decision-making needs of senior managers and 78% rated the issue as somewhat useful or very useful in supporting their own decision-making needs. For core issue 4, “achievement of expected outcomes,” 87% of program managers rated the issue as somewhat useful or very useful in supporting the decision-making needs of senior managers and 87% rated the issue as somewhat useful or very useful in supporting their own decision-making needs. For core issue 5, “demonstration of efficiency and economy,” 68% of program managers rated the issue as somewhat useful or very useful in supporting the decision-making needs of senior managers and 75% rated the issue as somewhat useful or very useful in supporting their own decision-making needs.

As shown in Figure 5, the majority of evaluators surveyed reported that the core issues requirement had the most positive impacts on the utility of evaluations for informing program improvement, and for accountability and public reporting.

Figure 5 - Text version

The figure shows the percentage of evaluators surveyed that reported that the core issues requirement had a positive impact on the utility of evaluations for each of three purposes. Of the 153 evaluators surveyed, 84% reported a positive impact on the utility of evaluations for program and policy improvement; 77% reported a positive impact on the utility of evaluations for accountability purposes; and 72% reported a positive impact on the utility of evaluations for public reporting.

Negative impacts associated with the core issues^{Footnote 63} related to the limited flexibility that departments reported for designing evaluations to reflect user needs and priorities (including those of deputy heads^{Footnote 64} and program managers) and program characteristics and context.^{Footnote 65}

The general support for addressing core issues was accompanied by observations that periodically and in specific circumstances, certain core issues might not be applicable in a given evaluation. Most stakeholder groups agreed that the following three core issues were important or essential and should be addressed in all evaluations:

• Demonstration of efficiency and economy, which examines a program's use of resources in achieving outcomes and was reported by some stakeholders as being of highest importance for parliamentarians;
• Achievement of expected outcomes, which examines program effectiveness and is of great interest to program managers, senior managers and deputy heads; and
• Continued need for the program, which was reported to provide useful information for both program managers and senior managers.

The first two are core performance issues and were of greatest concern to deputy heads, program managers and other users of evaluation,^{Footnote 66} and the third is a core relevance issue.

As well, as shown in Figure 6, most evaluators felt that these same core issues should be addressed in all evaluations (note that the survey divided the “demonstration of efficiency and economy” core issue into two parts, to measure evaluators’ perceptions of each).

Figure 6 - Text version

For each of the five core issues, the figure shows the percentage of evaluators surveyed that felt the issue should be addressed in all evaluations. Of note, the survey divided the “demonstration of efficiency and economy” core issue into two parts. From the highest percentage to the lowest percentage, 99% of the 153 evaluators felt that the “achievement of expected outcomes” core issue should be addressed in all evaluations; 85% of evaluators felt that “efficiency” (part of the “demonstration of efficiency and economy” core issue) should be addressed in all evaluations; 75% of evaluators felt that the “continued need for program” core issue should be addressed in all evaluations; 73% of evaluators felt that “economy” (part of the “demonstration of efficiency and economy” core issue) should be addressed in all evaluations; 37% of evaluators felt that the “alignment with federal roles” core issue should be addressed in all evaluations; and 36% of evaluators felt that the “alignment with priorities” core issue should be addressed in all evaluations.

Across the stakeholder groups, some stakeholders stated that two of the three relevance issues—alignment with federal roles and responsibilities, and alignment with government priorities—applied only to a subset of programs or only under certain conditions.^{Footnote 67} Other stakeholders stated that all three core relevance issues were essential or the most important, especially for informing senior management. Senior management and central agency respondents indicated that the relevance issues helped address questions in the expenditure management review framework and, in particular, the question of affordability.

When alignment with federal roles and responsibilities and alignment with government priorities were perceived to be inapplicable,^{Footnote 68} stakeholders felt that addressing them would be an inefficient use of evaluation resources and would be of little interest to senior managers. However, relevance issues were not the only issues that were perceived to be inapplicable for specific evaluations; for example, the applicability of each core issue could depend on factors related to the program and its environment. Case studies indicated that in both large and small organizations, the core issues requirement may have led to less useful evaluations that were more complex, time-consuming and resource-intensive than they could have been.

Deputy heads consulted in 2014 called for increased flexibility in applying core evaluation issues, to allow organizations to focus on areas of interest and to produce concise reports to inform decision making. At the same time, deputy heads recognized that increased flexibility would require senior management to become more engaged in evaluation design.

Most stakeholder groups favoured adding flexibility to the requirement for addressing all five core issues.^{Footnote 69} Although consultation findings showed that central agencies generally favoured the consistent application of all five core issues across evaluations, they, as well as program managers and evaluators, agreed that flexibility should be considered. Heads of evaluation desired flexibilities that ranged from not addressing a core issue, to minimizing the examination of a core issue, or to addressing one or more core issues less frequently—for example, in every second evaluation of the program or every ten years.

Among the small departments examined in the case studies, the core issues were perceived to increase the burden of evaluations without a corresponding internal demand for the information, leading these organizations to exercise their option to not address them all.^{Footnote 70}

It was generally noted by stakeholders that section 42.1 of the Financial Administration Act calls for a review of relevance and effectiveness every five years, without mandating the five core issues. The added requirement of both the Policy on Transfer Payments and the Policy on Evaluation for these evaluations to address all five core issues was viewed by evaluation units as imposing an added burden. At the same time, central agency analysts, who are key users of evaluations for advising the Treasury Board on spending proposals such as renewals for programs of grants and contributions, generally supported the need for addressing all five core issues.

The core issues were generally perceived to be very broad in nature. Although no other issue was consistently identified as one that should be added to the existing set of core issues, deputy heads and senior managers sometimes wanted individual evaluations to address additional issues.^{Footnote 71} As the Directive on the Evaluation Function explicitly permits, evaluations often addressed additional issues to meet departmental users' needs.^{Footnote 72} For example, 61% of evaluators surveyed indicated that issues other than the five core issues were addressed in evaluations at least half the time. When evaluators were asked which issues, other than the five core issues, were considered for inclusion in evaluations since 2009, the most frequent issue related to program design and delivery. Consultations with evaluators and central agencies indicated that design and delivery issues had sometimes been addressed under one or both of the two core performance issues; for example, the Directive on the Evaluation Function's description of the core issue “achievement of expected outcomes,” explicitly refers to the assessment of program design as part of this issue. Other non-core issues were identified,^{Footnote 73} which stakeholders stated could usually^{Footnote 74} or sometimes^{Footnote 75} be subsumed within the core issues.

In applying their available evaluation resources, evaluation units sometimes had to choose between addressing a non-core issue to satisfy their deputy head's information needs and complying with the core issues requirement. Consultations indicated that they tended to consider the deputy head's needs to be more important.

As noted in the Implementation Review, although flexibilities existed for addressing core issues with less evaluation effort or for providing a rationale for not addressing an issue, these flexibilities may not have been fully communicated to, or used by, departments. This finding may be explained, in part, by their concerns that Management Accountability Framework assessment ratings would be adversely affected.^{Footnote 76} Deputy heads mentioned this rationale in consultations, noting that while further flexibilities may be needed for the core issues requirement, existing flexibilities had not been fully used. Although stakeholders were aware of flexibilities, some perceived them as largely theoretical and saw no real opportunity for discretion when applying the core issues in specific evaluations, and in particular, no opportunity to justify that an issue should not be addressed. According to many stakeholders, the effort required to demonstrate that an evaluation does not need to address certain relevance issues (alignment with federal roles and responsibilities and alignment with federal government priorities) could be equivalent to the effort required to actually evaluate them. Many organizations did, however, calibrate their efforts^{Footnote 77} on these relevance questions.

14. Finding: Longstanding inadequacies in the availability and quality of program performance measurement data and incompatibly structured financial data continued to limit evaluators in providing assessments of program effectiveness, efficiency (including cost-effectiveness) and economy. Central agencies and senior managers desired, in particular, more and better information on program efficiency and economy.

Although it was not a specific focus of the evaluation, the impact of performance measurement data on the success of evaluation was a common theme that emerged. The 2009 Fall Report of the Auditor General of Canada noted that many evaluations did not adequately assess program effectiveness because analysis was limited by inadequate performance measurement data. This finding was echoed in the 2013 Spring Report of the Auditor General of Canada, which noted that in 14 of 20 evaluations sampled across three departments, weaknesses in program performance measurement data continued to limit evaluators in assessing program effectiveness and often required them to rely on more subjective data or to collect additional data to fill gaps. Although the Implementation Review found that the situation had improved to some extent in 2012–13, about 50% of organizations continued to report challenges related to insufficient performance measurement data. Although the survey of program managers found that 90% of programs had a performance measurement strategy in place, the same survey found that roughly 50% of the indicators were only somewhat useful, not very useful or not at all useful to evaluators. The ongoing nature of this challenge was further reflected in the Secretariat's 2013–14 Capacity Assessment Survey, where 30 of 33 evaluations approved between April and September 2013 experienced significant limitations owing to insufficient performance measurement data—despite a reported 70% rate of implementation of performance measurement strategies. The Directive on the Evaluation Function's requirement for evaluation units to annually produce reports for departmental evaluation committees on the state of performance measurement helped bring attention to these problems in a number of departments.

Case studies and stakeholder consultations showed a general consensus that despite the core issue requiring evaluations to examine program efficiency and economy, evaluations rarely met the information needs of users in this regard. For example, inadequate financial information limited the ability of evaluators to analyze the cost-effectiveness of programs. Although the available financial data were not inaccurate or incomplete, they were not structured according to the entity being evaluated—that is, resources were not linked to program activities or outcomes. There was concern among stakeholders that the Secretariat's guidance documents had provided too few concrete examples of how to assess efficiency and economy and did not address the prevailing structural challenges of financial data.

A 2006 Legislated Requirement for Evaluation

In June 2006, the President of the Treasury Board commissioned an independent blue ribbon panel, “to recommend measures to make the delivery of grants and contributions programs more efficient while ensuring greater accountability.” Following the release of the panel's report,^{Footnote 90} the Federal Accountability Act of December 2006 amended the Financial Administration Act to require that all ongoing programs of grants and contributions be reviewed for relevance and effectiveness every five years. The 2008 Policy on Transfer Payments and the 2009 Policy on Evaluation later defined these reviews as evaluations, and the 2009 Policy on Evaluation reflected the legal requirement in its own coverage requirements. Before 2006, the requirement for reviewing or evaluating ongoing programs of grants and contributions was contained only in the Policy on Transfer Payments, where it continues to appear today.

A Shift in Emphasis for Evaluation to Support Expenditure Management

The renewal of the Expenditure Management System in 2007 placed greater emphasis on using evaluation as an input to expenditure decisions. In accordance with Budget 2006 commitments, the renewed Expenditure Management System is based on the following key principles:

• Government programs should focus on results and value for money;
• Government programs must be consistent with federal responsibilities; and
• Programs that no longer serve the purpose for which they were created should be eliminated.

The system's renewal addressed the November 2006 Auditor General's recommendations that expenditure decisions be anchored by reliable information on program performance. In February 2008, the Standing Committee on Public Accounts^{Footnote 91} recommended that “the Treasury Board Secretariat reinforce the importance of evaluation by adding program evaluation as a key requirement in the Expenditure Management System.”

In the renewed Expenditure Management System, evaluation is positioned as an important source of neutral, credible evidence about program value for money, to provide support to expenditure decisions for each of the three pillars of the system, as follows:

Managing for Results:

Evaluations are used by departments on an ongoing basis to manage for results—that is, to determine whether programs are achieving expected results and to inform decisions about continuing, amending or terminating program spending.

Upfront Discipline:

Evaluation evidence is used in new spending proposals (such as in the Memorandum to Cabinet process) to help compare proposed spending with existing or past program results.

Ongoing Assessment:

Evaluations provide input to spending reviews (comprehensive or targeted) to support analyses of whether programs are effective and efficient, are focused on results, are providing value for taxpayers' money, and are aligned with government priorities.

Heads of Evaluation

The Implementation Review found that as departments implemented the policy, there was a shift toward designating heads of evaluation at higher executive levels, enabling them to play more strategic advisory roles in departmental decision making. Almost two thirds of heads of evaluation (64%) were designated at the EX-3 and EX-4 levels in 2012–13, compared with less than one third (30%) in 2009–10.

The review found that over the same four-year period, the pairing of the evaluation function with other functions became a common practice in both large and small organizations, as did the practice of combining the head of evaluation role with other leadership roles. In 2012–13, roughly three quarters of heads of evaluation fulfilled leadership roles in two or more other functions; in particular, 61% of heads of evaluation fulfilled the role of chief audit executive, compared with 39% in 2009–10. The prevalence of paired audit and evaluation units also increased, from 41% of departments in 2009–10 to 67% of departments in 2011–12. This trend toward pairing evaluation with other functions and combining the head of evaluation role with other roles appeared to have been driven primarily by the policy requirement for heads of evaluation to have unencumbered access to the deputy head. This trend was possibly enhanced by organizational restructuring following government-wide cost-containment exercises.

Departmental Evaluation Committees

The Implementation Review found that by the end of the policy's transition period, all large departments had established departmental evaluation committees, the large majority being chaired by deputy heads. Increasingly, evaluation committee members were senior decision makers representing all or most organizational divisions. In most cases, committee members were senior executives who were also members of the senior executive committee. In one third of departments in 2011–12, the membership composition of the departmental evaluation committee matched that of the senior executive committee. Departmental evaluation committees were also increasingly involved in activities such as tracking individual evaluation recommendations and advising on resources required by the function.

Departmental Evaluation Plans

The 2009 policy requires that deputy heads annually approve a rolling five-year departmental evaluation plan that aligns with and supports the Management, Resources and Results Structure, that supports the Expenditure Management System, and that evaluates all ongoing programs of grants and contributions, as required by section 42.1 of the Financial Administration Act. The evaluation plan is a vehicle for communicating within departments and with the Treasury Board of Canada Secretariat, especially with analysts involved in expenditure management processes.

The Implementation Review found that since 2009, more than 90% of large departments and agencies had annually submitted plans to the Secretariat. A majority of departments developed their plans according to the Secretariat's guidance, by broadly consulting with program areas and discussing performance measurement needs for supporting evaluation.

Resource Allocation

At the time the 2009 policy was introduced, the Treasury Board of Canada Secretariat had projected that departments would need to increase investment in the evaluation function to achieve and sustain comprehensive evaluation coverage every five years. Despite a temporary increase in resources in 2009–10, the Secretariat's monitoring showed that government-wide financial resources for the function were stable, at about $60 million annually until 2011–12.

Although financial resources for the evaluation function were relatively stable during the policy's transition period, the number of full-time equivalents dedicated to the function in 2011–12 was somewhat higher (500^{Footnote 98}) than in 2009–10 (474^{Footnote 99}). This increase in human resources appeared to be achieved by decreasing budgets for professional services and reallocating these funds to salaries.

Human Resources Capacity Building

After the Secretariat's regular engagement with departmental evaluation functions up to 2011–12 and a survey of federal evaluators to determine their professional development needs, it was concluded that introductory-level evaluation training was no longer a high-priority need. However, when heads of evaluation were consulted a year later for the Implementation Review, they indicated that some evaluators who had recently joined the function lacked evaluation expertise, causing novice-level training to re-emerge as a short-term need.

In addition, a number of heads of evaluation and directors of evaluation felt that the need for more experienced evaluators had increased under the 2009 policy because departments were choosing more complex evaluation designs. Specifically, they noted that the clustering of programs for evaluation purposes had increased the demand for expertise and experience typically held by senior evaluators.

Strategies Used to Implement Coverage Requirements

The Implementation Review found that departments used one or more of the following strategies to expand their evaluation coverage within their budgeted resources.

Case Studies

Stakeholder Consultations

Stakeholder consultations were conducted as part of ongoing policy dialogue, as well as for informing the evaluation of the Policy on Evaluation and the review of the Policy on Management, Resources and Results Structures. The Deputy Assistant Secretary of the Expenditure Management Sector at the Treasury Board of Canada Secretariat conducted semi-structured interviews with 15 deputy heads, associate deputy heads and assistant deputy heads from both large and small departments, and with six other key informants, including senior officials from central agencies and former federal public servants.

Surveys

Data Analysis

External consultants used SPSS software to perform descriptive and inferential statistical analyses on policy monitoring data previously collected by the Centre of Excellence for Evaluation as well as on the data from the surveys of program managers, evaluation managers and evaluators. Monitoring data included the following:

• Data from annual Capacity Assessment Surveys of departmental evaluation functions, from 2004–05 through 2013–14; and
• Management Accountability Framework assessment ratings of departmental evaluation functions from 2007–08 to 2011–12, including overall ratings and ratings for each of four sub-criteria (Quality of Evaluation Reports, Governance and Support of the Evaluation Function, Evaluation Coverage, and Use of Evaluation), along with overall evaluation ratings for 2006–07.

The focus of the Capacity Assessment Survey changed annually, and the Management Accountability Framework indicators evolved over the years. To conduct longitudinal analyses to show changes that were potentially attributable to the policy, only stable indicators from these two data sources were used.

Open-ended data from the surveys of program managers and the federal evaluation community were coded by the internal evaluation team, and then the coded data were transferred to the external team.

Process Mapping

A process map (see Appendix D) was developed by the external consultants, through the review of documents and case study data. The map provided an overview of how the evaluation function operates in departments, including processes for planning, conducting and using evaluations.

Document Review

The internal evaluation team conducted a document review to inform questions on the appropriateness of comprehensive coverage and core issues, on the approaches used to measure the performance of the policy, on baseline results for policy outcomes, and on factors affecting the achievement of outcomes. Approximately 60 internal and external documents were reviewed, including Auditor General reports to Parliament, reports of the Standing Committee on Public Accounts, publications of the Treasury Board of Canada Secretariat, and other documents.

Literature Review

The internal evaluation team conducted a literature review to compare and contrast the evaluation policies and practices of other jurisdictions with those of Canada, and in particular, those related to evaluation coverage and frequency and to the core issues addressed in evaluations. The review synthesized the most recently available literature of the following types:

• Official publications from governments (national and sub-national level) and international organizations (agency level), including evaluation policies, guidance documents, evaluation plans, a government constitution, legislation, information reports, evaluation standards, competencies, glossaries and other reports;
• Web pages;
• Academic articles and working papers; and
• Information from presentations given to the Treasury Board of Canada Secretariat.

A sample of nine countries and three international organizations was selected according to several criteria as described below:

• Relevance and comparability to the Canadian context: the United States, the United Kingdom and Australia;
• Comparability of evaluation activity levels, status of evaluation policies, or other reasons: Switzerland, Japan and India—as informed by a 2013 EvalPartners' report, Mapping the Status of National Evaluation Policies;
• For continuity, countries examined during the 2013 Implementation Review of the Policy on Evaluation: South Africa, Mexico Spain, the United States, the United Kingdom and Australia;
• Evaluation agencies or groups from three international organizations with established evaluation policies or guidance: the United Nations Evaluation Group, the Development Assistance Committee of the Organisation for Economic Co-operation and Development, and the World Bank Independent Evaluation Group; and
• Availability and reliability of online sources and documents in English and French.