Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Evaluation of the Research and Policy Initiatives Assistance Program - Treasury Board of Canada Secretariat

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.





Evaluation of the
Research and Policy Initiatives Assistance Program





Table of Contents

Executive Summary

1.0 Introduction

1.1 Background

2.0 Evaluation Methodology

2.1 Evaluation Issues and Questions

2.2 Methodology

2.3 Limitations of Methodology

3.0 Evaluation Findings

3.1 Program Relevance

3.2 Program Success

3.3 Program Cost-Effectiveness and Alternatives

4.0 Summary of Conclusions and Recommendations

Appendix A - List RPIA-Approved Projects

Appendix B - TBS RPIA Evaluation Matrix

Appendix C - TBS RPIA Program Logic Model

Appendix D - Interview Guide



Executive Summary

This report presents the findings and conclusions of the evaluation of the Research and Policy Initiatives Assistance Program (RPIA Program) for the Treasury Board of Canada Secretariat (the Secretariat). The evaluation was undertaken to respond to the requirements for an evaluation of the program for its renewal, in accordance with the Policy on Transfer Payments and Treasury Board requirements. The Secretariat's Internal Audit and Evaluation Bureau undertook a stewardship function over the course of the evaluation. The evaluation findings will also serve program managers by providing information to improve program delivery and performance. The evaluation focussed on three main areas: relevance, success, and cost-effectiveness. 

Description of the RPIA Program

The RPIA Program at the Secretariat is a class-contribution program used for small, ad hoc initiatives that contribute to Secretariat objectives but do not deliver a clear good or service. The program can provide up to 50 per cent of eligible costs of individual approved projects, up to a total of $500,000 paid out over the life of the initiative. There is no separate funding base for the program. The RPIA Program functions less as a program in and of itself and more as a mechanism for existing programs to deliver their objectives. As such, funding is provided from existing budgets. Use of this authority requires the recommendation of a branch head, the sign-off of the department's senior full-time financial officer, and the approval of the Secretary of the Treasury Board.

Over the five-year period of the evaluation (2003–04 to 2007–08), a total of $928,674 was approved through the RPIA Program. 

Evaluation methodology and limitations

The evaluation methodology was designed to ensure that the cost of the evaluation was commensurate with the relatively small amount of funding provided through this program. As a result, the evaluation integrated the use of only two lines of evidence: file review and key stakeholder interviews.

As with any evaluation, there are limitations to the evaluation methodologies. First, the interviews and file review did not include all projects that received funding through the RPIA Program. However, efforts were made to select those projects that accounted for the largest percentage of program spending over the evaluation period. Government Consulting Services (GCS) considers the files selected to be a good sample and believe they represent Secretariat projects under the RPIA Program. 

A second limitation of the evaluation relates to a lack of quantitative information. The interviews and file review conducted for the evaluation are qualitative sources of information. Little quantitative information was available about the RPIA Program, particularly with regard to benefits or outcomes.

Program relevance

RPIA Program projects are aligned with program and departmental objectives. Projects that were reviewed related to conducting research and holding events on topics relevant to the objectives of the Secretariat and the program. Topics examined through RPIA Program projects that were reviewed included public service governance; inter-jurisdictional service delivery; design of public policy, such as the design and administration of programs of grants, contributions, and other forms of financial support; and promotion of approaches to technology, information, and service. 

The RPIA Program is meeting needs of sectors by providing funding for projects that otherwise may not have been possible. Interviewees noted that recipient organizations would not have carried out the funded activities in the absence of the RPIA Program; as well, the RPIA Program's funding has generated discussion on areas of public sector management, where traditionally there has been little dialogue. According to interviewees, the Secretariat would not have had access to the benefits of the projects if funding had not been provided. Finally, interviewees noted, that to achieve the objectives of some projects, an independent, neutral, third-party organization was required. These interviewees noted that the Secretariat could not have undertaken the projects themselves, as they would have been viewed as biased. As well, the projects funded under the RPIA Program typically brought multiple organizations to work together. Without the RPIA Program project, stakeholders would have worked more in isolation. 

Program success

The file review conducted for this evaluation highlighted a lack of evidence in Secretariat files regarding the benefits of the projects. For the most part, the only information available from the file review, with regard to benefits of the projects, was information about anticipated benefits rather than actual benefits realized. The RPIA Program does not have a requirement to provide any kind of post-project summary for the project file. Thus, actual benefits to the Secretariat as a result of these projects are not routinely documented. 

Recommendation No. 1: The program area should implement a mechanism to ensure that the impact of the program can be measured with respect to actual benefits rather than anticipated benefits. For example, a post-project report could be a requirement for all RPIA Program projects. This could be as simple as a one-page template that is filled out for each project.

Interviewees believe that the RPIA Program has contributed to enhancing policies or programs, increasing knowledge of emerging policy issues and building capacity for analysis. 

According to interviewees, the RPIA Program has helped to enhance policies or programs for which Secretariat sectors are responsible. Interviewees also agreed that the RPIA Program has helped to increase knowledge of emerging policy issues, e.g., public service retention. They noted that RPIA Program projects brought together people from various groups, and these people were involved in discussions and shared information, which increased the knowledge base. Finally, interviewees agreed that the RPIA Program has helped to build capacity for analysis by increasing the information base of sectors and increasing analytical capacity as a result of RPIA Program projects. 

One interviewee found it hard to comment on project success, indicating that the benefits of the project were more directed to other departments rather than to the Secretariat. 

Program cost-effectiveness and alternatives

There appear to be few options for alternatives to the RPIA Program. Interviewees pointed to insufficient resources (i.e., human and financial) for the Secretariat to be able to conduct the work in-house. As well, in some cases an independent third party is needed to successfully achieve the objectives of the project. Issues were identified with regard to the only other potential alternative suggested (i.e., contracting the work), mainly due to the fact that there are often multiple partners for RPIA Program projects with shared benefits, and such projects do not necessarily have a clear output or deliverable. 

Interviewees believe the program is cost-effective because of the benefits of partnering with other organizations, i.e., leveraging. In most cases, the Secretariat provided only a portion of the funding to carry out projects. For the files reviewed, the Secretariat contributed anywhere from 8 to 44 per cent of the total cost of the projects.

Because of the nature of the program and its relatively small size, program efficiency was not seen as a relevant issue to focus on in this evaluation. However, some information was gathered during interviews that related to efficiency. Interviewees were generally positive in their views on program administration. Most interviewees noted that the application process was straightforward and easy and that approvals were timely. 

A few suggestions were identified to improve the program. First, interviewees believe that the program needs to be marketed better within the department. 

Recommendation No. 2: The Secretariat should increase the visibility of the RPIA Program within the department and ensure that employees are aware of the program and know whom to contact for more information.

In addition, the file review undertaken for this evaluation highlighted the fact that not all project-related documents, i.e., the submission and recommendation forms, were available through the RPIA Program manager, and, since post-project reports are not required to be on file, the actual benefits of projects are not routinely documented.

Recommendation No. 3: Project documentation should be collected and retained more systematically and stored in a central location.



1.0 Introduction

This report presents the findings, conclusions, and recommendations of the evaluation of the Research and Policy Initiatives Assistance Program (RPIA Program) for the Treasury Board of Canada Secretariat (the Secretariat). The evaluation was undertaken to respond to the requirements for an evaluation of the program for its renewal, in accordance with the Policy on Transfer Payments and Treasury Board requirements. The evaluation findings will also serve program managers by providing information to improve program delivery and performance. The Secretariat/Finance Canada (FIN) engaged Government Consulting Services (GCS) to undertake the evaluation.

The objective of the study was to evaluate the program in terms of relevance, success, cost-effectiveness, and alternatives. The research for this evaluation was conducted between December 2008 and January 2009.

The evaluation report is organized as follows:

  • Section 1 presents a description of the RPIA Program.
  • Section 2 presents the methodology for the evaluation.
  • Section 3 presents findings by evaluation issue and question.
  • Section 4 presents the conclusions and recommendations.

1.1 Background

The RPIA Program at the Secretariat is a class-contribution program used for small, ad hoc initiatives that contribute to Secretariat objectives but do not deliver a clear good or service. The program can provide up to 50 per cent of eligible costs of individual approved projects, up to a total of $500,000 paid out over the life of the initiative. There is no separate funding base for the program. The RPIA Program functions less as a program in and of itself and more as a mechanism for existing programs to deliver their objectives. As such, funding is provided from existing budgets. Use of this authority requires the recommendation of a branch head, the sign-off of the department's senior full-time financial officer, and the approval of the Secretary of the Treasury Board.

The terms and conditions of the program were set to expire on March 31, 2008, but were extended for one year under the terms of the Policy on Transfer Payments in effect at the time. A summative evaluation of the program is required to renew it for another five-year period. 

1.1.1 Budget

Over the five-year period of the evaluation (2003–04 to 2007–08), a total of $928,674 was approved through the RPIA Program, $423,674 of which was provided directly by the Secretariat. This represents an average of $185,735 per year allocated for projects. The breakdown of spending by year is provided in Table 1.

Table 1:  Secretariat RPIA Program - Approved Funding
Fiscal Year Funding Amount Provided by the Secretariat Funding Amount Provided by Other Government Departments Total RPIA Program Funding Provided
2003–04 $ 110,868 $ 300,000 $ 410,868
2004–05 $ 83,932 $ 175,000 $ 258,932
2005–06 $ 87,500 - $ 87,500
2006–07 $ 49,874 - $ 49,874
2007–08 $ 91,500 $ 30,000 $ 121,500
Total $ 423,674 $ 505,000 $ 928,674

The RPIA Program funding was allocated to a total of 11 projects that supported a wide range of activities. Note that some of these were multi-year projects. Research and the creation of, or participation in, councils were the activities most frequently undertaken with the projects (four and three projects, respectively). Funds were also used to hold workshops and symposia (two projects), to hold conferences (one project), and to develop strategies (one project). See Appendix A for a full list of RPIA Program–approved projects during the five-year evaluation period.



2.0 Evaluation Methodology

2.1 Evaluation issues and questions

This evaluation was undertaken to respond to the evaluation issues and questions, as articulated in the evaluation matrix developed for the evaluation and included in the Memorandum of Understanding between the Secretariat/FIN and GCS.

The evaluation questions examined are shown in Table 2. The complete evaluation matrix is presented in Appendix B. Appendix C contains the logic model for the RPIA Program.

Table 2:  Evaluation Questions
Evaluation Issue Evaluation Question
Relevance
  • Is the RPIA Program meeting the needs of programs within the Secretariat?
Success
  • Has the program achieved the intermediate outcomes identified in the logic model?
Cost-effectiveness
  • What leverage did the program generate. i.e., proportion of project funding provided by other organizations?

2.2 Methodology

The evaluation methodology was designed to ensure that the cost of the evaluation was commensurate with the relatively small amount of funding provided through this program. As a result, the evaluation integrated the use of only two lines of evidence: file review and key stakeholder interviews.

Complementary research methods were used as a means to enhance the reliability and validity of information and data to be collected. In particular, individuals interviewed for the evaluation were those associated with the projects selected for the file review.

2.2.1 File review

The selection of files to be included in the file review was driven by the dollar value of the files. In particular, GCS attempted to select the files that accounted for the largest proportion of the money spent by the program during the evaluation period. Five projects were selected for inclusion in the file review. The value of these projects amounted to $740,306 or approximately 80 per cent of total program spending over the five-year evaluation period. These five projects also corresponded to four different recipient organizations and included project spending in each of the five years included in the evaluation. Therefore, these projects provide a reasonable sample from the existing Secretariat projects under the RPIA Program, since the total sample of these selected projects represent the majority of the funding.

A file review template was created to gather the relevant information from the project files. Missing or additional information needed was gathered through interviews conducted with key stakeholders.

2.2.2 Stakeholder interviews

A total of three interviews were conducted with representatives of the projects selected for the file review.[1] All interviews were conducted in person. Interviewees were contacted in advance of the interview to schedule an appropriate time, and all interviewees received the interview guide in advance (see Appendix D for the interview guide).

The results of interviews were summarized in a template and then analyzed according to evaluation questions and indicators.

2.3 Limitations of methodology

As with any evaluation, there are limitations to the evaluation methodologies. First, the interviews and file review did not include all projects that received funding through the Secretariat's RPIA Program. However, as previously mentioned, efforts were made to select those projects that accounted for the largest percentage of program spending over the evaluation period. GCS considers the files selected to be a good sample and believe they represent Secretariat projects under the RPIA Program. Note that an interview could not be completed for one project because the individuals responsible were no longer with the sector. Therefore, results are based on interviews with three individuals who represented four RPIA Program projects. While the number of interviewees is limited, they represent a very large proportion of total project spending, as discussed in section 2.2.1.

A second limitation of the evaluation relates to a lack of quantitative information. The interviews and file review conducted for the evaluation are qualitative sources of information. Little quantitative information was available about the program, particularly with regard to its benefits or outcomes.



3.0 Evaluation Findings

This section of the report presents a summary of the evaluation findings, which are organized by evaluation question.

3.1 Program relevance

Conclusion: RPIA Program projects are aligned with program and departmental objectives. The program is meeting the needs of sectors by providing funding for projects that otherwise may not have been possible. Some sectors would not have had access to the project benefits if the funding had not been provided.

Findings: RPIA Program projects are aligned with program and departmental objectives. According to the RPIA Program's terms and conditions, program objectives include the following:

  • To improve, advance, and promote public service management practices by supporting research initiatives that contribute to the advancement of public policy research and public service management; and
  • To enhance the effectiveness of public service management practices, reporting, and service delivery to the public. [2]
  • The Secretariat's raison d'être, as articulated in its 2008–09 Report on Plans and Priorities (RPP), is to ensure that government is well-managed and accountable and that resources are allocated to achieve results.[3]

RPIA Program projects that were reviewed were related to conducting research and holding events on topics that were relevant to the objectives of Secretariat and the program. The projects reviewed, as part of the file review, consisted of projects producing research reports, symposia, educational events, workshops, seminars, conferences, lectures, and other studies. Often, the projects reviewed brought together a leading group of commentators with relevant experience and expertise on the topic of interest. Topics examined through RPIA Program projects reviewed included the following:

  • Public service governance;
  • Inter-jurisdictional service delivery;
  • The design of public policy, such as the design and administration of programs of grants, contributions, and other forms of financial support; and
  • The promotion of approaches to technology, information, and service (a commitment of the Secretariat's RPP and a priority of the Chief Information Officer for the Government of Canada).

As well, RPIA Program project documentation noted that the program supports the introduction of new ideas into the mainstream discourse on public sector management, reporting, and service delivery.

Interviewees also believe that RPIA Program projects are contributing to program objectives. For example, one interviewee noted that the RPIA Program was a catalyst toward government-wide reform and a move toward responsible management. Another interviewee noted the link between the RPIA Program and the Secretariat's strategic objective that the government be well-managed, noting that lectures supported through the program generate ideas on good management and that those ideas are shared between agencies.

The RPIA Program is meeting the needs of sectors by providing funding for projects that otherwise may not have been possible. Some sectors would not have had access to the project benefits if the funding had not been provided.

Interviewees believe that the RPIA Program met the needs of sectors, primarily because it allocated funds to undertake work that would not otherwise have been possible. Interviewees noted that without the RPIA Program, there would not have been funds available to undertake the projects. As noted by one interviewee, there is little discussion forums on public sector management, and it is difficult to generate dialogue. RPIA Program–funded projects have helped to fund activities to generate dialogue on this topic. 

Interviewees said that recipient organizations would not have carried out the activities without RPIA Program funding. In addition, according to interviewees, the department would not have had the same benefits had it not provided the funding. Two of the three interviewees for this evaluation specifically noted the need for an independent, neutral, third-party organization to achieve the objectives of the funded project. These interviewees noted that the Secretariat could not have done the projects themselves, as they would have been viewed as biased. As well, the projects funded under the RPIA Program brought together multiple organizations to work together. Without the RPIA Program, stakeholders would have worked more in isolation.

3.2 Program success

Conclusion: There is a lack of evidence on file regarding the benefits of the projects; however, interviewees believe that the RPIA Program has enhanced policies or programs and helped to increase knowledge and build capacity.  

Findings: Information from the file review shows that there is a lack of evidence on file regarding the benefits of the projects. For the most part, the only information available from the file review, with regard to benefits of the projects, was information about anticipated benefits as opposed to actual benefits realized. These anticipated benefits came from the submission and recommendation form on the file. They include the following:

  • To extend and deepen the dialogue across levels of government;
  • To advance public policy development around the design and use of the grants and contributions tool;
  • To provide an opportunity for exchange on contemporary public service issues and provide an opportunity for exchange on the advancement of public service management; and
  • To improve client service delivery.

The RPIA Program does not have a requirement to provide any kind of post-project summary for the project file. While some project reports were gathered during the course of the evaluation, they often did not provide information about the benefits of the project to the Secretariat. Thus, actual benefits of these projects are not known to the RPIA Program staff.

Recommendation No. 1: The program area should implement a mechanism to ensure that the impact of the program can be measured with respect to actual benefits rather than anticipated benefits. For example, a post-project report could be a requirement for all RPIA Program projects. This could be as simple as a one-page template that is filled out for each project.

Interviewees believe that the RPIA Program has contributed to its outcomes, i.e., enhancing policies or programs, increasing knowledge of emerging policy issues, and building capacity for analysis. The interviewee for one project noted that the project was impacting current policy development and the subsequent development of new standards, and that the project was essential to ensure that current policy development efforts align with other jurisdictions. A second interviewee noted that recommendations coming out of the RPIA Program led to a government action plan that was acceptable to stakeholders.  

Interviewees also agreed that the RPIA Program has helped to increase knowledge of emerging policy issues, e.g., public service retention. Interviewees noted that their RPIA Program projects had brought together people from various groups, such as public policy practitioners, academics, voluntary-sector representatives, public service employees, and representatives from organizations that deal with public policy. Bringing together people from various jurisdictions who are involved in discussions and share information increases the knowledge base.

Interviewees also agreed that the RPIA Program has helped to build capacity for analysis. For example, interviewees remarked that, as a result of the RPIA Program projects, sectors have a much better understanding of the needs of other jurisdictions and what they can contribute in federal policy development. One interviewee noted that the sector is in a better position to make analyses and recommendations to Treasury Board ministers as a result of an increased information base and analytical capacity as a result of RPIA Program projects. Another interviewee noted that getting different people together leads to establishing priorities and identifying key items that have the most impact. 

One interviewee found it hard to comment on project success, indicating that the benefits of the project were directed more to other departments rather than to the Secretariat.

3.3 Program cost-effectiveness and alternatives

Conclusion: Interviewees believe that the program is cost-effective because of the benefits of partnering with other organizations, i.e., leveraging, and because few alternatives are available. In most cases, the Secretariat provided a portion of the funding to carry out projects. The program is not seen as being sufficiently visible within the department, and records management could be improved.

Findings: Interviewees believe that the RPIA Program is cost-effective. While interviewees were not able to establish a numeric return on investment, all indicated that the program is cost‑effective. 

There appear to be few options for alternatives to the program. When asked if there are alternative means to achieve the objectives of the RPIA Program, interviewees suggested that work could be done in-house; however, there are insufficient resources (i.e., human and financial) to do so. As mentioned previously, in some cases an independent third party is needed to successfully achieve the objectives of the project. Contracting was also identified as an alternative, but interviewees were not certain that this could be done, given the requirements of the Financial Administration Act. According to interviewees, difficulties would be encountered due to the fact that there are often multiple partners for RPIA Program projects and shared benefits. As well, RPIA Program projects do not necessarily have a clear output or deliverable.

Cost-effectiveness was enhanced by the fact that the Secretariat typically provided only a portion of the funding to carry out projects. The department partnered with other organizations to undertake the projects, thus leveraging other resources. For the files reviewed, the Secretariat contributed anywhere from 8 to 44 per cent of the total cost of the project.

Because of the nature of the program and its relatively small size, program efficiency was not seen as a relevant issue to focus on in this evaluation. However, some efficiency-related information was gathered during interviews. 

Interviewees generally had positive views on program administration. Most interviewees noted that the application process was straightforward and easy and that approvals were timely. However, interviewees identified one improvement that should be made to the program. They believe that the program needs to be marketed better within the department. They reported that it was difficult to find out about the program. As well, one interviewee commented that there did not seem to be any one person who was able to guide him through the process and who could answer all his questions. Another interviewee noted that she had to find out about the program and do the administrative work herself. Interviewees agreed that the program needs to be more visible within the department.

Recommendation No. 2: The Secretariat should increase the visibility of the RPIA Program within the department and ensure that employees are aware of the program and know whom to contact for more information.

In addition, the file review undertaken for this evaluation highlighted a few issues relating to project documentation. First, not all project-related documents, i.e., the submission and recommendation forms, were available through the RPIA Program manager. Second, post‑project reports are not required on file, and the actual benefits of projects are not known to RPIA Program staff.

Recommendation No. 3: Project documentation should be collected and retained more systematically and stored in a central location.



4.0 Summary of Conclusions and Recommendations

Program relevance

RPIA Program projects are aligned with program and departmental objectives. The program is meeting needs of sectors by providing funding for projects that otherwise may not have been possible. Some sectors would not have had access to the project benefits if the funding had not been provided.

Program success

There is a lack of evidence on file regarding the benefits of the projects; however, interviewees believe that the RPIA Program has enhanced policies or programs helped to increase knowledge to build capacity. 

Recommendation No. 1: The program area should implement a mechanism to ensure that the impact of the program can be measured with respect to actual benefits rather than anticipated benefits. For example, a post-project report could be a requirement for all RPIA Program projects. This could be as simple as a one-page template that is filled out for each project.

Program cost-effectiveness and alternatives

Interviewees believe that the program is cost-effective because of the benefits of partnering with other organizations, i.e., leveraging, and because few alternatives are available. In most cases, the Secretariat provided a portion of the funding to carry out projects. The program is not seen as being sufficiently visible within the department, and records management could be improved.

Recommendation No. 2: The Secretariat should increase the visibility of the RPIA Program within the department and ensure that employees are aware of the program and know whom to contact for more information.

Recommendation No. 3: Project documentation should be collected and retained more systematically and stored in a central location.



Appendix A-List of RPIA Program Approved Projects

Project Secretariat Contribution Amount funded by Other Government Departments Total RPIA Program Contribution
2003 National Conference $7,500 - $ 7,500
Crossing Boundaries National Council - $ 300,000 $ 300,000
OECD Research Project: Reallocation - Role of Budget Institutions $ 15,000 - $  15,000
Rebuilding Trust in Canadian Institutions $ 25,000 - $ 25,000
Learning from the Best: A Program of Research and Education on "Best Practices" in Maximizing the Talents of Visible Minorities $ 50,000 - $ 50,000
National Association of Federal Government Executives $ 13,368 - $ 13,368
2003-04 Total $ 110,868 $ 300,000 $ 410,868
Crossing Boundaries National Council $ 83,932 $ 175,000 $ 258,932
2004-05 Total $ 83,932 $ 175,000 $ 258,932
Crossing Boundaries National Council $ 25,000 - $ 25,000
Public Sector Chief Information Officers Council $ 37,500 - $ 37,500
Conference Board Project $ 25,000 - $ 25,000
2005-06 Total $ 87,500 - $ 87,500
Symposium on Partnering for Public Purpose $ 29,874 - $ 29,874
Gordon Osbaldeston Lecture - November 2006 $ 20,000 - $ 20,000
2006-07 Total $ 49,874 - $ 49,874
Gordon Osbaldeston Lecture - November 2007 $ 30,000 - $ 30,000
For a June workshop in support of the Recommendations of the Blue Ribbon Panel  Report in Grants and Contributions $ 15,000 - $ 15,000
To assist in creating a pan-Canadian strategy for identity management and authentication $ 46,500 $ 30,000 $ 76,500
2007-08 Total $ 91,500 $ 30,000 $ 121,500
Total Secretariat RPIA Program Payments During Evaluation Period (2003-04 TO 2007-08) $ 423,674 $ 505,000 $ 928,674


Appendix B-Secretariat RPIA Program Evaluation Matrix

Issue Question Indicator Data Source/ Methodology
Relevance Is the RPIA Program meeting the needs of programs within the Secretariat? Internal stakeholder opinion Internal stakeholder interviews
Success Has the program achieved the intermediate outcomes identified in the logic model? Internal stakeholder opinion Internal stakeholder interviews
Achievements of 50% sample of contribution agreements (agreements with largest dollar values) File review
Internal stakeholder interviews
Cost-effectiveness What leverage did the program generate (i.e., proportion of project funding provided by other organizations)? Value of financial and "in kind" investments of 50% sample of contribution recipients (agreements with largest dollar values) File review
Internal stakeholder interviews


Appendix C-Secretariat RPIA Program Logic Model

Graphic 1



Appendix D-Interview Guide

Evaluation of the Research and Policy Initiatives Assistance Program (RPIA Program)

Interview Guide for Treasury Board Secretariat (TBS)

Government Consulting Services (GCS) has been engaged by Treasury Board Secretariat/Finance Canada (TBS/FC) to conduct an evaluation of the RPIA Program. The purpose of this evaluation is to examine the relevance, success, and cost-effectiveness of the program.

As part of the evaluation, GCS is conducting interviews with key internal stakeholders involved in the program. The goal of the interviews is to gain a better understanding of the program, collect information to assess their success, and identify possible improvements.

The following questions will serve as a guide for our interview. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department.

Background

1. Can you please briefly describe your role and involvement with the RPIA Program?

Relevance

2. What are the objectives of the RPIA Program in Treasury Board Secretariat (TBS)? 

3. Is the RPIA Program meeting the needs of TBS and of your sector? Please explain.

4. Has the RPIA program contributed to the achievement of TBS strategic objectives and government‑wide priorities? If so, in what way?

5. To what extent would organizations receiving contributions through the RPIA Program have been able to carry out the activities funded if they had not obtained a RPIA Program contribution?

a. What would have been the impact on TBS if funding had not been provided? (i.e., would TBS still have had access to the benefits/outcomes of the activities?)

Success

6. Has the RPIA Program contributed to enhancing policies and programs for which your sector is responsible? If yes, please provide specific examples (e.g., research findings and use of these findings, post-conference opinions recorded and used, etc.). If no, why not?

7. In your sector, has the RPIA Program contributed to increasing knowledge of emerging policy issues of interest to TBS? If yes, please provide specific examples (e.g., research findings, post-conference opinions recorded and used, etc.). If no, why not?

8. Has the RPIA Program contributed to increasing the capacity of your sector to produce more thorough and relevant analyses? If yes, please explain and provide specific examples. If no, why not?

Cost-Effectiveness/Alternatives

9. Is the RPIA Program cost-effective?

10. Are there alternative means to achieve the objectives of the RPIA Program? If yes, what are these alternatives and would they be more or less cost-effective than the existing RPIA Program?

11. Do you have any suggestions for improving the RPIA Program (i.e., to make it more efficient or cost-effective)?

Other

12. Has the RPIA Program had any impacts (either positive or negative) that were unintended? Please explain.

13. Do you have any other comments you would like to make regarding the RPIA Program in TBS that may be relevant to our evaluation?

Thank you very much for participating in our evaluation study.



Footnotes

1 One interviewee was the representative for two projects.

2 Treasury Board Secretariat. Terms and Conditions for the RPIA Program.

3 Treasury Board Secretariat. Report on Plans and Priorities, 2008-2009.



Graphic 1 - Text Version

TBS RPIA Program Logic Model

The logic model depicts the activities, outputs, and immediate, intermediate, and final outcomes of the RPIA Program.

The three activities outlined in the logic model are those funded by the RPIA Program and undertaken by recipient departments. These activities are defined as follows:

  • Establishing expedient and flexible contribution agreements
  • Identifying and seizing opportunities to leverage external partners
  • Scanning academic and professional communities for activities that could further key departmental objectives

These activities resulted in two outputs and are defined as follows:

  • Conferences, workshops, seminars, and other knowledge and outreach activities
  • Research analysis and data collection

Intended impacts of activities are called outcomes, and they are the reason for conducting activities. 

Immediate outcomes occur in the short term, typically within one year of carrying out an activity. The RPIA Program logic model has three immediate outcomes, which are defined as follows:

  • Access to an expanded knowledge base on subjects of interest
  • Better-informed policy context and expanded community of interest
  • Increased leverage of resources or other capacities from external partners toward departmental objectives

Intermediate outcomes occur after immediate outcomes. The RPIA Program logic model has three intermediate outcomes and are defines as follows:

  • Enhanced policies and programs of the department
  • Increased contribution to knowledge related to emerging policy issues of interest to the department
  • Departmental capacity to produce thorough and relevant analysis is enhanced

Final outcomes occur as a result of the achievement of intermediate outcomes. The RPIA Program logic model has two final outcomes and are defined as follows:

  • Contribution to the advancement of public service management and public policy research
  • Contribution to the effectiveness of public service management practices, reporting, and service delivery to the public

Return to Graphic 1