Common menu bar links
Breadcrumb Trail
ARCHIVED - Evaluation of the Research and Policy Initiatives Assistance Program - Treasury Board of Canada Secretariat
This page has been archived.
Archived Content
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
2.0 Evaluation Methodology
2.1 Evaluation issues and questions
This evaluation was undertaken to respond to the evaluation issues and questions, as articulated in the evaluation matrix developed for the evaluation and included in the Memorandum of Understanding between the Secretariat/FIN and GCS.
The evaluation questions examined are shown in Table 2. The complete evaluation matrix is presented in Appendix B. Appendix C contains the logic model for the RPIA Program.
| Evaluation Issue | Evaluation Question |
|---|---|
| Relevance |
|
| Success |
|
| Cost-effectiveness |
|
2.2 Methodology
The evaluation methodology was designed to ensure that the cost of the evaluation was commensurate with the relatively small amount of funding provided through this program. As a result, the evaluation integrated the use of only two lines of evidence: file review and key stakeholder interviews.
Complementary research methods were used as a means to enhance the reliability and validity of information and data to be collected. In particular, individuals interviewed for the evaluation were those associated with the projects selected for the file review.
2.2.1 File review
The selection of files to be included in the file review was driven by the dollar value of the files. In particular, GCS attempted to select the files that accounted for the largest proportion of the money spent by the program during the evaluation period. Five projects were selected for inclusion in the file review. The value of these projects amounted to $740,306 or approximately 80 per cent of total program spending over the five-year evaluation period. These five projects also corresponded to four different recipient organizations and included project spending in each of the five years included in the evaluation. Therefore, these projects provide a reasonable sample from the existing Secretariat projects under the RPIA Program, since the total sample of these selected projects represent the majority of the funding.
A file review template was created to gather the relevant information from the project files. Missing or additional information needed was gathered through interviews conducted with key stakeholders.
2.2.2 Stakeholder interviews
A total of three interviews were conducted with representatives of the projects selected for the file review.[1] All interviews were conducted in person. Interviewees were contacted in advance of the interview to schedule an appropriate time, and all interviewees received the interview guide in advance (see Appendix D for the interview guide).
The results of interviews were summarized in a template and then analyzed according to evaluation questions and indicators.
2.3 Limitations of methodology
As with any evaluation, there are limitations to the evaluation methodologies. First, the interviews and file review did not include all projects that received funding through the Secretariat's RPIA Program. However, as previously mentioned, efforts were made to select those projects that accounted for the largest percentage of program spending over the evaluation period. GCS considers the files selected to be a good sample and believe they represent Secretariat projects under the RPIA Program. Note that an interview could not be completed for one project because the individuals responsible were no longer with the sector. Therefore, results are based on interviews with three individuals who represented four RPIA Program projects. While the number of interviewees is limited, they represent a very large proportion of total project spending, as discussed in section 2.2.1.
A second limitation of the evaluation relates to a lack of quantitative information. The interviews and file review conducted for the evaluation are qualitative sources of information. Little quantitative information was available about the program, particularly with regard to its benefits or outcomes.