3. Scope of the Evaluation

The scope of this evaluation was limited to the current submission process and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.

The evaluation covered all steps of the current submission process involving the Secretariat:

The evaluation did not include federal organizations' internal processes for preparing Treasury Board submissions. Furthermore, the evaluation covers only "Part A" submissions. MCs, GIC submissions, and the federal budget, though outside the scope of the evaluation, were considered to the extent that they provided contextual information.

Canada's Economic Action Plan

The evaluation's planning and data collection phases were already completed at the time Canada's Economic Action Plan was announced in late January 2009. As a result, implementation of this stimulus package could not be included within the scope of the evaluation.

4. Contextual Issues

(a) Volume and complexity of submissions

Between 2004 and 2009 (September to August calendar), an average of 804 Treasury Board submissions were made each year. It should be noted that the Secretariat itself usually accounts for the largest number of Treasury Board submissions among all federal organizations given that it acts as both a department and the management board of the federal public service. As such, some of the Secretariat's submissions are to seek approval or amendments to Treasury Board policies or are related to Treasury Board's role as employer.

As shown in Table 1, the Secretariat's program sectors, with the exception of the International Affairs, Security and Justice Sector, reported a higher incidence of "standard" submissions than "complex" ones.12 Though in the minority, figures for complex submissions are nevertheless significant.

Table 1: "Standard" vs. "Complex" Submissions
Program Sector % Standard % Complex
Social and Cultural 54.9 34.3
Economic 48.7 38.1
International Affairs, Security and Justice 41.7 53.2
Government Operations (program side) 66.0 29.9
Overall Average 54.2 38.4

Evaluation participants described standard submissions as follows:

Complex submissions were said to involve one or more of the following:

(b) Logic model for the Treasury Board submission process

No logic model had previously been developed for the Treasury Board submission process. A logic model is essential to this (and to any) evaluation, because it describes the consensus among key stakeholders on the intended outcomes of the Treasury Board submission process. A consensus view on outcomes is the touchstone against which the evaluation must be conducted. Relevance, effectiveness, and economy cannot be assessed in a vacuum; rather, these terms only have meaning in relation to the ultimate outcome of the submission process.

It was therefore necessary for the evaluation team to develop a logic model that would provide the necessary foundation for the evaluation. Consultations were held to gather information on the intended outcomes of the Treasury Board submission process. The evaluation team interviewed 19 individuals from areas across the Secretariat who deal with Treasury Board submissions, including analysts and senior analysts, directors, and an assistant secretary. In addition, two working sessions were held with Secretariat analysts to develop and validate the logic model and evaluation matrix that had been developed by the evaluation team. The consensus among this group was that the ultimate outcome of the submission process was to support Treasury Board in making decisions that contributed to ensuring that programs and spending are aligned with the government's priorities.

Consultation with senior management provided another perspective. Participants in the Assistant Deputy Minister (ADM) Working Session observed that ensuring alignment of new initiatives with the government's priorities is the intended outcome of the MC process rather than the Treasury Board submission process. They suggested that the purpose of the Treasury Board submission process was to provide federal organizations with the resources and authorities needed to achieve the social and economic outcomes desired by Cabinet and to ensure that the programs delivered demonstrated value for money. This proposed ultimate outcome appears to be consistent with the Secretariat's documentation and communications about its expected results. The statement is consistent with the Secretariat's strategic outcome: "Government is well managed and accountable, and resources are allocated to achieve results." This ultimate outcome is also consistent with the expected results outlined in the Secretariat's 2008-09 Departmental Performance Report (DPR): "allocation of resources so that they are aligned with the government's priorities and responsibilities, thereby ensuring that federal programs are effective and efficient and provide value-for-money." The link to broader outcomes (e.g. social and economic) is captured in the DPR's reference to "alignment of government's priorities and responsibilities."

Given that the Treasury Board submission process is not limited to either a single Secretariat subactivity or one organization within the Secretariat, instead crossing a number of organizations, including the program sectors, EMS, and many policy Centres of Excellence, it is perhaps not unexpected that a logic model for the process did not exist before the evaluation and that the ultimate outcome of the Treasury Board submission process was therefore not defined. That said, the evaluation found that there is a sound basis for articulating the outcome, particularly from the strategic outcome and expectations articulated in the Secretariat's DPR. The need for a final logic model, including the ultimate outcome, is addressed in the Conclusion section of this report.

For the purposes of the evaluation, outcomes of the Treasury Board submission process were based on analysis of the information gathered from the consultation sessions as well as information found in documentation on the submission process. The Guide was examined, and while it is very detailed in terms of how to do Treasury Board submissions, it is less explicit about why they must be done. Nevertheless, it was possible to make inferences about the ultimate outcome based on other statements found in the Guide. The Guide's clearest statement about the ultimate outcome of the submission process is as follows:

The Treasury Board submission transforms policy rationale and objectives into a program that will achieve those objectives. It details how the federal organization will carry out the policy /initiative; why the proposed method of implementation is the best one; how the proposal contributes to government-wide aims such as accountability, transparency, and interoperability of information; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS);13 and how the federal organization will conduct monitoring and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets.

Although reference may be made to the MC and Cabinet decisions, it is generally unnecessary to repeat the policy rationale in a submission. The submission must nevertheless be consistent with the Cabinet decisions.14

In other words, questions of policy rationale and intended social and economic outcomes (or "objectives") are dealt with in the MC (as was suggested by participants in the ADM Working Session). The Treasury Board submission deals with more detailed questions of program design and implementation. It is therefore reasonable to infer that the ultimate value or outcome of the submission process has to do with ensuring that the initiative in question is well designed in relation to the policy rationale described in the MC and in terms of its implementation arrangements and its reporting, accountability, and control mechanisms. The evaluation team therefore proposed the following as the ultimate outcome for the Treasury Board submission process:

Program implementation serves the policy outcomes defined by Cabinet in the most economical manner possible and satisfies the government's obligations for transparency, accountability, and prudence.

It was not possible to validate this proposed ultimate outcome with an appropriate cross-section of stakeholders. The evaluation team therefore recognizes that it is simply a place holder for the purposes of the evaluation.

On the other hand, the other elements of the logic model proposed by the evaluation team-the immediate and intermediate outcomes and the outputs-do reflect the common view within the Secretariat.

The involvement of the Secretariat in the submission process consists of six major components, as outlined in the logic model developed by the evaluation team for the purposes of the evaluation (Appendix A; see also Figure 1):

The activities identified in the middle row of Figure 1 should lead to the outcomes indicated in the logic model developed by the evaluation team:

The foregoing outcomes are expected to result in high-quality Treasury Board submissions. These in turn are expected to contribute, ultimately, to federal organizations' effective implementation of programs that serve the government's desired policy outcomes in the most efficient possible manner while also fulfilling the government's obligations for transparency, accountability, and prudence.

Date modified: