This page has been archived.
Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.
Acronym | Meaning |
---|---|
ADM | Assistant Deputy Minister |
ARLU | Annual Reference Level Update |
BOSR | Budget Office Systems Renewal |
CFO | Chief Financial Officer |
CIO | Chief Information Officer |
COE | Centre of Expertise |
DM | Deputy Minister |
EMIS | Expenditure Management Information System |
EMS | Expenditure Management Sector |
ES-0# | Economics and Statistics (employee classification) |
EX-0# | Executive (employee classification) |
EXCO | Executive Committee |
FAA | Financial Administration Act |
FTE | Full-Time Equivalent |
GIC | Governor in Council |
GOS | Government Operations Sector |
HR | Human Resources |
MAF | Management Accountability Framework |
MC | Memorandum to Cabinet |
OIC | Order in Council |
PDG | Program Directors Group |
RDIMS | Records, Documents and Information Management System |
STS | Submission Tracking System |
Secretariat | Treasury Board of Canada Secretariat |
The Internal Audit and Evaluation Bureau (IAEB) of the Treasury Board of Canada Secretariat would like to thank all the individuals who contributed to this evaluation-interviewees, survey respondents, working session participants, and the Secretariat's Evaluation Committee-for their valuable input and support. The work of Secretariat officials involved in the validation of findings, particularly the Program Directors Group and assistant secretaries from the program sectors, was also appreciated. Finally, IAEB would like to thank the consultants, Government Consulting Services (Public Works and Government Services Canada), who undertook the evaluation, and Mark Schacter, who collaborated in writing the report.
The Treasury Board of Canada Secretariat (Secretariat) is the administrative arm of Treasury Board, a Cabinet committee whose ministers render decisions on the government's management and expenditures. The Secretariat supports Treasury Board ministers in this function and also fulfills the statutory responsibilities of a central government agency. When a federal organization is seeking approval or authority from Treasury Board for an initiative that it would not otherwise be able to undertake or that is outside its delegated authorities,1 it prepares a Treasury Board submission in consultation with the Secretariat.
This evaluation assessed the relevance, effectiveness, and economy of the Treasury Board submission process. Given that the Secretariat is the "owner"; of the Treasury Board submission process, the evaluation paid close attention to the role it plays in contributing to the success of the process. It should be noted that the scope of this evaluation was limited to the current process used to support Treasury Board ministers and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, the review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The evaluation framework used multiple lines of evidence and complementary research methods to ensure the reliability and validity of the data. The following research methods were used:
(i) document review; (ii) interviews with stakeholders and experts; (iii) a working session with assistant secretaries from the Secretariat and assistant deputy ministers from selected federal organizations; (iv) review of administrative, financial, and statistical data; and (v) a survey of analysts at the Secretariat and in federal organizations. A limitation of the methodology was that Treasury Board ministers could not be interviewed because data collection was undertaken during an election period.
The evaluation covered all steps of the current submission process involving the Secretariat:
The evaluation did not include federal organizations' internal processes for preparing Treasury Board submissions. Furthermore, the evaluation covers only "Part A" submissions.2
This is the first formal evaluation of the Treasury Board submission process; therefore, it was not possible to anticipate, at the outset, all the evaluation issues to be addressed and lines of inquiry to be pursued. Because the data-gathering exercise uncovered evaluation issues that were not initially apparent, new lines of evidence were introduced, such as additional document reviews, additional interviews, and validation sessions with Secretariat senior management. The research for the evaluation was conducted between November 2008 and March 2009.
The current Treasury Board submission process is relevant, appears to be effective, and has key strengths; however, some important opportunities exist to improve and enhance its effectiveness and efficiency, which should be addressed through a combination of action and further investigation.
Opportunities to improve the Treasury Board submission process fall into three areas:
1. The Secretariat should finalize the logic model for the Treasury Board submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
2. The Secretariat should examine in detail, and address where necessary, survey results relating to the advice its analysts provide to federal organizations during the submission process.
3. Secretariat should examine how human resources issues, especially turnover among program analysts, affect the Treasury Board submission process and knowledge management.
4. The Secretariat should account for the costs of managing the submission process-at least for the department if not for the entire public administration.
5. The Secretariat should review, and improve where necessary, its professional development, training, and outreach activities related to the Treasury Board submission process.
6. The Secretariat should explore options for a more systematic approach to information and knowledge management in the Treasury Board submission process.
The Treasury Board is a Cabinet committee of the Queen's Privy Council of Canada. It was established in 1867 and given statutory powers in 1869. The Financial Administration Act (FAA) provides the authority for the Treasury Board to exercise responsibilities in areas relating to general administrative policy in the federal public administration, the organization of the federal public administration, financial management, the review of annual and longer term expenditure plans and programs of federal organizations, the management and development of lands by federal organizations, human resources management in the federal public administration, the terms and conditions of employment, internal audit, and other matters determined by the Governor in Council (GIC).
The Treasury Board is headed by a President, whose formal role is to chair the Treasury Board. He carries out his responsibility for the management of the government by translating the policies and programs approved by Cabinet into operational reality and by providing federal organizations with the resources and the administrative environment they need to do their work.
The Treasury Board of Canada Secretariat (Secretariat) is the administrative arm of the Treasury Board and has a dual mandate: to support the Treasury Board itself, which is a committee of ministers, in rendering decisions on the government's management and expenditures and to fulfill the statutory responsibilities of a central government agency. With respect to the Treasury Board submission process, the Secretariat provides advice and support to Treasury Board ministers in fulfilling their responsibilities as outlined in the FAA and their role of ensuring value for money in government spending through oversight of federal organizations' financial management functions. The Secretariat makes recommendations and provides advice to the Treasury Board on policies, directives, regulations, and program expenditure proposals involving the management of the government's resources. Treasury Board's responsibilities for the general management of the government have an impact on initiatives, issues, and activities that cut across all policy sectors managed by federal organizations and organizational entities.
A Treasury Board submission is required when a federal organization is seeking approval or authority from Treasury Board for an initiative that it would not otherwise be able to undertake or that is outside its delegated authorities. Treasury Board submissions can relate to one or more of the Treasury Board responsibilities outlined in the FAA.
The Treasury Board submission process has never been evaluated; as a key line of business of the Secretariat, it was important to do so. This evaluation assessed the relevance, effectiveness, and economy of the Treasury Board submission process. Given that the Secretariat is the "owner" of the Treasury Board submission process, the evaluation paid close attention to the role played by the Secretariat in contributing to the success of the process. It should be noted that the scope of this evaluation was limited to the current process used to support Treasury Board ministers and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The purpose of the evaluation was to assess the relevance, effectiveness, and economy of the Treasury Board submission process, with a focus on the following issues:4
The research for this evaluation was conducted between November 2008 and March 2009.
The Change Agenda, an initiative launched in 2007, is the Secretariat's plan for building management excellence across the federal government by refocusing its relationship and the manner in which it conducts business with other federal organizations. Through the Change Agenda, it is expected that the Secretariat will demonstrate leadership in management excellence, while taking a more strategic and less transactional approach to federal organizations. The efficiencies gained from such an approach would allow for a greater focus on higher value-added activities, would improve relationships with federal organizations, and would result in better advice to Treasury Board ministers. Although not a subject of this evaluation, the Change Agenda is integral to the Treasury Board submission process because it addresses how the Secretariat's business is conducted.
Treasury Board submissions are one of the three main documents, besides Memoranda to Cabinet (MC) and GIC submissions, that support the formal decision-making process in government.
Legislation, Treasury Board policies, or other Cabinet decisions usually establish the requirements for Treasury Board approval.5 While Treasury Board submissions may relate to any of the Treasury Board responsibilities outlined in the FAA, typical examples of submissions include seeking:
It should be noted that the Treasury Board submission process has many links to the government's budget, Estimates, and supply processes. For instance:
Treasury Board submissions are divided into "Part A" and "Part B" submissions. All Treasury Board submissions (including strategic reviews) are considered "Part A" submissions with the exception of OICs requiring Treasury Board recommendation. "Part B" submissions are therefore those with OIC attached. As a result, the process, procedures, and relevant actors for "Part A" and "Part B" submissions differ. While OICs are legislative instruments, Treasury Board submissions are a mechanism used by a legal body, Treasury Board.
What is generally referred to as the "Treasury Board submission process" in fact incorporates three phases: pre-submission, submission, and post-approval (depicted in Figure 1).
The pre-submission phase begins when a federal organization contacts the Secretariat either to seek guidance on whether a submission is required or to give notice that it intends to make a submission. Even when a federal organization is simply giving notice of its intent to put forth a submission, the Secretariat may provide its advice on whether or not the submission is in fact necessary. Strictly speaking, there is no Treasury Board submission at this point. Nevertheless, the evaluation-reflecting general practice within government-treats the related activities as being part of the submission process.
The submission phase covers the period between when a decision is taken by the federal organization to prepare a submission and the point at which the submission is considered at a Treasury Board meeting. This phase includes the draft submission stage in which program analysts, along with their managers, act as the single window to provide advice to representatives of federal organizations, including advice from Centre of Expertise (COE) analysts. It also includes the final submission stage in which a précis is prepared and discussed at the Strategy Committee, a senior governance body chaired by the Secretary of the Treasury Board. The Strategy Committee, which is part of Secretariat-wide due diligence and oversight, enables the provision of considered, coherent, and consolidated advice to Treasury Board ministers. The committee meets frequently and reviews all submissions and associated risk assessments and advice.
The submission is then presented to the Treasury Board for consideration and approval. At this point, Secretariat advice on risks and considerations is also presented to ministers.
The post-approval phase covers actions taken by the Secretariat following approval of a submission, such as communicating the decision to the federal organization, communicating conditions (if any) attached to the submission, and monitoring these conditions.
Figure 1: Treasury Board Submission Process
As noted earlier, the Change Agenda is integral to the Treasury Board submission process. The Secretariat in fact performs several of the roles described in the Change Agenda throughout the submission process. In the pre-submission phase, the Secretariat assumes its enabler role when providing the federal organization with guidance and advice. In the submission phase, the Secretariat challenges the submission. However, once a submission is ready to be presented to Treasury Board, the Secretariat shifts to its support role, providing the best advice and recommendations to Treasury Board ministers. This shift in the Secretariat's role is necessary in the Canadian federal government model6 because the Secretariat both assists the client department in drafting the submission and ensures that the assistant secretary presenting the submission to Treasury Board is well-equipped with the best possible advice to support the recommendation.
The Treasury Board submission process is a deliberate and iterative process, at times involving intensive discussion and negotiation between the Secretariat and federal organizations as well as discussion within the Secretariat itself. Given the Secretariat's single-window approach, much of the process is coordinated by the Secretariat program sector that is responsible for the federal organization's portfolio7 and remains the primary point of contact.
While the program analyst plays the lead role in relation to the client department, many Secretariat sectors, apart from the related program sector (e.g. the Expenditure Management Sector (EMS), the Office of the Comptroller General (OCG), and various policy centres), typically get involved in reviewing draft submissions and formulating advice to Treasury Board ministers. Analysts from these sectors, COE analysts, play a substantially different role than program analysts: they review and provide feedback on the sections of a submission that pertain to their area of expertise and policy mandate.
Following review of the submission, the program analyst drafts a précis, which is essentially the Secretariat's opinion and advice meant for Treasury Board ministers. The précis includes a summary of the federal organization's request, risk assessment against pre-established risk criteria (which are described in the Guide8), and a recommendation to Treasury Board to either approve, not approve, or approve with conditions the proposals put forth in the submission.
Upon consideration of the Treasury Board submission and Secretariat advice, Treasury Board ministers may attach further conditions to a submission beyond those proposed by the Secretariat or may request follow-up from the federal organization or the Secretariat on a particular issue. For example, a condition may relate to the time frame in which to conduct a formal evaluation of a program.
How risk is handled in the Treasury Board submission process
The Guide states that federal organizations should "provide details of specific risks that need to be managed, measures proposed to mitigate those risks, and any residual risks." Beyond this, it becomes the responsibility of Secretariat analysts9 to assess submissions based on a standardized risk framework that includes the following general risk categories:
Before preparing the précis, the Secretariat assigns a risk rating-low, medium, high, or very high-to each of these categories as well as an overall risk rating to the submission. The risk ranking is discussed by Secretariat senior management at the strategy meeting and adjusted if required. It serves to frame Secretariat recommendations and advice to Treasury Board ministers.
The Treasury Board submission process is one element in a larger process that involves federal organizations obtaining approval to undertake new initiatives or to continue with existing ones. The following description of this larger process is simplified and does not pretend to cover all cases; however, it is adequate for setting the Treasury Board submission process within the larger context.
Federal organizations undertake their own process of policy development through research and consultation, both internally and interdepartmentally, and ensure that any proposed initiative supports one or more of its substantive policy objectives and is consistent with its mandate. At the end of this developmental process, the sponsoring minister seeks approval for the new initiative. The primary document used for this purpose is the MC. The MC is the vehicle through which the minister proposes and explains the organization's new initiative to Cabinet, including related options and considerations, and obtains its approval.
The MC gives Cabinet an overview of the proposed new initiative's objectives and financial implications and of the links between the recommended course of action and the government's policies and objectives. It includes a communications plan. Central agencies such as the Privy Council Office (PCO), the Department of Finance Canada, and the Secretariat all have roles to play in the MC process.
PCO is responsible for the MC process and it advises the federal organization on, among other things, whether the aims of the initiative described in the MC are consistent with government priorities. The Department of Finance Canada supports PCO by providing advice and guidance on the MC's fiscal information to ensure that it fits with the government's fiscal priorities. The Secretariat, in its supporting role, helps ensure that the implementation requirements are realistically stated, that the MC includes appropriate accountability and transparency provisions, that it is consistent with Treasury Board policy, and that it makes appropriate linkages between the proposal and other existing programs and federal organizations.
A Cabinet policy committee considers the MC and recommends whether the proposal should be referred to Cabinet for ratification. The MC may either be fully approved as presented, approved in principle, postponed pending clarification, or approved with changes. After the MC has been ratified by Cabinet, the sponsoring federal organization develops a Treasury Board submission to obtain the specific authorities10 needed to implement Cabinet's decision.
The fundamental distinction between an MC and a Treasury Board submission is that, while an MC focuses primarily on the policy rationale and overall funding for a new policy or program initiative, the Treasury Board submission provides details on program design, specific costs, expected results and outcomes, and program delivery and implementation. The Treasury Board submission transforms policy rationale and objectives into a program that will achieve those objectives. It details how the federal organization will carry out the policy initiative; why the proposed method of implementation is the best one; how the proposal ensures accountability and transparency; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS); and how the federal organization will conduct monitoring, performance measurement, and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets. The Treasury Board submission may also seek approval for expenditure authority, contract authorities, or transfer payment authorities.
The scope of this evaluation was limited to the current submission process and did not include an assessment of the process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation; however, this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore.
The evaluation covered all steps of the current submission process involving the Secretariat:
The evaluation did not include federal organizations' internal processes for preparing Treasury Board submissions. Furthermore, the evaluation covers only "Part A" submissions. MCs, GIC submissions, and the federal budget, though outside the scope of the evaluation, were considered to the extent that they provided contextual information.
The evaluation's planning and data collection phases were already completed at the time Canada's Economic Action Plan was announced in late January 2009. As a result, implementation of this stimulus package could not be included within the scope of the evaluation.
Between 2004 and 2009 (September to August calendar), an average of 804 Treasury Board submissions were made each year. It should be noted that the Secretariat itself usually accounts for the largest number of Treasury Board submissions among all federal organizations given that it acts as both a department and the management board of the federal public service. As such, some of the Secretariat's submissions are to seek approval or amendments to Treasury Board policies or are related to Treasury Board's role as employer.
As shown in Table 1, the Secretariat's program sectors, with the exception of the International Affairs, Security and Justice Sector, reported a higher incidence of "standard" submissions than "complex" ones.12 Though in the minority, figures for complex submissions are nevertheless significant.
Program Sector | % Standard | % Complex |
---|---|---|
Social and Cultural | 54.9 | 34.3 |
Economic | 48.7 | 38.1 |
International Affairs, Security and Justice | 41.7 | 53.2 |
Government Operations (program side) | 66.0 | 29.9 |
Overall Average | 54.2 | 38.4 |
Evaluation participants described standard submissions as follows:
Complex submissions were said to involve one or more of the following:
No logic model had previously been developed for the Treasury Board submission process. A logic model is essential to this (and to any) evaluation, because it describes the consensus among key stakeholders on the intended outcomes of the Treasury Board submission process. A consensus view on outcomes is the touchstone against which the evaluation must be conducted. Relevance, effectiveness, and economy cannot be assessed in a vacuum; rather, these terms only have meaning in relation to the ultimate outcome of the submission process.
It was therefore necessary for the evaluation team to develop a logic model that would provide the necessary foundation for the evaluation. Consultations were held to gather information on the intended outcomes of the Treasury Board submission process. The evaluation team interviewed 19 individuals from areas across the Secretariat who deal with Treasury Board submissions, including analysts and senior analysts, directors, and an assistant secretary. In addition, two working sessions were held with Secretariat analysts to develop and validate the logic model and evaluation matrix that had been developed by the evaluation team. The consensus among this group was that the ultimate outcome of the submission process was to support Treasury Board in making decisions that contributed to ensuring that programs and spending are aligned with the government's priorities.
Consultation with senior management provided another perspective. Participants in the Assistant Deputy Minister (ADM) Working Session observed that ensuring alignment of new initiatives with the government's priorities is the intended outcome of the MC process rather than the Treasury Board submission process. They suggested that the purpose of the Treasury Board submission process was to provide federal organizations with the resources and authorities needed to achieve the social and economic outcomes desired by Cabinet and to ensure that the programs delivered demonstrated value for money. This proposed ultimate outcome appears to be consistent with the Secretariat's documentation and communications about its expected results. The statement is consistent with the Secretariat's strategic outcome: "Government is well managed and accountable, and resources are allocated to achieve results." This ultimate outcome is also consistent with the expected results outlined in the Secretariat's 2008-09 Departmental Performance Report (DPR): "allocation of resources so that they are aligned with the government's priorities and responsibilities, thereby ensuring that federal programs are effective and efficient and provide value-for-money." The link to broader outcomes (e.g. social and economic) is captured in the DPR's reference to "alignment of government's priorities and responsibilities."
Given that the Treasury Board submission process is not limited to either a single Secretariat subactivity or one organization within the Secretariat, instead crossing a number of organizations, including the program sectors, EMS, and many policy Centres of Excellence, it is perhaps not unexpected that a logic model for the process did not exist before the evaluation and that the ultimate outcome of the Treasury Board submission process was therefore not defined. That said, the evaluation found that there is a sound basis for articulating the outcome, particularly from the strategic outcome and expectations articulated in the Secretariat's DPR. The need for a final logic model, including the ultimate outcome, is addressed in the Conclusion section of this report.
For the purposes of the evaluation, outcomes of the Treasury Board submission process were based on analysis of the information gathered from the consultation sessions as well as information found in documentation on the submission process. The Guide was examined, and while it is very detailed in terms of how to do Treasury Board submissions, it is less explicit about why they must be done. Nevertheless, it was possible to make inferences about the ultimate outcome based on other statements found in the Guide. The Guide's clearest statement about the ultimate outcome of the submission process is as follows:
The Treasury Board submission transforms policy rationale and objectives into a program that will achieve those objectives. It details how the federal organization will carry out the policy /initiative; why the proposed method of implementation is the best one; how the proposal contributes to government-wide aims such as accountability, transparency, and interoperability of information; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS);13 and how the federal organization will conduct monitoring and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets.
Although reference may be made to the MC and Cabinet decisions, it is generally unnecessary to repeat the policy rationale in a submission. The submission must nevertheless be consistent with the Cabinet decisions.14
In other words, questions of policy rationale and intended social and economic outcomes (or "objectives") are dealt with in the MC (as was suggested by participants in the ADM Working Session). The Treasury Board submission deals with more detailed questions of program design and implementation. It is therefore reasonable to infer that the ultimate value or outcome of the submission process has to do with ensuring that the initiative in question is well designed in relation to the policy rationale described in the MC and in terms of its implementation arrangements and its reporting, accountability, and control mechanisms. The evaluation team therefore proposed the following as the ultimate outcome for the Treasury Board submission process:
Program implementation serves the policy outcomes defined by Cabinet in the most economical manner possible and satisfies the government's obligations for transparency, accountability, and prudence.
It was not possible to validate this proposed ultimate outcome with an appropriate cross-section of stakeholders. The evaluation team therefore recognizes that it is simply a place holder for the purposes of the evaluation.
On the other hand, the other elements of the logic model proposed by the evaluation team-the immediate and intermediate outcomes and the outputs-do reflect the common view within the Secretariat.
The involvement of the Secretariat in the submission process consists of six major components, as outlined in the logic model developed by the evaluation team for the purposes of the evaluation (Appendix A; see also Figure 1):
The activities identified in the middle row of Figure 1 should lead to the outcomes indicated in the logic model developed by the evaluation team:
The foregoing outcomes are expected to result in high-quality Treasury Board submissions. These in turn are expected to contribute, ultimately, to federal organizations' effective implementation of programs that serve the government's desired policy outcomes in the most efficient possible manner while also fulfilling the government's obligations for transparency, accountability, and prudence.
Table 2 presents the evaluation issues and questions that were derived from the logic model developed by the evaluation team. The detailed evaluation matrix, which outlines the indicators and data sources used to address the evaluation questions, is presented in Appendix B.
Evaluation Issue | Evaluation Questions |
---|---|
Relevance |
|
Effectiveness |
Short Term
|
Economy |
|
The evaluation framework uses multiple lines of evidence and complementary research methods to ensure the reliability and validity of the data. The following research methods were used:
Each of these methods is described in more detail below.
Document review. Three main types of documents were reviewed during the evaluation:
For the full list of documents reviewed, please see Appendix C. Note that the evaluation did not review or assess Treasury Board submissions and précis for quality.
Interviews. Twenty-six interviews were completed (Table 3 and Appendix D). Interviewees included
Interview Group | Number of Interviews |
---|---|
Program analysts and COE analysts | 11 |
Representatives from federal organizations | 12 |
External stakeholders | 3 |
Total | 26 |
All interviews were conducted by telephone. Interviewees were sent an interview guide (see Appendix E) before the interviews were conducted.
Survey. A survey was administered over the Internet to program analysts, COE analysts, and representatives of federal organizations that put forward a Treasury Board submission in the last five years. A total of 547 individuals were asked to complete the questionnaire; 220 useable responses were received, for an overall response rate of 40% (see Table 4).19
Survey Group | Total Sent | Received | Removed | Total Kept | Response Rate | Confidence Interval |
---|---|---|---|---|---|---|
Program analysts | 135 | 60 | 0 | 60 | 44.4% | 95% ± 9.5% |
COE analysts | 181 | 66 | 0 | 66 | 36.5% | 95% ± 9.6% |
Federal organizations | 231 | 99 | 5 20 | 94 | 40.7% | 95% ± 7.8% |
Total | 547 | 225 | 5 | 220 | 40.2% |
All of the Secretariat's program analysts were invited to participate in the survey. They were also asked to provide contact information for all COE analysts they had consulted for advice on Treasury Board submissions during the last five years. Furthermore, the program analysts were asked to provide contact information for the individuals in federal organizations (including the Secretariat) that put forward a submission within the last year. Federal organizations were encouraged to forward the survey to any individual within the organization who had been involved in the Treasury Board submission review process.
Survey results are provided in Appendix G.
Working session with ADMs. A two-hour working session was conducted to gather qualitative information on the relevance, effectiveness, and economy of the Treasury Board submission process. Assistant secretaries from the Secretariat and selected ADMs responsible for corporate and/or strategic planning as well as selected departmental chief financial officers (CFO) from federal organizations were invited to participate. The issues to be discussed during the session (see Appendix F) were provided to the participants in advance.
Administrative, financial, and statistical data. Administrative, financial, and statistical data were gathered for the purpose of assessing the effectiveness and efficiency of the submission process. The evaluation team worked with the Treasury Board Submission Centre to review data related to Treasury Board submissions and to gain a greater understanding of the Submission Tracking System (STS). The evaluation team also reviewed data from the Management Accountability Framework (MAF) database.
Costing. Evaluating economy requires an analysis of the costs involved throughout the submission process. A costing exercise was undertaken; however, because direct costs related to Treasury Board submissions are not tracked separately, only the level of effort of some participants involved in the submission process was available. Section 6(c)(i), "Resources allocated to the submission process," therefore does not identify an estimated cost for the process.
Related international practice: The evaluation team undertook a limited review of submission process models used in other international jurisdictions.21 Given the cursory nature of this review, the evaluation team could not draw conclusions on the appropriateness of other models compared to the Canadian context. Findings from this review are therefore not presented in this report, though some interesting information was discovered during the review. For example, the role played by the Secretariat's assistant secretaries-whereby they present a federal organization's submission to Treasury Board-may be unique internationally. As is the case in the Canadian federal model, Secretariat equivalents in other jurisdictions are responsible for logistical and technical functions related to sessions of Cabinet, strategic and work planning, policy advice, legal functions, some monitoring functions, and their own internal management functions.22 They scrutinize material presented to Cabinet, ensuring that legal and policy considerations have been accounted for within structured submissions. In six of ten surveyed Organisation for Economic Co-operation and Development (OECD) countries,23 the Secretariat equivalent prepares a recommendation on how the submission should be handled in the Cabinet-level meeting. In these jurisdictions, however, it is the deputy minister of the submitting organization who presents the submission at the Cabinet-level meeting and not the equivalent of a Secretariat assistant secretary. Assistant secretary equivalents can therefore focus their attention on submissions that are most strategic or sensitive or for which their recommendation runs counter to that of the submitting organization.
Other information. Once the data were collected and analyzed, information gaps were discovered in a few key areas. To fill these gaps, the following methodologies were used:
Time frame. A federal election was called shortly after the evaluation was launched. Not long after the election, Parliament was prorogued. These events delayed approval for the opinion research to be conducted for the evaluation; consequently, the time available to perform the research was limited. Another consequence of the election and subsequent prorogation was that the evaluation team could not interview Treasury Board ministers regarding the support they receive through the Treasury Board submission process.
Logic model. While the evaluation team believes that the ultimate outcome proposed in the logic model it developed is a valid description of the purpose of the Treasury Board submission process and therefore a valid basis for the evaluation, it was not possible to validate the ultimate outcome with an appropriate cross-section of stakeholders.
Review of performance measurement data. Performance data have not been collected on all aspects of the Treasury Board submission process. For instance, data are not collected on the extent to which the Treasury Board submissions officially submitted by federal organizations are actually required, or their compliance with policies and processes, and on the extent to which Treasury Board decisions reflect recommendations in the précis. In the absence of this information, the evaluation team was unable to assess the quality of submissions and précis. The evaluation therefore relied more heavily on survey and interview data to assess the effectiveness of the process.
Administrative data. Limited administrative data were available on the submission process and its results. Furthermore, as noted in the Treasury Board of Canada Secretariat Audit of Leave and Overtime (2008),24 some overtime data are not reliable, thereby limiting the extent to which the evaluation team could use such data to assess the amount (and related costs) of overtime claimed by program analysts in connection with Treasury Board submissions. In addition, other administrative data such as the number of days between the Treasury Board decision and the issuance of the decision letter would have provided additional lines of evidence.
Surveys and interviews. Given the highly variable nature of Treasury Board submissions, it stands to reason that the submission process experience would differ significantly from one submission to the next. More interviews would have provided better data on the effect of variations in a submission's size, scope, value, and complexity. The evaluation methodology attempted to address this limitation by inviting individuals from all federal organizations to participate in the Web-based survey. The results of the survey were cross-validated with the interview responses.
Costing methodology. Cost information to support a complete and accurate costing of the Secretariat's involvement in the Treasury Board submission process was not available.
Single-window approach to service delivery. Because the Secretariat uses a single-window approach for submissions, in which program analysts and their directors or executive directors are the point of contact for representatives from federal organizations, interaction between COE analysts and representatives from federal organizations is extremely limited. As a result, despite COE analysts having a clear role and contribution at the pre-submission and draft stages of the submission process, representatives from federal organizations may not be fully aware of the extent of this role. This may have led to the evaluation's greater focus on program analysts.
While there are some limitations with the evaluation methodology, multiple lines of evidence were used to draw conclusions about the Treasury Board submission process, strengthening the reliability and validity of the evaluation results. Despite the limitations, the methodology meets the requirements of the Treasury Board Policy on Evaluation and associated standards.
Does the Treasury Board submission process address a demonstrable need, and is it appropriate to the federal government?
The Treasury Board submission process aligns well with the Secretariat's strategic outcome: "Government is well managed and accountable, and resources are allocated to achieve results."25 This strategic outcome is consistent with the responsibilities of Treasury Board ministers as set out in the FAA and therefore speaks to the role of the Secretariat in providing advice and recommendations to Treasury Board ministers through the Treasury Board submission process. The need for a Treasury Board submission process to support ministers is particularly relevant, given the recent focus on responsible spending and the renewal of the expenditure management system, which includes the strategic review of the direct program spending of all departments and agencies of the Government of Canada to reallocate funds from low-priority and low-performing programs to higher priorities.
The Guide states the following:
Improving the quality of information and accountability for results are key elements of the new approach to managing spending across government. The new approach supports managing for results by establishing clear responsibilities for departments to better define the expected outcomes of new and existing programs. It supports decision making for results by ensuring that all new programs are fully and effectively integrated with existing programs and by reviewing all spending to ensure efficiency, effectiveness, and ongoing value for money. Finally, it supports reporting for results by improving the quality of departmental and government-wide reporting to Parliament.26
This excerpt describes the "demonstrable need" that the Treasury Board submission process is meant to address. Essentially, for good management of public programs, the government requires rules, practices, and processes that:
In assessing relevance, the key evaluation issue is therefore whether or not the Treasury Board submission process is capable of meeting these requirements. The question of whether it actually meets these requirements is addressed under the assessment of effectiveness. The question of whether, and to what extent, other processes or mechanisms would be capable of meeting these requirements is outside the scope of this evaluation.
The documents reviewed for this evaluation, as well as the interviews conducted, point to the conclusion that the Treasury Board submission process is relevant to the government's "demonstrable needs" for good public management. As noted above, the submission process aims to ensure that minimum standards are met with regard to:
how the federal organization will carry out the policy initiative; why the proposed method of implementation is the best one; how the proposal contributes to government-wide aims such as accountability, transparency, and interoperability of information; what the expected outcomes and deliverables are, as per the organization's Management, Resources, and Results Structure (MRRS); and how the federal organization will conduct monitoring and evaluation to ensure the program is meeting its policy objectives. This includes progress reports on outcomes, projected efficiency, timelines, and cost targets.27
According to the interviews, general perceptions about the purpose of the submission process broadly correspond to this formal description from the Guide.
This formal description of the Treasury Board submission process is well aligned with the government's stated needs for good public management.
To what extent has the Treasury Board submission process achieved its expected outcomes?
Ideally, this evaluation would assess the effectiveness of the Treasury Board submission process in relation to its impact on the ultimate outcome in the logic model that the evaluation team developed for the purposes of the evaluation, i.e. "program implementation serves the policy outcomes defined by Cabinet in the most economical manner possible and fulfills the government's obligations for transparency, accountability, and prudence." However, attempts to link the Treasury Board submission process directly to this ultimate outcome would face data-gathering and methodological challenges that could not be addressed within the evaluation's scope and its time and resource constraints. The evaluation's assessment of the effectiveness of the Treasury Board submission process is therefore focused on the immediate and intermediate outcomes specified in the logic model developed by the evaluation team. If the evaluation finds that the Treasury Board submission process is generally effective in contributing to these outcomes, then it would be reasonable to conclude that the submission process is also making a significant contribution to the ultimate outcome.
The evaluation therefore focused on the following questions:
The majority of survey participants in all three categories (83.3% of program analysts, 76.9% of COE analysts, and 83% of representatives from federal organizations) believed they had a strong understanding of the Treasury Board submission process. There was no correlation between the length of time respondents had been in their job and their perception of their own understanding of the process.
Survey respondents rated their counterparts' understanding of the Treasury Board submission process less favourably than they rated their own (Table 5). For example, 83% of federal organization respondents believed they had a strong understanding of the process, whereas less than 50% of the program analysts surveyed agreed that this was true. There was a similar divergence of opinion between program and COE analysts within the Secretariat. Each group identified a need for more training for the other.
% Agreed | Program Analysts | COE Analysts | Federal Organization Respondents |
---|---|---|---|
I have a strong understanding of the Treasury Board submission process. | 83.3 n=60 |
n/a | n/a |
I have a strong understanding of the elements of Treasury Board submissions, policies, and processes. | n/a | 76.9 n=65 |
83.0 n=94 |
The federal organizations I work with have demonstrated, over the years, an increased understanding of the elements of Treasury Board submissions, policies, and processes. | 48.3 n=58 |
n/a | n/a |
Program analysts I work with have demonstrated, over the years, an increased understanding of the elements of Treasury Board submissions, policies, and processes. | n/a | 50.0 n=62 |
n/a |
Notwithstanding the above, it should be noted that among the 12 interviews conducted with federal organizations, all 4 federal organizations with high submission rates believed that their analysts had a strong understanding of the Treasury Board submission process. Results were mixed among federal organizations with moderate and low submission rates; they indicated that the level of understanding varied depending on the analyst assigned to them.
HR and survey data showed that approximately two-thirds of program analysts had been in their position for two years or less, which may at least partially explain the external perceptions about program analysts' understanding of the submission process.
Training. To assess the Secretariat's tools, support, and services, the evaluation team examined documentation on the boot camps held for program analysts and its guidance document on Treasury Board submissions, A Guide to Preparing Treasury Board Submissions. Furthermore, the evaluation team asked whether courses on the Treasury Board submission process were taken from the Canada School of Public Service.
The Secretariat offers program sector boot camps, which have received positive overall ratings and are considered useful in terms of providing a general overview of what program analysts should know. However, in the participant feedback forms that are completed following the boot camps28 and in interviews, Secretariat analysts indicated that the boot camps are not offered often enough and their duration (two days) does not allow for sufficient coverage of the Treasury Board submission process. As one boot camp participant noted, "could… have more hands-on training."
Evaluation participants also indicated that:
Relationship between Secretariat analysts and federal organizations. Survey results demonstrated that efforts are being made to foster a positive working relationship between program analysts and federal organizations. Almost all of the program analysts surveyed (91.6%) reported that they maintain regular contact with federal organizations regardless of whether a submission is currently being developed or processed. All of the program analysts interviewed stated that they have positive and productive relationships with federal organizations. It is worth noting, however, that when asked how they would characterize this relationship, Secretariat analysts did not refer to the three roles-enabler, challenger, and champion-defined in the Change Agenda.
Federal organizations, for their part, indicated that their relationships with program analysts were generally good or had improved. Nearly all federal organization respondents said they knew whom to consult at the Secretariat with respect to their submissions.
Appropriateness29 of Treasury Board submissions put forward. Secretariat analysts have a role to play at the pre-submission stage when federal organizations are considering proceeding with a submission. At the outset, federal organizations may want early feedback to confirm the directsion of their intended submission or to confirm that a submission is in fact necessary. According to the interviews, 58% of federal organizations seek pre-submission assistance, especially when the submission is complex or perceived as higher risk.30 Advice provided by the Secretariat at the pre-submission stage is intended (among other things) to help federal organizations make informed decisions about whether or not to put forth a submission. The assumption is that in the absence of such advice, federal organizations would proceed with a greater proportion of unnecessary submissions, creating a burden on the Secretariat and Treasury Board and reducing the efficiency of the submission process. The appropriateness of submissions was therefore regarded as an indicator of the suitability and quality of the services the Secretariat provides at the pre-submission stage.31
Secretariat analysts were asked about the appropriateness of submissions (Table 6 32).33 Approximately 40% of the program analysts surveyed responded that they received draft submissions at the pre-submission stage (for preliminary guidance) for initiatives that did not in fact require a submission. The views of COE analysts were similar.
Was the Secretariat consulted prior to sharing? | COE analysts | Program analysts |
---|---|---|
n = sample size | n=29 | n=24 |
Yes | 44.8 | 62.5 |
No | 10.3 | 25.0 |
Do not know | 44.8 | 12.5 |
Were any drafts submitted? | COE analysts | Program analysts |
---|---|---|
n = sample size | n=65 | n=58 |
Yes | 10.8 | 19.0 |
No | 36.9 | 55.2 |
Do not know | 52.3 | 25.9 |
Secretariat analysts were also asked how often federal organizations put forward draft submissions for formal consideration (as opposed to simply seeking informal feedback on a preliminary draft). At this stage, a much smaller proportion of analysts (19% of program analysts and 10.8% of COE analysts) felt that inappropriate drafts were submitted. The reported drop in unnecessary submissions between the stage at which the Secretariat is providing preliminary guidance and the stage at which a submission is formally put forward suggests that the Secretariat is having a positive impact on reducing the number of inappropriate submissions.
Accuracy, consistency, usefulness, and timeliness of Secretariat advice. Interviewees identified factors they believed had a positive or negative impact on the consistency, accuracy, and timeliness of the advice the Secretariat provides to federal organizations during the submission process.
Factors seen by interviewees as having a positive impact:
Factors seen by interviewees as having a negative impact:
Survey respondents were asked whether the submission review process ensures that Treasury Board submissions comply with government authorities and policies. A large majority of respondents from each of the groups agreed that it did (Table 7).
However, when survey respondents were asked whether the services provided by analysts enable federal organizations to put forth draft submissions that comply with Treasury Board authorities, policies and directions, there was a divergence of opinion. The federal organization representatives and program analysts had very similar agreement percentages to the ones for the above question and as shown in Table 7, whereas only 49.2% of COE analysts agreed. This divergence could be a function of program analysts acting as the single point of contact for federal organizations and COE analysts not always being consulted at the pre-submission stage.
Secretariat advice and guidance to federal organizations. Survey results (Table 8) show that the majority of survey respondents from federal organizations agreed that Secretariat analysts were providing consistent, accurate, useful, and timely advice. For those who disagreed, consistency and timeliness of advice had the lowest levels of agreement for both representatives of federal organizations and program analysts. Specifically:
It is important to note that the issues around timeliness and consistency of advice are recognized internally by the Secretariat, not only by federal organizations.
Consultations with senior management highlighted the value of the Secretariat's work with federal organizations during the pre-submission stage. However, the following areas were identified as needing improvement:
Adequacy of time for input into submission documents. The time required to complete the submission review process can vary greatly depending on a submission's characteristics. The Guide advises federal organizations to allow at least six weeks34 for the submission process and cautions that it is not unusual for the process to last more than six weeks. Survey results (Table 9) indicate that the process normally takes 8 to 10 weeks. This suggests that the Guide sets up unrealistic expectations, which could lead to a frustrating experience for someone who is new to the process.35
Secretariat analysts were asked for their views on whether they had enough time to review draft submissions properly. While not a majority, a significant number of program analysts and COE analysts (42.5% and 49.2%, respectively) said that they did not have enough time.
Participants in the ADM Working Session argued that the fast-track system used by the Secretariat to give priority handling to certain submissions is another factor affecting the time available to Secretariat analysts to review submissions.36 They said that it causes uncontrollable delays for submissions that are not fast-tracked. In discussions with Secretariat senior management, however, it was noted that there is no formal system for fast-tracking submissions. It is nevertheless true that certain submissions may receive priority treatment on an ad hoc basis due to urgent situations that require them to "jump the queue." It is at the discretion of the President of the Treasury Board to determine when a submission should receive priority handling; the decision is normally taken following negotiation with the minister responsible for the submission. Consultations between Secretariat officials and sponsoring federal organizations regarding time sensitivity of their proposals are held regularly; however, decisions regarding priority handling are not normally shared with the federal organizations whose submissions may be displaced as a result.
Extent to which Secretariat analysts' input is reflected in final Treasury Board submissions. Most Secretariat analysts stated that their advice is included in final submissions that go forward to Treasury Board. Some noted that, if their advice was not included, they might recommend that conditions be placed on the submission or a remark be included in the précis. Survey results were similar, with 86.6% of program analysts and 71.2% of COE analysts responding that they felt their advice was reflected in the final submission.
This was confirmed with federal organization representatives, who stated that they include all or almost all of the comments provided by Secretariat analysts. Some noted that if they are in disagreement with the advice, they consult further to resolve the issue. A small minority (18.0%) of federal organization representatives stated that they include Secretariat comments due to time pressures and the perceived power of the Secretariat and not because they are in agreement with them.
Extent to which Treasury Board final decisions reflect Secretariat recommendations. Once the program analyst is satisfied that the submission is complete, he or she prepares a précis that includes recommendations to Treasury Board. The recommendations are discussed and agreed to at the Secretariat's Strategy Committee, which is chaired by the Secretary and includes the participation of assistant secretaries from across the Secretariat. A presentation based on the précis is then made to Treasury Board by the appropriate assistant secretary. Although the Secretariat has no authority over Treasury Board's decisions, the decisions normally reflect its recommendations. Data from the survey of program analysts suggest that Treasury Board decisions are consistent with the Secretariat's recommendations 82.2% of the time.
Information management. Once Treasury Board makes a decision, the Secretariat records the decision and formally advises the deputy minister of the sponsoring organization within fifteen calendar days. Although program analysts have no formal requirement to do so, they will normally advise the federal organization verbally of the Treasury Board decision as a matter of courtesy. This is usually done as soon as possible, generally the day after the Treasury Board meeting.37 While most federal organizations did not have an issue with the timeliness of the communication of Treasury Board decisions, it should be noted that one-third of federal organization representatives did not agree that Treasury Board decisions are communicated to them in a timely manner.
Of the program analysts and COE analysts interviewed, many indicated that they use their own private filing system for Treasury Board submissions. Most keep a hard copy of the decision for a period of time, after which the documents are archived or sent to the Treasury Board Submission Centre. Some analysts did mention using the Records, Documents and Information Management System (RDIMS),38 but most program analysts stated that RDIMS is difficult to use. Senior managers observed that a central repository for submission-related information would greatly facilitate their work.
The interviews and survey results also indicated that there is no formal system in use within the Secretariat for tracking conditions attached to Treasury Board submissions.
It should be noted that the Submission Tracking System (STS) maintained by the Treasury Board Submission Centre produces the agenda for Treasury Board meetings and indicates whether or not a submission has conditions attached to it. One interviewee observed that there are weekly requests to the Treasury Board Submission Centre from program analysts for background information on Treasury Board submissions, which indicates the potential value of a central repository of information on Treasury Board submissions. There is also the potential to use the system to track the implementation of conditions.
The Secretariat had developed the Expenditure Management Information System (EMIS) in part to ensure a single set of timely, complete, consistent, and accurate financial and non-financial data and a standard, automated, end-to-end expenditure management process. This was to allow for increased monitoring, tracing, and tracking of information, which would in turn support improved analysis and decision making. The expected result was to be more effective management of the government's supply and budget process.39
When the Budget Office Systems Renewal (BOSR) Project was launched (second phase of EMIS), features were developed to accommodate program sector needs with respect to Treasury Board submissions, including the tracking of conditions. These features, however, have not as yet been implemented. Program sectors maintain that these features did not meet their operational needs. Subsequent interviews suggested that this may be due in part to the fact that the Secretariat's program sectors have not fully analyzed and mapped their own business processes.
As indicated in the logic model developed by the evaluation team, the following assumption underlies the Secretariat's involvement in the submission process as it currently exists: awareness and education tools combined with sound advice to federal organizations should contribute to the quality of Treasury Board submissions and their quality should steadily improve over time.
Overall changes in quality of submissions. Program analysts were asked how the overall quality of Treasury Board submissions had changed during their time with the Secretariat (Table 10). While half of the respondents indicated that the quality had not changed, a little more than one-third of the program analysts (37.3%) stated that the quality of submissions had improved and about one in ten (11.9%) observed that their quality had declined. About half of the program analysts noticed no significant change. However, the usefulness of this data is limited, given that most of the Secretariat analysts surveyed had been in their job for no more than two years. Also, it should be noted that the evaluation did not review Treasury Board submissions and related précis for quality.
Management Accountability Framework rating. The Management Accountability Framework (MAF) is structured around ten key elements that set out Treasury Board's expectations of senior public service managers for good public service management.40 Under one of these key elements, "Policy and Programs," the Secretariat assesses the "quality of analysis in Treasury Board submissions" according to the following criteria:
Federal organizations' MAF ratings were analyzed for MAF Rounds IV and V. MAF assessments indicate that the overall quality of analysis in Treasury Board submissions increased from Round IV to Round V. Although performance improved in relation to items 1, 2, and 4 cited above, performance with respect to the timeliness of consultations with the Secretariat actually decreased, which supports earlier findings on the adequacy of time available for input. Again, however, the usefulness of this data is limited, given that two years of results were available at the time of data collection and that the rating criteria were amended slightly for Round V.41
Secretariat analysts' views on submission quality. For the purposes of this evaluation, Secretariat analysts identified what they believed to be the key criteria of a high-quality Treasury Board submission. They cited the following:
As seen in Table 11, a significant number of Secretariat analysts, though not an overwhelming majority, indicated that submissions did not meet these criteria. This supports earlier findings on the divergence of opinion with respect to Secretariat analysts and federal organizations having a strong understanding of the submission process (see Section 6(b)(i)).
Average Rating (%) | Program Analysts | COE Analysts | Federal Organization Respondents |
---|---|---|---|
n = sample size | n=59 | n=56 | n=85 |
Appear to have completed internal42 consultations | 57.3 | 55.1 | 83.1 |
Were submitted with enough time for Secretariat analysts' review | 57.5 | 50.8 | 83.0 |
Followed the Guide to Preparing Treasury Board Submissions | 56.7 | 62.7 | 83.7 |
Risk and risk mitigation strategies identified | 59.5 | 52.5 | 76.4 |
Asked for the right authorities | 60.2 | 61.9 | 90.3 |
Had an appropriate level of justification | 54.7 | 55.9 | 89.8 |
Contained accurate information (e.g. accurate financial tables) | 54.2 | 54.3 | 90.8 |
Were written in a clear manner | 51.4 | 60.6 | 88.3 |
Does the Treasury Board submission process consume the minimum amount of resources required to achieve its expected outcomes?
As indicated in the Scope section, this evaluation was limited to the current submission process used to support Treasury Board ministers and did not assess this process against potential alternative mechanisms for providing this support. A limited review of international practices was conducted as part of the evaluation, however this review was not sufficiently in-depth to suggest alternative mechanisms for the Secretariat to explore. The question of economy therefore only considered the resources required to achieve the expected outcomes within the context of the current process. A comparison of alternative mechanisms or processes and the estimated cost of those processes was not undertaken.
As noted earlier in the report, the cost of the Treasury Board submission process within the Secretariat could not be established. Nevertheless, it is clear through the widespread level of effort and interest in submission-related activities that the process represents one of the key functions of the Secretariat. The evaluation can establish through interview evidence that Secretariat senior management sees the Treasury Board submission process as a core function. In line with this finding, the level of effort devoted to the function within the program sectors and Centres of Expertise is significant. There are approximately 135 program analysts within the Secretariat acting as the single window of advice to federal organizations and supporting Treasury Board ministers in their submission-related decision making. According to their survey responses (60 responses), program analysts devote, on average, 73% of their time to Treasury Board submission-related activities. Acting as that single window, program analysts in turn consult with COE analysts to provide coordinated advice to federal organizations and in support of advice to Treasury Board ministers. In the past 5 years, program analysts have consulted 181 COE analysts. According to their survey responses (66 responses), COE analysts estimate that, on average, 60% of their time is spent on Treasury Board submissions.43 A separate study examining the full cost of the submission process may well be warranted.
The proportion of more senior level analysts involved in the Treasury Board submission process has declined in recent years. In 2004, 60% of program sector ESs were ES-06s or ES-07s, compared to approximately 50% in 2008. While the proportion of ES-01s, 02s, and 03s involved in the process has remained relatively constant, the proportion of ES-04s and 05s has risen from approximately 32% to approximately 43% over the same period (as depicited in Figure 2). Note that these statistics do not take into account senior management efforts throughout the process.
Figure 2: Change in Proportion of Junior to Senior Staff
How analysts spend their time. Program analysts report spending on average 72.6% of their time working on activities related to Treasury Board submissions. As shown in Table 12,44 the reported time spent on pre-submission advice and consultation is relatively modest. This provides some corroboration of senior management's comments that inter- and intradepartmental communication is not adequate at the beginning of the submission process.
COE analysts report spending on average 60.3% of their time on activities related to Treasury Board submissions, with the majority devoted to reviewing and consulting on submissions. It should be noted, however, that program analysts are responsible for reviewing the entire submission document, while COE analysts normally review only specific sections.
A significant number of survey respondents believe that the submission process is "inefficient" (Table 13 45).
The following factors were specifically identified as having an impact on the efficiency of the process:
Further research into workload peaks would be beneficial, especially as they appear to occur on a regular and cyclical basis that aligns with the government expenditure cycle and reporting cycle. Also, it is an issue over which the Secretariat, as the administrative arm of Treasury Board, may have some influence.
Federal organizations provide material in their Treasury Board submissions that enable the Secretariat to perform risk assessments. Risks are discussed during the development of the Treasury Board submission, but the ratings are established once the submissions are complete. Risk ratings are included in the précis, and they may have an impact on the amount of time the submission is allocated at the Treasury Board meeting.
Secretariat analysts and the staff of federal organizations commonly suggest that treating low-risk submissions differently would reduce the level of resources consumed by the submission process (Table 14). They generally argue that the risk rating should be established earlier in the submission review process and that the amount of time allocated and scrutiny given to a submission should correspond to the amount of risk the submission presents. As it is now, most submissions receive the same level of attention as they go through the submission review process, regardless of their level of risk or materiality.
Secretariat senior management expressed scepticism on this point, arguing that submissions that appear low risk at the beginning of the submission review process are sometimes perceived as high risk by the end of the process. As well, there may be risks that are of concern to Treasury Board ministers, which the Secretariat may or may not be familiar with. These may arise only when a particular submission is heard within the context of others on the Treasury Board agenda.
Though a large proportion of analysts (both at the Secretariat and in federal organizations) suggested that risk assessment be used to reduce the use of resources (see Table 14), Secretariat senior management clarified that risk assessment is intended to allow Treasury Board ministers to spend more time on proposals of higher risk and not to reduce workload. The results point to an opportunity to further explore how risk assessment can be used to enhance the efficiency of the process.
Other proposed measures for improving the efficiency of the current submission process that emerged during the evaluation's working sessions and interviews include the following:
In addition, evaluation participants proposed increasing the level of authority delegated to federal organizations as a measure for improving efficiency. Since this suggestion would require more fundamental changes to the mechanism used to support Treasury Board ministers, it is beyond the scope of the evaluation. However, it should be noted that the Secretariat, outside the scope of this evaluation, is currently conducting an "earned and risk-based delegation" project that is examining various approaches and options for potentially increasing the level of authority delegated to federal organizations.
The current Treasury Board submission process is relevant, appears to be effective, and has key strengths; however, some important opportunities exist to improve and enhance its effectiveness and efficiency, which should be addressed through a combination of action and further investigation.
Strengths:
Opportunities for improvement:
As noted earlier in the report, the evaluation team developed a proposed ultimate outcome to be used for the purposes of the evaluation. Finalizing and validating a logic model that includes a clear ultimate outcome would provide a solid basis for enhanced performance measurement activities and further study and research.
While final submissions presented to Treasury Board are considered to be of high quality, evidence regarding the quality of federal organizations' submissions at the beginning of the process is mixed. MAF ratings related to Treasury Board submissions improved in MAF Round V (with the exception of the timeliness of consultations with the Secretariat); however, fewer than 40% of the program analysts surveyed found that the overall quality of draft submissions had improved. A significant number of Secretariat analysts surveyed (in excess of 40% for most factors47) felt that draft submissions do not meet minimum quality standards.
This evaluation could not draw conclusions on the efficiency of the submission process given the absence of costing information. However, data show that approximately 40% of Secretariat analysts believe the process to be either somewhat or very efficient. The results were very similar among federal organization representatives.
Of the program analysts surveyed, 48.3% agreed that federal organizations have, over the years, demonstrated an improved understanding of the policies and processes related to Treasury Board submissions. As for training tools provided by the Secretariat, federal organizations consider the Guide to be useful. According to the interviews conducted and review of the documentation on program sector boot camps, it was found that the boot camps are not offered often enough and their duration (two days) does not allow for sufficient coverage of the Treasury Board submission process. Half of the COE analysts surveyed indicated that program analysts have developed a strong understanding of the submission process over the years.
This practice, compounded by turnover among Secretariat analysts, represents a gap in the management of Treasury Board submissions. Specifically, the Secretariat's ability to efficiently and effectively track workload changes, undertake higher level thematic analysis, forecast future demand, and access archived information when needed is impeded. There is also an increased risk of information loss. Although ensuring the security of the information would present its challenges, improvements to managing information related to Treasury Board submissions would enhance oversight and monitoring as well as the service provided to client departments.
Opportunities to improve the Treasury Board submission process fall into three areas:
The Secretariat should finalize the logic model for the Treasury Boad submission process to support performance measurement, clear communication of the process's ultimate outcome, and further study.
Since the ultimate outcome should not exist in isolation, the Secretariat should take the lead in refining, in consultation with federal organization representatives, its logic model for the submission process. Such a logic model would become the basis for:
The Secretariat should examine in detail, and address where necessary, survey results relating to advice its analysts provide to federal organizations during the submission process.
A majority of federal organization respondents agreed that the advice received from Secretariat analysts is timely and consistent. However, the number of respondents that did not agree is sufficient enough to warrant further work by the Secretariat to understand factors that may impede the timely and consistent delivery of advice.
The Secretariat should examine how human resources issues, especially the turnover rate among program analysts, affect the Treasury Board submission process and knowledge management.
Turnover among program analysts is likely to be an important factor underlying many of the issues noted in this evaluation. It is reasonable to assume, for example, that turnover would contribute to the reported levels of dissatisfaction among federal organizations with the services provided by program analysts.48
During the conduct of the evaluation, the Secretariat undertook work to further understand the factors contributing to turnover; a recommendation in this regard is therefore not made.
Notwithstanding steps that may be taken by the Secretariat to address turnover among program analysts, it is nevertheless reasonable to expect that turnover will persist for the foreseeable future. Therefore, other than simply seeking to reduce turnover, the Secretariat should explore ways to reduce its negative impact on the Treasury Board submission process. As noted below (Recommendation 6), implementing an effective system for knowledge and information management is an example of how the Secretariat could reduce the negative impact of turnover.
The Secretariat should account for the costs of managing the submission process-at least for the department if not for the entire public administration.
In the absence of complete and reliable cost information, the Secretariat cannot fully assess the extent to which its own resources are being properly and efficiently allocated to manage the submission process.
The Secretariat should review, and improve where necessary, its professional development, training, and outreach activities related to the Treasury Board submission process.
Program analysts' concerns about the capacity of federal organizations to produce high-quality submissions, along with the reported level of dissatisfaction from a minority of federal organizations with the advice received, point to a need for the Secretariat to review the professional development that it offers to its own staff and the training and outreach activities for the submission process that it extends to federal organizations.
In seeking to improve its professional development, training, and outreach activities, the Secretariat's course of action should be based on the first three recommendations noted above and the anticipated results that their implementation would bring. Finalization of a logic model for the Treasury Board submission process (Recommendation 1), together with a deeper understanding of federal organizations' dissatisfaction with the Secretariat's service (Recommendation 2), along with an awareness of how human resources issues affect the Treasury Board submission process (Recommendation 3) are likely to help the Secretariat understand how to improve the way it communicates with its own staff and with federal organizations regarding the submission process.
Given that the submission process has a conventional structure and that there are important similarities between certain submissions, implementation of an effective knowledge management system has the potential to yield significant improvements to the economy of the submission process. A systematic approach to knowledge management would create an archive of institutional memory related to Treasury Board submissions. This would allow Secretariat analysts to easily access and use the lessons learned and successful practices that resulted from the handling of previous submissions.
Creating an easily accessible bank of institutional memory related to the submission process would also help overcome some of the negative effects of the turnover rate among Secretariat analysts. A good knowledge and information management system would shorten the learning curve for new analysts and address some of the causes of federal organizations' complaints about "inconsistent" advice from the Secretariat.
Finally, a systematic approach to recording and retrieving information on the status of present and proposed submissions would facilitate the management of the submission process's workload, helping to smooth out the workload "crunches" that arise during peak submission periods.
The Secretariat should bear in mind two issues if it decides to explore options for a systematic approach to knowledge and information management.
Appendix A: Logic Model - Text versions
Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
---|---|---|---|---|---|---|---|---|---|---|---|
1.0 What need does the Treasury Board (TB) submission process fulfill? | 1.1 | Rationale for TB submission process (FAA sections 7-11, TBS RPP, Guide to Preparing TB Submissions) | Legal only | 1 | 1 | ||||||
1.2 | Alignment with relevant legislative requirements | 1 | |||||||||
1.3 | Comments and rationale for any changes in what is required to go to TB (e.g. changes in delegation of authority, changes in number of submissions) | 1 | 1 | 1 | 1 | ||||||
1.4 | Appropriateness of what is required to be submitted (TBS reached an appropriate balance, e.g. regarding different types of submissions, different transactions) | 1 | 1 | 1 |
Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
---|---|---|---|---|---|---|---|---|---|---|---|
2.0 Do federal organizations have an increased understanding of the elements of TB submissions, policies, and process? | 2.1 | Usefulness of guides, tools, and outreach provided by TBS | 1 | 1 | 1 | ||||||
3.0 Do TBS employees have an increased understanding of the elements of TB submissions and policies? | 3.1 | Usefulness of guides, tools, and internal courses | 1 | 1 | 1 | 1 | |||||
4.0 Is TBS offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions? | 4.1 | Accuracy/consistency of TBS's pre-submission guidance | 1 | 1 | 1 | 1 | |||||
4.2 | Appropriateness of TB submissions being put forward | 1 | 1 | 1 | |||||||
4.3 | Comparison of understanding of time required for TBS to review a submission | 1 | 1 | 1 | |||||||
4.4 | Timeliness - Number of late submissions | 1 | |||||||||
4.5 | Relationship of TBS analysts to federal organizations (Change Agenda: strategic partner and trusted advisor, proactive risk manager, overseer vs. enabler, challenge vs. facilitation) | 1 | 1 | 1 | 1 | 1 | |||||
5.0 Does TBS's submission review process ensure that TB submissions comply with government authorities and policies? | 5.1 | Accuracy, consistency, usefulness, and timeliness of TBS advice/consultation during submission process | 1 | 1 | 1 | 1 | |||||
5.2 | Adequacy of time provided for input into TB submission documents | 1 | 1 | ||||||||
5.3 | Confidence that input from TBS consultations is reflected in final TB submission documents | 1 | 1 | 1 | 1 | 1 | |||||
5.4 | Limited legal contestation of the submission | 1 | Legal only | 1 | |||||||
6.0 Is the TB submission process risk-based? | 6.1 | Extent of knowledge of existence of risk criteria | 1 | 1 | 1 | 1 | 1 | ||||
6.2 | Appropriateness of risk criteria (government-wide perspective, reflect ministers' concerns, reflect financial and non-financial risks) | 1 | 1 | 1 | |||||||
6.3 | Extent/consistency of use of criteria | 1 | |||||||||
6.4 | Extent that TB submission process, due diligence, précis, and decisions are impacted by risk rating | 1 | 1 | ||||||||
6.5 | Appropriateness of level of involvement in risk assessments (TBS vs. federal organizations) | 1 | 1 | ||||||||
7. 0 Are TB decisions well-informed and consistent with advice, guidance, and recommendations provided by TBS analysts? | 7.1 | Consistency of decision with recommendations, number and frequency of additional conditions applied by TB, number and frequency of challenges raised | 1 | 1 | 1 | ||||||
7.2 | Number of deferrals and requests for more information | 1 | |||||||||
7.3 | Confidence in recommendations provided (e.g. with respect to risk, cost implications for government) | 1 | 1 | ||||||||
8. 0 Are mechanisms in place to ensure TB decisions are carried out? | 8.1 | Existence/adequacy of condition tracking processes:
|
1 | 1 | 1 | 1 | |||||
8.2 | Timeliness/completeness of decision communication | 1 | 1 | 1 | |||||||
8.3 | Completeness/usability of TB submission filing system | 1 | 1 | 1 |
Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
---|---|---|---|---|---|---|---|---|---|---|---|
9.0 What is the level of quality of TB submissions? | 9.1 | MAF score element 5 | 1 | ||||||||
9.2 | Completeness of TB submission documents:
|
1 | 1 | 1 | 1 | 1 | 1 | ||||
10.0 Does the TB submission process contribute to ensuring departmental/government management, programs, and spending are aligned with Government of Canada priorities? | 10.1 | Documented accounts of TB submissions contributing to alignment (e.g. Auditor General reports, MAF, Results for Canadians, TBS DPR) | 1 | 1 | |||||||
10.2 | Explanations of whether or not TB submission process contributes to ensuring alignment | 1 | 1 | ||||||||
11.0 Are there any factors that are impeding the achievement of results? | 11.1 | Description of factors impeding achievement of results | 1 | 1 | |||||||
12.0 Have there been any unexpected outcomes? | 12.1 | Description of unexpected outcomes | 1 | 1 |
Evaluation Question | Individual Number | Indicators | Focus Group: Assistant Secretaries | Interviews: TBS Analysts / Internal Contacts (n=10) | Interviews: Federal Organizations (n=12) | Interviews: External Stakeholders ( n=5) | Web-Based Survey: Federal Organizations | Web-Based Survey: TBS Analysts | Web-Based Survey: TBS Internal Contacts | Data Analysis | Literature Review |
---|---|---|---|---|---|---|---|---|---|---|---|
13.0 What is the level of resource allocation to submission process? | 13.1 | Changes in level of TB submission effort 5 years ago relative to today | 1 | 1 | 1 | ||||||
13.2 | Trend analysis of number of submissions | 1 | |||||||||
13.3 | Estimates of time spent:
|
1 | 1 | ||||||||
13.4 | Identify rate, determine step and where resource increases would have most impact on process flow | 1 | 1 | 1 | 1 | 1 | 1 | ||||
13.5 | Factors impeding analysts' ability to contribute to submission process | 1 | 1 | ||||||||
14.0 Is the process efficient? What could be done to make the process more efficient? | 14.1 | Usefulness of measures taken as a result of the 2003 audit and the 2007 EXCO Retreat (Roadmap for enabling change) | 1 | 1 | 1 | 1 | 1 | 1 | |||
14.2 | Identification of any measures taken by TBS to manage workload (e.g. risk-based approach to submission review, delegation of authority) | 1 | 1 | 1 | 1 | 1 | |||||
14.3 | Identification of any areas where efficiency can be improved | 1 | 1 | 1 | 1 | 1 | 1 | 1 | |||
15.0 Are Canadians getting value for their tax dollars? | 15.1 | Analysis of efficiency, effectiveness, and impact indicators |
Title | Author | Date |
---|---|---|
Risk-Based Approaches to Treasury Board Submissions | TBS | 2008 |
Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions | TBS | 2007 |
Role of the Program Analyst | TBS | |
Analyst Survival Guide | TBS | |
Program Analyst Tool Checklist | TBS | |
Departmental Contact List - Template | TBS | |
System Accounts for New Program Analysts | TBS | |
Transport and Transmittal of Protected and Classified Information | RCMP | 2006 |
Submissions: Suggested Review Process | TBS | |
Submissions: A Guide to Preparing Treasury Board Submissions | TBS | 2007 |
Submissions: Treasury Board Submissions Process - Chart | TBS | 2005 |
Submissions: Preferred Wording for a Proposal in 'STS' | TBS | |
Submissions: Deadline Schedule for Signed Departmental Submissions | TBS | 2008 |
Submissions: Flow Chart - Important Dates for Submissions and Précis | TBS | 2008 |
Submissions: Treasury Board and Submission Control Document Centre Requirements | TBS | |
Memorandum to Cabinet: Memorandum to Cabinet Process Reference Guide | TBS | |
Memorandum to Cabinet: Reference Material for Briefing Notes | TBS | 2006 |
Memorandum to Cabinet: What is Treasury Board Looking for in a Memorandum to Cabinet | TBS | |
Expenditure Management: Expenditure Management Cycle Chart | TBS | |
Expenditure Management: Program Activity Architecture (PAA) | TBS | 2008 |
Expenditure Management: Annual Reference Level Update (ARLU) | TBS | 2008 |
Management Accountability Framework | TBS | 2008 |
Capital Plans, Projects and Procurement - Policies and publications | TBS | 2002 |
Contracting - Policies and publications | TBS | 2003 |
Government of Canada Estimates documents and other supporting documents | TBS | 2007 |
Guide on Grants, Contributions and Other Transfer Payments | TBS | 2000 |
Project Approval | TBS | 2005 |
Project Management | TBS | 1994 |
Real Property - Policies and procedures | TBS | |
Guide on Revolving Funds | TBS | |
Policy on Transfer Payments | TBS | 2008 |
Crown Corporations Policies and Guidelines | TBS | |
Submission Checklist Tool | TBS | |
Preparing Treasury Board Submissions and Cabinet Documents: Participant's Manual | Canada School of Public Service | 2008 |
TBS Evaluation Template | TBS | 2008 |
Change Agenda Triangle | TBS | 2007 |
Budget Office Systems Renewal, Chapter 1: Business Blueprint | TBS | 2007 |
Budget Office Systems Renewal, Chapter 3: Realization Plan; Appendix A: Risk Statements | TBS | 2007 |
Budget Office Systems Renewal, Chapter 3: Realization Plan; Appendix D: Change Management | TBS | 2007 |
EMS BOSR Outcomes Management Project | Fujitsu | 2006 |
Financial Administration Act | Government of Canada | 1985 |
Treasury Board of Canada Secretariat 2007-08 Report on Plans and Priorities | TBS | 2008 |
"Putting it Together," TBS Boot Camp on The Treasury Board Submission Process | TBS | 2005 |
Draft Analysis of Grants and Contributions Delegations | TBS | 2007 |
Public Sector Reform: Delegated Authority in Australia and UK | TBS | |
2007 Comprehensive Spending Review: Guidance for Departmental Submissions | UK | 2007 |
Heavy submitters (greater than 2 per month on average in the last 10 years)
Moderate submitters (between 1 and 2 per month on average over the last 10 years)
Occasional submitters (less than 1 per month on average over the last 10 years)
Program Sectors
Internal Contacts
(Note: this is an example of the TBS Analyst Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps.
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
1. What is your role and involvement with the TB submission process?
2. Based on your understanding, what are the TB submissions to be used for?
3. Do TBS analysts have a strong understanding of the elements of a TB submission, the policies and guidance related to TB submissions, and the TB submission process? If not, what would help improve their level of understanding?
4. In the past year, have you consulted any of the following material regarding the TB submission process? Did it help improve your understanding of the elements of TB submissions, policies, and TB submission process?
5. Are TBS analysts providing accurate, consistent, useful, and timely policy advice/interpretation in regards to TB submissions:
6. In general, do TBS analysts have the appropriate training and support to enable them to provide accurate, consistent, useful, and timely advice to federal organizations in regards to the TB submission process? If not, what is required?
7.
8. In the past year, how frequently have federal organizations sought exemptions and exclusions with their TB submissions? Please explain.
9. How would you characterize your relationship with the individuals in federal organizations who you interact with in regards to the TB submission process?
10. Does your program area have a review process in place to validate the advice provided by its analysts? If yes, please briefly describe this process.
11. Do you have confidence that the advice and consultation you provide is reflected in the final TB submission documents? Please explain.
12. Are you aware of the risk criteria applied to TB submissions?
13. Please describe the role of each stakeholder (federal organization and TBS) in the risk assessment process?
14. Do the risk criteria appropriately define the risks associated with TB submissions?
15. Are the risk criteria being applied consistently across TB submissions? Please explain.
16. In your view, in the past year, how has the risk rating associated with a submission influenced your program or policy division's response to it with regards to the process, due diligence, précis, etc.?
17. Has the quality of the submission documents being put forward to Treasury Board changed over the past five years? If yes, how so?
18. Are there mechanisms in place to ensure that TB decisions are carried out?
19. What is the process for tracking the conditions applied to approved submissions?
20. How are submission approvals filed for future reference?
21. Have there been any unexpected benefits or challenges related to the process? Please explain.
22. Is the submission process efficient?
23. Are you aware of the measures proposed by internal studies such as the 2003 internal audit of the TB submission process and the 2007 EXCO Retreat? Did you find that they helped to improve the efficiency or effectiveness of the program?
Some measures undertaken as a result of the 2003 audit include:
24. Do you have anything else to add?
25. Has the amount of legal contestation of TB submissions changed over the last few years? If so, how?
(Note: this is an example of the Federal Organization Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps:
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
1. What is your role and involvement with the TB submission process?
2. Based on your understanding, what is the TB submission process to be used for? Are there other vehicles that might be more appropriate for these areas?
3. In the past year, have you consulted any of the following material regarding the submission process? Did it help improve your understanding of the elements of TB submissions, the policies and guidance related to TB submissions, and the TB submission process?
4. Do the TBS analysts you have been in contact with have a strong understanding of the elements of TB submissions, the policies and guidance related to TB submissions, and the TB submission process? If not, what would help improve their level of understanding?
5. When deciding whether a TB submission is required, do you consult a TBS analyst? Why or why not?
6. In the past year, how frequently has your organization sought exemptions and exclusions through TB submissions?
7. How would you characterize your relationship with TBS and the analysts that you interact with at TBS?
8. Once you submit a draft to TBS, to what extent is the advice and consultation provided by TBS on the draft accurate, consistent, useful, and timely?
9. Does TBS Legal Services provide advice consistent with your own organization's Legal Counsel?
10. To what extent do you incorporate the input TBS provides into the final version of the submission?
11. Are you aware of the risk criteria applied to TB submissions?
12. Please describe the role of each stakeholder (federal organizations and TBS) in the risk assessment process?
13. Do the risk criteria appropriately define the risks associated with TB submissions?
14. Has the quality of the submissions your organization puts forward to Treasury Board changed over the past five years?
15. Have there been any unexpected benefits or challenges related to the process? Please explain.
16. Is the submission process efficient?
17. Do you have anything else to add?
(Note: this is an example of the External Stakeholder Interview Guide)
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process been achieved, and whether it is cost-effective and any improvements could be made.
As part of the evaluation, GCS is conducting interviews with key stakeholders involved in the process. The goal of the interviews is examine how the process is being implemented, collect information to assess the success of the process, and identify possible process improvements.
The following questions will serve as a guide for our interview. In some cases, questions will not be relevant to your particular situation. The interviewers will focus on those questions relevant to you. Please note that the responses you provide will not be attributed to you in the evaluation report (only aggregate information will be released) or in any documentation provided to the responsible department. If you have any questions regarding the enclosed, please feel free to contact the GCS project manager, Susan Cole, at (613) 943-3401.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps:
The scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
1. What is your familiarity with the TB submission process?
2. Based on your understanding, what is the TB submission process to be used for? Are there other vehicles that might be more appropriate for these areas?
3. Does the TB submission process contribute to ensuring that departmental/government management, programs, and spending are aligned with Government of Canada priorities? Please explain.
4. Are there any particular benefits or challenges related to the process?
5. Is the submission process efficient?
6. Do you have anything else to add?
(note: this is an example of the questionnaire used for the ADM Working Session)
The Evaluation of the Treasury Board Submission Process
Government Consulting Services (GCS) has been engaged by the Treasury Board of Canada Secretariat (TBS) to conduct an evaluation of the Treasury Board (TB) submission process. The purpose of this evaluation is to examine if the process continues to be relevant, if the intended results of the process have been achieved, and whether it is cost-effective and any improvements could be made.
For the purpose of this evaluation, the TB submission process within TBS is defined by the following five steps.
Please note that the scope of the evaluation focuses only on "Part A" submissions and excludes "Part B" submissions.
The purpose of this questionnaire
As part of this evaluation, GCS is conducting a working session with assistant deputy ministers (ADMs) from TBS as well as from some federal organizations that have submitted a TB submission within the last year.
To ensure that a maximum amount of time is focused on discussions, a short list of questions has been prepared in survey format to ensure that key details have been addressed.
We thank you for taking the time to answer this short questionnaire and to return it to us at the end of the session.
The questionnaire should not take more than a few minutes to complete. Please note that the responses you provide will not be attributable to you and that the results will be reported in aggregate form only.
1. Please identify which group you are representing:
2. Please identify to what extent you agree or disagree with the following statements in terms of the TB submissions you have seen over the last few years:
Statement | Disagree | Somewhat disagree | Neither agree nor disagree | Somewhat agree | Agree | Do not know / Not applicable |
---|---|---|---|---|---|---|
Follow the Guide to Preparing Treasury Board Submissions | • | • | • | • | • | • |
Have risk and mitigation strategies identified | • | • | • | • | • | • |
Ask for the right authorities | • | • | • | • | • | • |
Include the appropriate level of justification | • | • | • | • | • | • |
Contain accurate information | • | • | • | • | • | • |
Are written in a clear manner | • | • | • | • | • | • |
Have the risk criteria consistently applied to them | • | • | • | • | • | • |
Are correctly rated for risk | • | • | • | • | • | • |
Have a risk rating aligned with Treasury Board's interpretation of risk | • | • | • | • | • | • |
3. Please identify if the following items have improved, remained unchanged, or worsened over time:
Item | Worsened | Somewhat worsened | Unchanged | Somewhat improved | Improved | Do not know/ No answer |
---|---|---|---|---|---|---|
Completeness of the submissions | • | • | • | • | • | • |
Your level of confidence in the advice/recommendations provided by TBS analysts | • | • | • | • | • | • |
Respect of timelines associated with submissions | • | • | • | • | • | • |
Overall quality of the submissions | • | • | • | • | • | • |
4. Please indicate what impact you believe the following measures could have on the efficiency of the TB submission process:
Impact | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know / No answer |
---|---|---|---|---|---|---|
Ministerial renewal of terms and Conditions - Authorizing ministers to extend expiring terms and conditions for up to three years within certain parameters, which include strategic reviews, evaluations, and audits | • | • | • | • | • | • |
Increasing departmental delegated authorities - Program sectors review delegations for departments to determine if adjustments are warranted | • | • | • | • | • | • |
Chief financial officer (CFO) Sign-off - CFO or delegate to sign off on all draft TB submissions before TBS review to signal appropriate quality and completeness | • | • | • | • | • | • |
Managing low-risk submissions differently - Flag and process TB submissions based on combined risk rating | • | • | • | • | • | • |
Altering the submission calendar - Emphasize prerogative of TBS to determine scheduling of submissions | • | • | • | • | • | • |
Streamlining supply processes - Review the Estimates processes, looking at each activity through lenses of accountability, duplication, and value added, including options to reduce the overall burden of preparatory work for Senate briefings | • | • | • | • | • | • |
Number of years | % |
---|---|
n=sample size | n=60 |
Less then one year | 33.3 |
1 to 2 years | 35.0 |
2 to 3 years | 15.0 |
3 to 4 years | 8.3 |
4 to 5 years | 3.3 |
More then 5 years | 5.0 |
Sector | % |
---|---|
n=sample size | n=59 |
Social and Cultural | 22.0 |
Economic | 18.6 |
International Affairs, Security and Justice | 23.7 |
Government Operations (program side) | 35.6 |
Classification and level | % |
---|---|
n=sample size | n=60 |
ES-02 or equivalent | 5.0 |
ES-03 or equivalent | 6.7 |
ES-04 or equivalent | 20.0 |
ES-05 or equivalent | 18.3 |
ES-06 or equivalent | 40.0 |
ES-07 or equivalent | 6.7 |
EX-03 or equivalent | 3.3 |
Number of departments | % |
---|---|
n=sample size | n=59 |
Mean | 3.9 |
1 | 22.0 |
2 | 18.6 |
3 | 16.9 |
4 | 8.5 |
5 | 15.3 |
6 | 3.4 |
8 | 3.4 |
9 | 5.1 |
10 | 6.8 |
Federal Department\Agency | n= | % |
---|---|---|
Note: Due to the ability to select multiple responses for this question, total percentage values will not equal 100. | ||
Agriculture and Agri-Food Canada | 2 | 3.3 |
Canadian Heritage | 6 | 10.0 |
Canadian International Development Agency | 5 | 8.3 |
Fisheries and Oceans Canada | 3 | 5.0 |
Foreign Affairs and International Trade Canada | 8 | 13.3 |
Health Canada | 5 | 8.3 |
Human Resources and Skills Development Canada | 7 | 11.7 |
Indian and Northern Affairs Canada | 8 | 13.3 |
Industry Canada | 6 | 10.0 |
Infrastructure Canada | 2 | 3.3 |
National Capital Commission | 1 | 1.7 |
National Defence and the Canadian Forces | 7 | 11.7 |
Natural Resources Canada | 5 | 8.3 |
Privy Council Office | 10 | 16.7 |
Public Works and Government Services Canada | 12 | 20.0 |
Royal Canadian Mounted Police | 6 | 10.0 |
Transport, Infrastructure and Communities Portfolio | 7 | 11.7 |
Treasury Board of Canada Secretariat | 16 | 26.7 |
**Other | 19 | 31.7 |
** Other Federal Department\Agency | n= | % |
---|---|---|
Note: Due to the ability to select multiple responses for this question, total percentage values will not equal 100. | ||
Atlantic Canada Opportunities Agency | 2 | 2.9 |
Assisted Human Reproduction Canada | 1 | 1.4 |
Canada Border Services Agency | 1 | 1.4 |
Canada Council for the Arts | 1 | 1.4 |
Canada Lands Company Limited | 1 | 1.4 |
Canada Mortgage and Housing Corporation | 1 | 1.4 |
Canada Public Service Agency (now Office of the Chief Human Resources Officer) | 2 | 2.9 |
Canada Revenue Agency | 1 | 1.4 |
Canada School of Public Service | 1 | 1.4 |
Canadian Artists and Producers Professional Relations Tribunal | 1 | 1.4 |
Canadian Centre for Occupational Health and Safety | 1 | 1.4 |
Canadian Food Inspection Agency | 1 | 1.4 |
Canadian Grain Commission | 1 | 1.4 |
Canadian Industrial Relations Board | 1 | 1.4 |
Canadian Institutes of Health Research | 2 | 2.9 |
Canadian International Trade Tribunal | 1 | 1.4 |
Canadian Museum for Human Rights | 1 | 1.4 |
Canadian Museum of Civilization | 2 | 2.9 |
Canadian Museum of Nature | 2 | 2.9 |
Canadian Radio-television Telecommunications Commission | 1 | 1.4 |
Canadian Science and Technology Museum | 2 | 2.9 |
Canadian Security Intelligence Service | 2 | 2.9 |
Canadian Space Agency | 1 | 1.4 |
Canadian Tourism Commission | 1 | 1.4 |
Citizenship and Immigration Canada | 1 | 1.4 |
Correctional Service Canada | 1 | 1.4 |
Department of Finance Canada | 8 | 11.4 |
Environment Canada | 1 | 1.4 |
Farm Credit Canada | 1 | 1.4 |
First Nations Statistical Institute | 1 | 1.4 |
Registry of the Specific Claims Tribunal of Canada | 2 | 2.9 |
Indian Residential Schools Resolution Canada | 2 | 2.9 |
Indian Residential Schools Truth and Reconciliation Commission | 1 | 1.4 |
Library of Parliamentr | 1 | 1.4 |
National Gallery of Canada | 1 | 1.4 |
Office of the Commissioner of Official Languages | 1 | 1.4 |
Office of the Information Commissioner | 1 | 1.4 |
Office of the Privacy Commissioner | 1 | 1.4 |
Parks Canada | 2 | 2.9 |
Public Health Agency of Canada | 2 | 2.9 |
Public Safety Canada | 2 | 2.9 |
Science and Engineering Research Canada | 1 | 1.4 |
Social Sciences and Humanities Research Council of Canada | 1 | 1.4 |
Standards Council of Canada | 1 | 1.4 |
Statistics Canada | 1 | 1.4 |
Veterans Affairs Canada | 4 | 5.7 |
Number of Submissions | % |
---|---|
n=sample size | n=60 |
Mean | 12.5 |
1 | 1.7 |
2 | 1.7 |
3 | 10.0 |
5 | 15.0 |
6 | 11.7 |
7 | 1.7 |
8 | 10.0 |
9 | 6.7 |
10 | 11.7 |
12 | 8.3 |
15 | 5.0 |
17 | 1.7 |
20 | 3.3 |
25 | 3.3 |
40 | 3.3 |
42 | 1.7 |
50 | 1.7 |
100 | 1.7 |
% completed that were: | Standard % | Complex % | Strategic Review % |
---|---|---|---|
n= sample size | n=57 | n=54 | n=38 |
Overall submission distribution | 54.2% | 38.4% | 7.5 |
0 | 1.8 | 3.7 | 36.8 |
1 | 7.9 | ||
5 | 1.9 | 5.3 | |
9 | 1.8 | ||
10 | 5.3 | 5.6 | 21.1 |
11 | 2.6 | ||
15 | 1.9 | ||
20 | 7.0 | 11.1 | 13.2 |
25 | 1.8 | 7.4 | 5.3 |
30 | 1.8 | 11.1 | |
33 | 1.8 | ||
34 | 1.9 | 2.6 | |
35 | 1.8 | 1.9 | |
40 | 3.5 | 9.3 | |
45 | 1.8 | 1.9 | |
50 | 14.0 | 14.8 | |
55 | 1.8 | 1.9 | |
60 | 10.5 | 5.6 | |
65 | 1.8 | ||
66 | 3.5 | ||
67 | 1.9 | ||
70 | 8.8 | 1.9 | 2.6 |
75 | 7.0 | 1.9 | |
79 | 1.8 | ||
80 | 10.5 | 7.4 | |
89 | 1.8 | ||
90 | 3.5 | 5.6 | 2.6 |
94 | 1.8 | ||
100 | 5.3 | 1.9 |
% | % |
---|---|
n=sample size | n=59 |
Mean | 14.5 |
0 | 49.2 |
10 | 16.9 |
20 | 11.9 |
25 | 3.4 |
30 | 8.5 |
40 | 1.7 |
50 | 3.4 |
75 | 1.7 |
100 | 3.4 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
I have a strong understanding of the TB submission process. | 5.0 | 10.0 | 36.7 | 46.7 | 1.7 | n=60 | |
The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 5.0 | 13.3 | 51.7 | 30.0 | n=60 | ||
TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 8.3 | 40.0 | 31.7 | 16.7 | 3.3 | n=60 | |
TBS Analyst Boot Camp helped to improve my understanding of the elements of TB submissions, policies, and process. | 2.5 | 17.5 | 32.5 | 27.5 | 15.0 | 5.0 | n=40 |
Canada School of Public Service (CSPS) training on TB submissions helped to improve my understanding of the elements of TB submissions, policies, and process. | 11.1 | 11.1 | 27.8 | 5.6 | 44.4 | n=18 | |
I have a strong understanding of who to consult within different TBS areas. | 3.3 | 5.0 | 10.0 | 35.0 | 46.7 | n=60 | |
I benefit from ongoing support, mentoring, and training as a program analyst. | 5.4 | 10.7 | 14.3 | 39.3 | 30.4 | n=56 | |
I have sufficient time to provide input into TB submission documents. | 3.3 | 10.0 | 58.3 | 28.3 | n=60 | ||
I am confident that the advice I provide is reflected in final TB submission documents. | 3.3 | 10.0 | 58.3 | 28.3 | n=60 | ||
The federal organizations I work with have demonstrated, over the years, an increased understanding of the elements of TB submissions, policies, and process. | 3.4 | 15.5 | 22.4 | 44.8 | 3.4 | 10.3 | n=58 |
I maintain ongoing, regular contact with my client departments regardless of the submission process. | 1.7 | 6.7 | 38.3 | 53.3 | n=60 | ||
It is easy for me to obtain input from other TBS sectors, as required, when working on TB submissions. | 5.0 | 8.3 | 23.3 | 41.7 | 21.7 | n=60 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
In general, program analysts are offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions. | 3.3 | 46.7 | 48.3 | 1.7 | n=60 | ||
In general, program analysts are providing consistent policy advice/ interpretation regarding TB submissions at the pre-submission stage (when federal organizations are trying to decide whether a TB submission is required). | 5.0 | 20.0 | 56.7 | 5.0 | 13.3 | n=60 | |
In general, program analysts are providing accurate policy advice/ interpretation regarding TB submissions at the pre-submission stage (when federal organizations are trying to decide whether a TB submission is required). | 1.7 | 15.0 | 68.3 | 6.7 | 8.3 | n=60 | |
In general, the submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 6.8 | 55.9 | 35.6 | 1.7 | n=59 | ||
In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing consistent advice regarding TB submissions during the TB submission review stage. | 1.7 | 15.3 | 16.9 | 57.6 | 5.1 | 3.4 | n=59 |
In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing accurate advice regarding TB submissions during the TB submission review stage. | 1.7 | 16.9 | 71.2 | 8.5 | 1.7 | n=59 | |
In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing useful advice regarding TB submissions during the TB submission review stage. | 3.4 | 10.2 | 72.9 | 13.6 | n=59 | ||
In general, TBS analysts (Program, Policy, and Enabling Sectors) are providing timely advice in regards to TB submissions during the TB submission review stage. | 1.7 | 10.0 | 28.3 | 40.0 | 20.0 | n=60 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
TBS has adequate mechanisms in place to track conditions placed on TB submissions. | 23.3 | 45.0 | 18.3 | 1.7 | 1.7 | 10.0 | n=60 |
TBS communicates the TB decision in a timely manner to those involved. | 3.3 | 11.7 | 51.7 | 30.0 | 3.3 | n=60 | |
TBS communicates the TB decision to all required parties (e.g. federal organization, Expenditure Management Sector, program analysts, other TBS analysts, etc.). | 1.7 | 13.3 | 11.7 | 45.0 | 13.3 | 15.0 | n=60 |
TBS's decision filing system is complete. | 10.2 | 10.2 | 13.6 | 23.7 | 0.0 | 40.7 | n=59 |
TBS's decision filing system is usable. | 8.3 | 11.7 | 13.3 | 25.0 | 0.0 | 40.0 | n=60 |
12. In the past year, have federal organizations shared drafts for consideration/review that did not require a TB submission?
Were any drafts shared? | Did the authors consult TBS prior to sharing? | Were any drafts submitted? | Did the authors consult TBS prior to submitting? | |
---|---|---|---|---|
n=sample size | n=59 | n=24 | n=58 | n=11 |
Yes | 40.7 | 62.5 | 19.0 | 81.8 |
No | 40.7 | 25.0 | 55.2 | 9.1 |
Do not know | 18.6 | 12.5 | 25.9 | 9.1 |
% | Shared | Submitted |
---|---|---|
n=sample size | n=24 | n=11 |
Mean | 17.5 | 9.1 |
0 | 16.7 | 36.4 |
10 | 58.3 | 45.5 |
20 | 12.5 | 9.1 |
30 | 4.2 | 9.1 |
90 | 4.2 | |
100 | 4.2 |
Frequency | Shared | Submitted |
---|---|---|
n=sample size | n=24 | n=11 |
Increasing | 4.2 | |
Decreasing | 8.3 | |
Remaining the same | 29.2 | 36.4 |
Do not know | 58.3 | 63.6 |
Weeks | % |
---|---|
n=sample size | n=53 |
Mean | 8.5 |
4 | 1.9 |
5 | 3.8 |
6 | 22.6 |
7 | 5.7 |
8 | 32.1 |
9 | 5.7 |
10 | 5.7 |
11 | 1.9 |
12 | 15.1 |
14 | 3.8 |
20 | 1.9 |
% | Appear to have completed internal consultations? | Were submitted with enough time for TBS analysts to review? | Follow the Guide to Preparing Treasury Board Submissions? | Have risk and mitigation strategies identified? | Ask for the right authorities? | Have the appropriate level of justification? | Contain accurate information? | Are written in a clear manner? |
---|---|---|---|---|---|---|---|---|
n=sample size | n=58 | n=59 | n=59 | n=59 | n=58 | n=59 | n=59 | n=59 |
Ind. (9.2) | Ind. (5.1) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
Mean | 57.3 | 57.5 | 56.7 | 59.5 | 60.2 | 54.7 | 54.2 | 51.4 |
0 | 3.4 | 1.7 | 3.4 | 1.7 | 3.4 | 6.8 | 1.7 | |
10 | 5.2 | 3.4 | 6.8 | 10.2 | 3.4 | 1.7 | 3.4 | 5.1 |
20 | 1.7 | 6.8 | 3.4 | 3.4 | 3.4 | 6.8 | 1.7 | 5.1 |
25 | 5.2 | 6.8 | 3.4 | 1.7 | 6.9 | 8.5 | 6.8 | 5.1 |
30 | 3.4 | 1.7 | 1.7 | 10.2 | 6.9 | 8.5 | 1.7 | 10.2 |
40 | 5.2 | 1.7 | 3.4 | 1.7 | 1.7 | 3.4 | 3.4 | 5.1 |
50 | 25.9 | 23.7 | 25.4 | 15.3 | 12.1 | 16.9 | 25.4 | 20.3 |
60 | 6.9 | 11.9 | 6.8 | 5.1 | 17.2 | 8.5 | 16.9 | 18.6 |
70 | 6.9 | 16.9 | 18.6 | 10.2 | 8.6 | 13.6 | 11.9 | 10.2 |
75 | 8.6 | 5.1 | 8.5 | 8.5 | 20.7 | 10.2 | 8.5 | 10.2 |
80 | 10.3 | 6.8 | 8.5 | 8.5 | 6.9 | 8.5 | 3.4 | 3.4 |
90 | 8.6 | 6.8 | 5.1 | 10.2 | 8.6 | 5.1 | 5.1 | 3.4 |
100 | 8.6 | 6.8 | 5.1 | 13.6 | 3.4 | 5.1 | 5.1 | 1.7 |
% | A TB decision that is consistent with the recommendations you put forward? | A TB decision that has additional conditions to those you proposed? | Challenges to the submission and/or recommendations? | Legal contestations? |
---|---|---|---|---|
n=sample size | n=58 | n=58 | n=58 | n=58 |
Mean | 82.2 | 5.6 | 8.4 | 3.5 |
0 | 5.2 | 63.8 | 55.2 | 74.1 |
10 | 1.7 | 22.4 | 13.8 | 3.4 |
20 | 3.4 | 5.2 | 5.2 | |
25 | 5.2 | 3.4 | ||
30 | 1.7 | 1.7 | ||
50 | 3.4 | 1.7 | ||
70 | 5.2 | |||
75 | 1.7 | |||
80 | 1.7 | |||
90 | 22.4 | |||
100 | 56.9 | 1.7 | ||
Do not know | 6.9 | 6.9 | 13.8 | 12.1 |
Level of awareness | % |
---|---|
n=sample size | n=58 |
Yes, I am aware, but I do not apply them when I analyze and provide advice. | 3.4 |
Yes, I am aware, and I do apply them when I analyze and provide advice. | 91.4 |
No, I am not aware. | 5.2 |
Risk Criteria being applied consistently? | % |
---|---|
n=58 | 100.0 |
Yes | 43.1 |
No | 25.9 |
Do not know | 31.0 |
Level of quality? | % |
---|---|
n=sample size | n=59 |
Significantly worse | 3.4 |
Somewhat worse | 8.5 |
No change (neither better nor worse) | 50.8 |
Somewhat better | 32.2 |
Significantly better | 5.1 |
% | Quality of first draft | Not enough time for TBS review, consultation, due diligence | Lack of program analysts' knowledge of department / need of coordination with department | Policy centre advice not appropriate, consistent, coordinated | Federal organizations' poor follow-up to TBS input or needs | Poor TBS condition tracking | Record keeping | TBS requirements unclear, unresponsive to federal organizations' needs, awkward for federal organizations | Political pressure | n=sample size |
---|---|---|---|---|---|---|---|---|---|---|
1st | 32.8 | 43.1 | 3.4 | 10.3 | 1.7 | 1.7 | 0.0 | 3.4 | 1.7 | n=57 |
2nd | 15.5 | 31.0 | 3.4 | 8.6 | 19.0 | 0.0 | 3.4 | 10.3 | 3.4 | n=55 |
3rd | 8.6 | 20.7 | 5.2 | 12.1 | 27.6 | 5.2 | 0.0 | 8.6 | 0.0 | n=51 |
% | TBS tools | Provides insight and knowledge of departmental operations | Keeps ministers informed | Strong process | TBS staff support, good consultation process | Recent changes beneficial | Enhances operations of federal organizations | n=sample size |
---|---|---|---|---|---|---|---|---|
1st Strength | 5.5 | 3.6 | 3.6 | 38.2 | 40.0 | 1.8 | 10.9 | n=55 |
2nd Strength | 9.6 | 6.8 | 6.8 | 40.9 | 29.5 | 0.0 | 6.8 | n=45 |
3rd Strength | 7.9 | 10.5 | 7.9 | 39.5 | 26.3 | 2.6 | 5.3 | n=38 |
% | Pre-submission advice and consultation? | Reviewing/consulting on submissions? | Briefing/preparing briefing documents (e.g. précis)? | Other TBS submission activities? | Other TBS non-submission activities? |
---|---|---|---|---|---|
n=sample size | n=59 | n=59 | n=59 | n=37 | n=55 |
Average time for each activity | 15.8% | 28.5% | 21.6% | 6.8% | 27.4% |
0 | 3.4 | 0.00 | 3.4 | 18.9 | |
5 | 20.3 | 3.4 | 3.4 | 29.7 | 3.6 |
10 | 32.2 | 8.5 | 13.6 | 18.9 | 16.4 |
15 | 8.5 | 13.6 | 13.6 | 5.4 | 5.5 |
20 | 15.3 | 16.9 | 27.1 | 18.9 | 7.3 |
25 | 3.4 | 8.5 | 10.2 | 2.7 | 10.9 |
30 | 6.8 | 18.6 | 15.3 | 0.0 | 18.2 |
35 | 0 | 3.4 | 5.1 | 2.7 | 3.6 |
40 | 6.8 | 11.9 | 5.1 | 0.0 | 18.2 |
45 | 1.7 | 1.7 | 0.0 | 2.7 | 3.6 |
50 | 1.7 | 5.1 | 3.4 | 7.3 | |
55 | 3.4 | 1.8 | |||
60 | 3.4 | 1.8 | |||
65 | 1.8 | ||||
70 | 1.7 |
% | Document preparation (administrative) | MAF related | Provision of advice | Strategic reviews | Oversight | Post-approval activities | Financial exercises | ARLU and Supple-mentary Estimates, Memoranda to Cabinet | Policy input |
---|---|---|---|---|---|---|---|---|---|
n=65 | 4.6 | 16.9 | 15.4 | 3.1 | 3.1 | 12.3 | 15.4 | 21.5 | 7.7 |
Frequency | % |
---|---|
n=sample size | n=60 |
Never | 1.7 |
Rarely | 10.0 |
Sometimes | 50.0 |
Often | 23.3 |
Always | 13.3 |
Do not know | 1.7 |
Level | % |
---|---|
n=sample size | n=59 |
Decreased | 8.5 |
Remained the same | 45.8 |
Increased | 45.8 |
Percentage of workload | % |
---|---|
n=sample size | n=59 |
Mean | 49.6 |
0 | 1.7 |
20 | 1.7 |
25 | 3.4 |
30 | 5.1 |
40 | 35.6 |
50 | 22.0 |
60 | 6.8 |
70 | 10.2 |
75 | 8.5 |
80 | 5.1 |
Level of efficiency | % |
---|---|
n=sample size | n=59 |
Very inefficient | 3.4 |
Somewhat inefficient | 30.5 |
Neither inefficient nor efficient | 27.1 |
Somewhat efficient | 37.3 |
Very efficient | 1.7 |
Measures taken? | % |
---|---|
N=sample size | n=58 |
Yes | 50.0 |
No | 50.0 |
Measures taken? | % |
---|---|
N=sample size | n=56 |
Yes | 51.8 |
No | 48.2 |
% | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n= |
---|---|---|---|---|---|---|---|
Ministerial renewal of terms and conditions | 1.7 | 3.4 | 13.8 | 51.7 | 12.1 | 17.2 | n=58 |
Increasing department delegated authorities | 1.7 | 5.1 | 20.3 | 50.8 | 16.9 | 5.1 | n=59 |
Chief financial officer (CFO) sign-off | 6.8 | 5.1 | 25.4 | 42.4 | 20.3 | 0.0 | n=59 |
Managing low-risk submissions differently | 0.0 | 0.0 | 6.9 | 51.7 | 37.9 | 3.4 | n=58 |
Altering the submission calendar | 1.7 | 0.0 | 29.3 | 31.0 | 22.4 | 15.5 | n=58 |
Streamlining supply processes | 0.0 | 1.7 | 13.6 | 40.7 | 18.6 | 25.4 | n=59 |
Additional measures | Reduce analyst turnover, better stability | Change timelines – internal and external | Have more tools | Consistent message to clients and internally | More regular involvement with Finance unit, better internal communications | Increased knowledge of federal organizations, skills, quality control | More program analyst support (increased tools and training, reallocate workload) | n= |
---|---|---|---|---|---|---|---|---|
% | 6.3 | 31.3 | 15.6 | 9.4 | 9.4 | 43.8 | 9.4 | n=32 |
Steps | Analyst turnover (re-education) | Acceptance of late submissions / supply cycle points | Principal analyst and executive director review stage | Getting EMS sign-off during Estimates period | Federal organization delays after first draft | Input from policy centres | Workload planning not efficient | n= |
---|---|---|---|---|---|---|---|---|
% | 2.9 | 11.4 | 11.4 | 2.9 | 28.6 | 31.4 | 25.7 | n=35 |
Additional comments | Federal organizations doing poor submissions | Better information sharing, knowledge access, and workflow management needed | Need more time for proper due diligence | More training needed for federal organizations | n= |
---|---|---|---|---|---|
% | 12.5 | 50 | 31.3 | 6.25 | n=16 |
Number of years | % |
---|---|
n=sample size | n=66 |
Less then one year | 15.2 |
1 to 2 years | 33.3 |
2 to 3 years | 19.7 |
3 to 4 years | 10.6 |
4 to 5 years | 4.5 |
More then 5 years | 16.7 |
Sector | % |
---|---|
n=sample size | n=66 |
Policy Sectors: Chief Information Officer Branch | 19.7 |
Policy Sectors: Government Operations (policy side) | 12.1 |
Policy Sectors: Labour Relations and Compensation Operations | 1.5 |
Policy Sectors: Office of the Comptroller General | 16.7 |
Policy Sectors: Pension and Benefits | 9.1 |
Policy Sectors: Expenditure Management | 21.2 |
Policy Sectors: Service | 1.5 |
Enabling Sectors: Priorities and Planning | 1.5 |
Enabling Sectors: Corporate Services | 3.0 |
Enabling Sectors: Legal Services | 7.6 |
Enabling Sectors: Regulatory Affairs | 4.5 |
Other | 1.5 |
Number of Submissions | % |
---|---|
n=sample size | n=63 |
Mean | 38.7 |
0 | 1.6 |
1 | 1.6 |
2 | 9.5 |
3 | 1.6 |
3 | 4.8 |
4 | 1.6 |
5 | 1.6 |
6 | 6.3 |
7 | 1.6 |
8 | 1.6 |
10 | 9.5 |
12 | 1.6 |
15 | 1.6 |
20 | 3.2 |
22 | 1.6 |
25 | 4.8 |
27 | 1.6 |
30 | 4.8 |
31 | 3.2 |
35 | 1.6 |
40 | 4.8 |
45 | 1.6 |
50 | 3.2 |
60 | 6.3 |
65 | 1.6 |
75 | 3.2 |
80 | 1.6 |
100 | 6.3 |
115 | 1.6 |
135 | 1.6 |
200 | 1.6 |
275 | 1.6 |
% completed that were: | Standard % | Complex % | Strategic Review % |
---|---|---|---|
n= sample size | n=60 | n=57 | n=33 |
Overall submission distribution | 61.1% | 34.3% | 4.5% |
0 | 21.2 | ||
1 | 18.2 | ||
2 | 1.8 | 3.0 | |
3 | 3.0 | ||
4 | 1.8 | 3.0 | |
5 | 18.2 | ||
6 | 3.0 | ||
8 | 3.0 | ||
10 | 1.7 | 5.3 | 3.0 |
13 | 3.0 | ||
14 | 3.0 | ||
15 | 8.8 | ||
19 | 1.7 | 1.8 | |
20 | 5.0 | 5.3 | 9.1 |
22 | 1.7 | ||
23 | 1.7 | ||
25 | 1.7 | 19.3 | 6.1 |
26 | 1.8 | ||
30 | 5.3 | ||
33 | 8.8 | ||
35 | 1.8 | ||
37 | 1.8 | ||
40 | 5.0 | 1.8 | |
45 | 1.8 | ||
50 | 15.0 | 14.0 | |
53 | 1.7 | ||
55 | 1.7 | ||
58 | 1.7 | ||
60 | 5.0 | 5.3 | |
65 | 1.7 | 1.8 | |
67 | 5.0 | ||
70 | 6.7 | ||
74 | 1.7 | ||
75 | 11.7 | 3.5 | |
77 | 3.0 | ||
79 | 1.7 | ||
80 | 8.3 | 3.5 | |
84 | 1.7 | ||
85 | 1.7 | ||
90 | 5.0 | 1.8 | |
92 | 1.7 | ||
95 | 1.7 | ||
98 | 1.7 | ||
100 | 6.7 | 3.5 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
I have a strong understanding of the elements of TB submissions, policies, and process. | 9.2 | 12.3 | 33.8 | 43.1 | 1.5 | n=65 | |
The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 6.6 | 11.5 | 57.4 | 21.3 | 3.3 | n=61 | |
TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 1.7 | 6.9 | 22.4 | 41.4 | 15.5 | 12.1 | n=58 |
TBS Analysts Boot Camp helped to improve my understanding of the elements of TB submissions, policies, and process. | 12.0 | 16.0 | 28.0 | 28.0 | 16.0 | n=25 | |
Canada School of Public Service (CSPS) training on TB submissions helped to improve my understanding of the elements of TB submissions, policies, and process. | 14.3 | 21.4 | 64.3 | n=14 | |||
I have sufficient time to provide input into TB submission documents. | 10.8 | 33.8 | 13.8 | 35.4 | 6.2 | n=65 | |
I am confident that the advice I provide is reflected in final TB submission documents. | 4.5 | 18.2 | 50.0 | 21.2 | 6.1 | n=66 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
The program analysts I work with have demonstrated, over the years, an increased understanding of the elements of TB submissions, policies, and process. | 3.2 | 17.7 | 19.4 | 33.9 | 16.1 | 9.7 | n=62 |
Program analysts have a strong understanding of the elements of TB submissions, policies, and process as a result of TBS-provided awareness education activities. | 10.3 | 22.4 | 15.5 | 8.6 | 43.1 | n=58 | |
Program analysts consult our team in a timely manner with respect to submissions. | 32.8 | 23.4 | 32.8 | 9.4 | 1.6 | n=64 | |
Program analysts provide me with enough information to adequately respond and provide advice with respect to submissions. | 3.1 | 18.5 | 13.8 | 49.2 | 15.4 | n=65 | |
TBS is offering services that enable federal organizations' draft submissions to comply with TB authorities, policies, and directions. | 1.7 | 15.3 | 37.3 | 11.9 | 33.9 | n=59 | |
The submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 1.6 | 1.6 | 9.7 | 54.8 | 24.2 | 8.1 | n=62 |
Statement | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
TBS has adequate mechanisms in place to track conditions placed on TB submissions. | 14.1 | 25.0 | 7.8 | 17.2 | 1.6 | 34.4 | n=64 |
TBS communicates the TB decision in a timely manner to those involved. | 9.8 | 6.6 | 9.8 | 27.9 | 6.6 | 39.3 | n=61 |
TBS communicates the TB decision to all required parties. | 9.8 | 26.2 | 16.4 | 14.8 | 3.3 | 29.5 | n=61 |
TBS's decision filing system is complete. | 6.5 | 11.3 | 12.9 | 8.1 | 3.2 | 58.1 | n=62 |
TBS's decision filing system is usable. | 4.8 | 9.7 | 16.1 | 11.3 | 1.6 | 56.5 | n=62 |
8. In the past year, have federal organizations shared drafts for consideration/review that did not require a TB submission?
% | Were any drafts shared? | Did the authors consult TBS prior to sharing? | Were any drafts submitted? | Did the authors consult TBS prior to submitting? |
---|---|---|---|---|
Yes | 45.3 | 44.8 | 10.8 | 42.9 |
No | 20.3 | 10.3 | 36.9 | |
Do not know | 34.4 | 44.8 | 52.3 | 57.1 |
n=sample size | n=64 | n=29 | n=65 | n=7 |
% | Shared | Submitted |
---|---|---|
n=sample size | n=29 | n=7 |
Mean | 15.3 | 8.6 |
0 | 17.2 | 28.6 |
10 | 69.0 | 57.1 |
20 | 14.3 | |
40 | 3.4 | |
50 | 3.4 | |
75 | 3.4 | |
80 | 3.4 |
% | Shared | Submitted |
---|---|---|
Increasing | 21.4 | |
Decreasing | 7.1 | |
Remaining the same | 32.1 | 14.3 |
Do not know | 39.3 | 85.7 |
n=sample size | n=28 | n=7 |
Weeks | % |
---|---|
n=sample size | n=49 |
Mean | 10.3 |
6 | 6.1 |
3 | 4.1 |
4 | 4.1 |
5 | 2.0 |
6 | 10.2 |
7 | 2.0 |
8 | 24.5 |
9 | 4.1 |
10 | 8.2 |
12 | 14.3 |
14 | 4.1 |
15 | 4.1 |
16 | 6.1 |
18 | 2.0 |
26 | 2.0 |
56 | 2.0 |
% | Appear to have completed internal consultations? | Were submitted with enough time for your team to adequately review them? | Follow the Guide to Preparing Treasury Board Submissions? | Have risk and mitigation strategies identified relevant to your area? | Ask for the right authorities relevant to your area? | Have an appropriate level of justification relevant to your area? | Contain accurate information relevant to your area? | Are written in a clear manner? |
---|---|---|---|---|---|---|---|---|
n=sample size | n=58 | n=60 | n=53 | n=50 | n=55 | n=55 | n=59 | n=61 |
Ind. (9.2) | Ind. (5.1) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
Mean | 55.1 | 50.8 | 62.7 | 52.5 | 61.9 | 55.9 | 54.3 | 60.6 |
0 | 13.8 | 8.3 | 5.7 | 12.0 | 7.3 | 10.9 | 8.5 | 3.3 |
10 | 1.7 | 5.0 | 4.0 | 3.4 | 1.6 | |||
20 | 3.4 | 1.7 | 3.8 | 6.0 | 3.6 | 3.6 | 1.7 | 1.6 |
25 | 6.9 | 15.0 | 1.9 | 4.0 | 3.6 | 5.1 | ||
30 | 5.2 | 3.3 | 5.7 | 6.0 | 5.5 | 1.8 | 5.1 | 3.3 |
40 | 5.0 | 1.9 | 2.0 | 7.3 | 10.9 | 8.5 | 9.8 | |
50 | 22.4 | 18.3 | 18.9 | 16.0 | 14.5 | 16.4 | 16.9 | 24.6 |
60 | 3.4 | 5.0 | 7.5 | 12.0 | 9.1 | 7.3 | 13.6 | 4.9 |
70 | 5.2 | 5.0 | 3.8 | 2.0 | 5.5 | 12.7 | 3.4 | 16.4 |
75 | 8.6 | 15.0 | 22.6 | 14.0 | 9.1 | 9.1 | 10.2 | 11.5 |
80 | 5.2 | 6.7 | 7.5 | 8.0 | 21.8 | 7.3 | 10.2 | 11.5 |
90 | 12.1 | 6.7 | 13.2 | 2.0 | 5.5 | 10.9 | 8.5 | 8.2 |
100 | 12.1 | 5.0 | 7.5 | 12.0 | 10.9 | 5.5 | 5.1 | 3.3 |
Aware of risk criteria | % |
---|---|
n=sample size | n=66 |
Yes, I am aware, but I do not apply them when I analyze and provide advice. | 33.3 |
Yes, I am aware, and I do apply them when I analyze and provide advice. | 41.3 |
No, I am not aware. | 25.4 |
Applied consistently? | % |
---|---|
n=sample size | n=61 |
Yes | 14.8 |
No | 21.3 |
Do not know | 63.9 |
Level of overall understanding | % |
---|---|
n=sample size | n=62 |
Significantly worse | 3.2 |
Somewhat worse | 19.4 |
No change (neither better nor worse) | 43.5 |
Somewhat better | 27.4 |
Significantly better | 6.5 |
% | Quality of submission | Timelines | Document manage-ment processes used throughout TBS | Insufficient feedback from TB decisions, poor TBS condition tracking | Separation of the two TBS roles, as central agency and department, and its affect on roles and responsibi-lities | Limited program analyst under-standing of policy role, program analyst workload and turnover | Inconsis-tency of advice, lack of coordinated approach | Cumber-some process | Disorgani-zation of department, don't understand tools | Problems with financial information | n=sample size |
---|---|---|---|---|---|---|---|---|---|---|---|
1st | 29.1 | 40.0 | 3.6 | 0.0 | 0.0 | 10.9 | 5.5 | 5.5 | 3.6 | 1.8 | n=55 |
2nd | 21.6 | 19.6 | 13.7 | 0.0 | 2.0 | 7.8 | 3.9 | 13.7 | 11.8 | 5.9 | n=51 |
3rd | 7.0 | 11.6 | 20.9 | 11.6 | 2.3 | 9.3 | 2.3 | 16.3 | 11.6 | 7.0 | n=43 |
% | TBS tools | Strong process, i.e. is comprehensive, has clear results, based on risk assessment, focuses priorities | TBS staff support, good internal consultation process | Recent changes beneficial | Good communications, promotes external relationships | STS repository, submission tracking | n=sample size |
---|---|---|---|---|---|---|---|
1st Strength | 6.5 | 50.0 | 32.6 | 0.0 | 6.5 | 4.3 | n=46 |
2nd Strength | 11.8 | 41.2 | 20.6 | 5.9 | 17.6 | 2.9 | n=34 |
3rd Strength | 13.0 | 43.5 | 21.7 | 17.4 | 0.0 | 4.3 | n=23 |
% | Pre-submission advice and consultation? | Reviewing/consulting on submissions? | Briefing/ preparing briefing documents (e.g. précis)? | Other TBS submission activities? | Other TBS non-submission activities? |
---|---|---|---|---|---|
Average time for each activity | 15.5% | 29.2% | 6.6% | 9.0% | 39.7% |
% | Development of policies, management activities | Part B submissions | MAF related | Provision of advice | Strategic reviews | IT oversight | Post-approval activities | Financial exercises | ARLU and Supplementary Estimates |
---|---|---|---|---|---|---|---|---|---|
n=38 | 18.4 | 5.3 | 15.8 | 21.1 | 5.3 | 7.9 | 7.9 | 10.5 | 7.9 |
Workload provided time to provide advice | % |
---|---|
n=sample size | n=66 |
Never | 1.5 |
Rarely | 15.2 |
Sometimes | 34.8 |
Often | 36.4 |
Always | 6.1 |
Do not know | 6.1 |
Level of workload | % |
---|---|
n=sample size | n=64 |
Decreased | 7.8 |
Remained the same | 37.5 |
Increased | 48.4 |
Do not know | 6.3 |
Level of efficiency | % |
---|---|
n=sample size | n=52 |
Very inefficient | 7.9 |
Somewhat inefficient | 34.9 |
Neither inefficient nor efficient | 19.0 |
Somewhat efficient | 34.9 |
Very efficientt | 3.2 |
Measures undertaken? | % |
---|---|
n=sample size | n=63 |
Yes | 38.5 |
No | 61.5 |
% | Delegated authorities | New tools | Focus on high-risk, high-value submissions | None taken / Not aware of | More staff | Removed the need to sign off on submission after the Treasury Board date | Change to EMIS |
---|---|---|---|---|---|---|---|
n=24 | |||||||
Awareness of particular measures | 5 | 5 | 4 | 5 | 2 | 2 | 1 |
Believe measure is better | 0 | 1 | 0 | 0 | 0 | 0 | 0 |
Believe measure is no better | 0 | 1 | 0 | 0 | 0 | 0 | 1 |
% | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n=sample size |
---|---|---|---|---|---|---|---|
Ministerial renewal of terms and conditions | 1.5 | 7.6 | 36.4 | 13.6 | 40.9 | n=66 | |
Increasing department delegated authorities | 1.5 | 4.5 | 4.5 | 37.9 | 18.2 | 33.3 | n=66 |
Chief financial officer (CFO) sign-off | 4.5 | 9.1 | 28.8 | 28.8 | 28.8 | n=66 | |
Managing low-risk submissions differently | 1.5 | 3.0 | 6.1 | 39.4 | 33.3 | 16.7 | n=66 |
Altering the submission calendar | 1.6 | 6.3 | 27.0 | 27.0 | 38.1 | n=63 | |
Streamlining supply processes | 1.5 | 1.5 | 1.5 | 30.3 | 12.1 | 53.0 | n=66 |
% | Make the process more objective and less subjective, better aligned and streamlined | Better training on internal process to have more knowledgeable analysts | Better federal organization and TBS engagement, earlier federal organization consultation and support | Better time frames | Better internal tracking and document system | Aligning Part A and Part B submissions | Regular TBS working groups to assess ongoing enhancements and changes |
---|---|---|---|---|---|---|---|
n=29 | 20.7 | 27.6 | 44.8 | 6.9 | 27.6 | 3.4 | 6.9 |
% | Delays with federal organizations, federal organizations wait for last sitting, bad drafts from federal organizations | Lack of internal knowledge of internal processes and internal communications | Sign-offs and time pressures owing to political pressures and waves of submissions | None |
---|---|---|---|---|
n=22 | 31.8 | 22.7 | 22.7 | 9.1 |
% | Need more time to do a better job (CIOB) | Resist unreal departmental time expectations, educate federal organizations on this | More conditions and monitoring thereof | Greater internal information sharing, more precise information available, and reduction in conflicting information | More intensive program analyst training | Introduce more performance measurement, audit, and evaluation evidence at the Memorandum to Cabinet stage | Increase review time for high-risk submissions | Investments in government-wide solutions needed rather than costly one-off solutions by individual departments |
---|---|---|---|---|---|---|---|---|
n=18 | 11.1 | 11.1 | 5.6 | 33.3 | 22.2 | 5.6 | 5.6 | 5.6 |
Number of years | % |
---|---|
n=sample size | n=94 |
Less then one year | 18.1 |
1 to 2 years | 28.7 |
2 to 3 years | 25.5 |
3 to 4 years | 10.6 |
4 to 5 years | 3.2 |
More then 5 years | 13.8 |
Federal organization | % |
---|---|
n=sample size | n=92 |
Canadian Heritage | 4.3 |
Foreign Affairs and International Trade Canada | 7.6 |
Fisheries and Oceans Canada | 1.1 |
National Defence and the Canadian Forces | 4.3 |
Health Canada | 1.1 |
Human Resources and Skills Development Canada | 17.4 |
Indian and Northern Affairs Canada | 1.1 |
Industry Canada | 1.1 |
Infrastructure Canada | 1.1 |
Natural Resources Canada | 1.1 |
Privy Council Office | 1.1 |
Public Works and Government Services Canada | 8.7 |
Royal Canadian Mounted Police | 1.1 |
Treasury Board of Canada Secretariat | 7.6 |
Transport, Infrastructure and Communities Portfolio | 6.5 |
Other | 34.8 |
Other organizations | % |
---|---|
n=sample size | n=30 |
Atlantic Canada Opportunities Agency | 3.3 |
Atomic Energy of Canada Limited | 3.3 |
Canada Border Services Agency | 3.3 |
Canada Post Corporation | 3.3 |
Canada Public Service Agency (now Office of the Chief Human Resources Officer) | 13.3 |
Canadian Environmental Assessment Agency | 3.3 |
Canadian Food Inspection Agency | 6.7 |
Canadian Museum of Nature | 3.3 |
Canadian Nuclear Safety Commission | 3.3 |
Canadian Museum of Civilization | 3.3 |
Correctional Service Canada | 3.3 |
Courts Administration Service | 3.3 |
Department of Finance Canada | 3.3 |
Environment Canada | 3.3 |
Export Development Canada | 3.3 |
National Film Board of Canada | 3.3 |
Office of the Commissioner for Federal Judicial Affairs Canada | 3.3 |
Office of the Commissioner of Lobbying of Canada | 3.3 |
Public Safety Canada | 0.0 |
Public Service Commission of Canada | 10.0 |
Public Service Labour Relations Board | 3.3 |
Social Sciences and Humanities Research Council of Canada | 3.3 |
Status of Women Canada | 3.3 |
Veterans Affairs Canada | 3.3 |
Number of analysts | % |
---|---|
n=89 | |
Mean | 3.9 |
1 | 29.2 |
2 | 18.0 |
3 | 21.3 |
4 | 11.2 |
5 | 5.6 |
6 | 2.2 |
7 | 4.5 |
8 | 1.1 |
9 | 1.1 |
10 | 4.5 |
11 | 1.1 |
Who to consult? | % |
---|---|
n=94 | |
Yes | 95.7 |
No | 4.3 |
Number of Submissions | % |
---|---|
n=91 | |
Mean | 11.1 |
0 | 3.3 |
1 | 17.6 |
2 | 16.5 |
3 | 17.6 |
4 | 6.6 |
5 | 7.7 |
8 | 2.2 |
9 | 1.1 |
10 | 2.2 |
12 | 1.1 |
16 | 1.1 |
17 | 1.1 |
20 | 3.3 |
25 | 2.2 |
27 | 1.1 |
30 | 3.3 |
35 | 2.2 |
40 | 2.2 |
50 | 1.1 |
51 | 1.1 |
75 | 1.1 |
100 | 2.2 |
% completed that were: | Standard % | Complex % | Strategic Review % |
---|---|---|---|
n=sample size | n=80 | n=71 | n=41 |
Overall submission distribution | 56.6% | 39.3% | 4.1% |
0 | 2.5 | 5.6 | 53.7 |
2 | 2.4 | ||
3 | 2.4 | ||
6 | 2.4 | ||
10 | 1.4 | 12.2 | |
17 | 1.4 | 2.4 | |
20 | 5.6 | 4.9 | |
23 | 2.4 | ||
25 | 5.0 | 8.5 | 9.8 |
28 | 1.4 | ||
30 | 6.3 | 4.2 | 2.4 |
33 | 1.3 | 5.6 | |
34 | 4.2 | ||
37 | 1.3 | ||
40 | 8.8 | 5.6 | 2.4 |
44 | 1.3 | ||
45 | 1.3 | ||
47 | 1.3 | 1.4 | |
50 | 20.0 | 23.9 | 2.4 |
60 | 3.8 | 5.6 | |
66 | 5.0 | ||
67 | 3.8 | 1.4 | |
70 | 2.5 | 7.0 | |
75 | 5.0 | 1.4 | |
80 | 5.0 | ||
90 | 26.3 | 15.5 |
Number of submissions seeking exemptions and exclusions | % |
---|---|
n=sample size | n=87 |
Mean | 10.5 |
0 | 57.5 |
10 | 19.5 |
20 | 5.7 |
25 | 4.6 |
30 | 4.6 |
50 | 5.7 |
70 | 1.1 |
100 | 1.1 |
Attended training and learning opportunities | % |
---|---|
n=sample size | n=94 |
Yes | 34.0 |
No | 66.0 |
% | Strongly disagree | Disagree | Neutral | Agree | Strongly agree | Do not know | n= |
---|---|---|---|---|---|---|---|
I have a strong understanding of the elements of TB submissions, policies, and process. | 5.3 | 10.6 | 39.4 | 43.6 | 1.1 | n=94 | |
The Guide to Preparing Treasury Board Submissions helps to improve my understanding of the elements of TB submissions, policies, and process. | 1.1 | 7.4 | 11.7 | 50.0 | 23.4 | 6.4 | n=94 |
TBS-developed tools help to improve my understanding of the elements of TB submissions, policies, and process. | 13.0 | 20.7 | 40.2 | 9.8 | 16.3 | n=94 | |
TBS outreach helped to improve my understanding of the elements of TB submissions, policies, and process. | 2.2 | 22.6 | 22.6 | 23.7 | 10.8 | 18.3 | n=94 |
In general, TBS analysts are offering us services that enable our draft submissions to comply with TB authorities, policies, and directions. | 4.3 | 10.6 | 50.0 | 28.7 | 6.4 | n=94 | |
In general, TBS analysts are providing consistent policy advice/interpretation regarding TB submissions at the pre-submission stage. | 1.1 | 22.6 | 18.3 | 34.4 | 15.1 | 8.6 | n=94 |
In general, the submission review process within TBS ensures that TB submissions comply with government authorities and policies. | 1.1 | 2.2 | 9.9 | 47.3 | 31.9 | 7.7 | n=94 |
In general, TBS analysts are providing consistent advice regarding TB submissions during the TB submission review stage. | 6.4 | 21.3 | 11.7 | 41.5 | 13.8 | 5.3 | n=94 |
In general, TBS analysts are providing accurate advice regarding TB submissions during the TB submission review stage. | 1.1 | 8.5 | 21.3 | 43.6 | 18.1 | 7.4 | n=94 |
In general, TBS analysts are providing useful advice regarding TB submissions during the TB submission review stage. | 2.1 | 10.6 | 19.1 | 45.7 | 17.0 | 5.3 | n=94 |
In general, TBS analysts are providing timely advice regarding TB submissions during the TB submission review stage. | 5.3 | 17.0 | 19.1 | 35.1 | 19.1 | 4.3 | n=93 |
TBS communicates the TB decision with respect to our submission(s) in a timely manner. | 2.2 | 6.5 | 17.2 | 30.1 | 35.5 | 8.6 | n=94 |
Our federal organization has adequate mechanisms in place to track conditions placed on our TB submissions. | 4.3 | 14.9 | 12.8 | 30.9 | 24.5 | 12.8 | n=94 |
Weeks | Valid Percent |
---|---|
n=sample size | n=85 |
Mean | 8.9 |
2 | 2.4 |
3 | 3.5 |
4 | 7.1 |
5 | 4.7 |
6 | 18.8 |
7 | 3.5 |
8 | 18.8 |
9 | 8.2 |
10 | 8.2 |
11 | 2.4 |
12 | 9.4 |
13 | 1.2 |
14 | 3.5 |
16 | 3.5 |
22 | 1.2 |
26 | 3.5 |
% | Were complete with respect to your required internal consultations prior to submitting to TBS? | Were, in your opinion, submitted with enough time for TBS to adequately review them? | Followed the Guide to Preparing Treasury Board Submissions? | Had risk and mitigation strategies identified? | Asked for the right authorities? | Had an appropriate level of justification? | Contained accurate information (e.g. accurate financial tables)? | Were written in a clear manner? |
---|---|---|---|---|---|---|---|---|
n=sample size | n=85 | n=87 | n=83 | n=85 | n=85 | n=85 | n=85 | n=84 |
Ind. (9.2) | Ind. (5.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | Ind. (9.2) | |
Mean | 83.1 | 83.0 | 83.7 | 76.4 | 90.3 | 89.8 | 90.8 | 88.3 |
0 | 6.0 | 7.1 | 1.2 | |||||
10 | 2.3 | |||||||
20 | 1.1 | 2.4 | 1.2 | 1.2 | ||||
25 | 2.4 | 1.1 | 2.4 | 1.2 | ||||
30 | 4.7 | 1.1 | 1.2 | 1.2 | ||||
40 | 1.2 | 1.2 | ||||||
50 | 9.4 | 5.7 | 2.4 | 15.3 | 1.2 | 2.4 | 3.5 | 6.0 |
60 | 3.5 | 2.3 | 2.4 | 7.1 | 2.4 | 2.4 | 1.2 | 1.2 |
70 | 3.5 | 5.7 | 3.6 | 4.7 | 2.4 | 4.7 | 2.4 | 1.2 |
75 | 4.7 | 8.0 | 1.2 | 5.9 | 5.9 | 5.9 | 3.5 | 8.3 |
80 | 12.9 | 18.4 | 15.7 | 4.7 | 4.7 | 9.4 | 11.8 | 10.7 |
90 | 9.4 | 11.5 | 16.9 | 12.9 | 14.1 | 16.5 | 21.2 | 21.4 |
100 | 49.4 | 42.5 | 49.4 | 41.2 | 64.7 | 56.5 | 55.3 | 48.8 |
Level of awareness | % |
---|---|
n=sample size | n=94 |
Yes, we are aware, but we do not apply them when we prepare our TB submissions. | 7.8 |
Yes, we are aware, and we do apply them when we prepare our TB submissions. | 50.0 |
No, we are not aware. | 42.2 |
Own risk criteria? | % |
---|---|
n=sample size | n=94 |
Yes, but we do not apply them when we prepare our TB submissions. | 3.3 |
Yes, and we do apply them when we prepare our TB submissions. | 52.7 |
No | 44.0 |
Risk criteria being applied consistently? | % |
---|---|
n=sample size | n=94 |
Yes | 23.1 |
No | 19.8 |
Do not know | 57.1 |
% | Inconsistent or inadequate advice from TBS analysts, analyst turnover | Complexity of information asked, too much information in submissions | Guide and tools insufficient | Time issues | Process issues (including political pressures) and changing requirements | Internal departmental issues | Receiving TBS communications, TBS capacity | n= |
---|---|---|---|---|---|---|---|---|
1st Challenge | 20.5 | 9.6 | 2.4 | 36.1 | 21.7 | 7.2 | 2.4 | n=83 |
2nd Challenge | 26.3 | 13.8 | 1.3 | 18.8 | 21.3 | 13.8 | 5.0 | n=80 |
3rd Challenge | 19.2 | 20.5 | 5.5 | 21.9 | 17.8 | 6.8 | 8.2 | n=74 |
% | Helpful tools (generally) | Support provided by helpful analysts | Ensures strong business case, diligence, accountability, challenge, and standards | Timelines understood | Provides future rigour to departments and Government of Canada | TBS willing to listen to suggestions, cooperative and collaborative process with feedback functions, TBS capacity | Promotes internal departmental strengths | n= |
---|---|---|---|---|---|---|---|---|
1st Strength | 18.3 | 29.6 | 36.6 | 0.0 | 4.2 | 8.5 | 2.8 | n=71 |
2nd Strength | 6.7 | 31.7 | 28.3 | 5.0 | 8.3 | 15.0 | 5.0 | n=60 |
3rd Strength | 2.7 | 16.2 | 29.7 | 8. | 13.5 | 24.3 | 5.4 | n=37 |
Level of workload | % |
---|---|
n=sample size | n=89 |
Decreased | 4.5 |
Remained the same | 28.1 |
Increased | 67.4 |
Level of workload | % |
---|---|
n=sample size | n=91 |
Very inefficient | 8.8 |
Somewhat inefficient | 33.0 |
Neither inefficient nor efficient | 17.6 |
Somewhat efficient | 38.5 |
Very efficient | 2.2 |
Measures taken to improve? | % |
---|---|
n=sample size | n=81 |
Yes | 45.7 |
No | 54.3 |
% | Guide | Outreach | Early role of analyst and role of analyst as trusted advisor | Reduced submissions | New templates, protocols | Clear timelines | Not aware of new measures | Involve Finance unit for financial tables | Better communications |
---|---|---|---|---|---|---|---|---|---|
n=45 | |||||||||
Awareness of particular measures | 26.7 | 8.9 | 13.3 | 2.2 | 17.8 | 8.9 | 4.4 | 8.9 | 8.9 |
Believe measure is better | 8.9 | 4.4 | 4.4 | 0.0 | 4.4 | 0.0 | 0.0 | 0.0 | 0.0 |
Believe measure is no better | 2.2 | 0.0 | 2.2 | 0.0 | 2.2 | 2.2 | 0.0 | 4.4 | 0.0 |
% | Significantly less efficient | Somewhat less efficient | No change | Somewhat more efficient | Significantly more efficient | Do not know | n=sample size |
---|---|---|---|---|---|---|---|
Ministerial renewal of terms and conditions | 1.1 | 2.2 | 9.0 | 30.3 | 38.2 | 19.1 | n=89 |
Increasing department delegated authorities | 1.1 | 3.3 | 11.1 | 22.2 | 43.3 | 18.9 | n=90 |
Chief financial officer (CFO) sign-off | 13.3 | 24.4 | 23.3 | 13.3 | 22.2 | 3.3 | n=90 |
Managing low-risk submissions differently | 1.1 | 2.2 | 6.6 | 29.7 | 47.3 | 13.2 | n=91 |
Altering the submission calendar | 6.7 | 15.6 | 20.0 | 18.9 | 11.1 | 27.8 | n=90 |
Streamlining supply processes | 2.2 | 12.2 | 32.2 | 21.1 | 32.2 | n=90 |
Better tools, specific tailored templates for recurring submissions, updated Guide to Preparing Treasury Board Submissions | More regular information meetings with TBS staff, earlier discussions, proactive not reactive | More training that includes TBS context and needs | Harder deadlines between Crown corporations and departments, Crown corporations work directly with TBS | TBS analysts with departmental knowledge, analyst stability | Limited review of draft and comprehen-sive, timely, and consistent feedback | An ombudsman or arbitrator to review/ bypass petty changes or difficult individuals | Clear explanation, decision, and rationale for next time | Delegation within TBS for submissions with pre-approved spending | Better calendar | Streamline process, eliminate departments' role for Supplemen-tary Estimates and ARLU | n=sample size | |
---|---|---|---|---|---|---|---|---|---|---|---|---|
% | 6.8 | 13.6 | 4.5 | 6.8 | 13.6 | 18.2 | 4.5 | 4.5 | 4.5 | 4.5 | 18.2 | 44 |
Good analysts have made the process flow well | Challenges owing to analysts' wide discretion, inconsistent comments, and instability, to inconsistent standards, and to sometimes unclear role | Political considerations undermines the process | Needs to be streamlined for some submissions, to become more efficient | Needs to be a cooperative and consultative process | Taking the time to produce a good quality first draft is essential to the efficiency of the process | n=sample size | |
---|---|---|---|---|---|---|---|
% | 16.7 | 26.7 | 10.0 | 13.3 | 30.0 | 3.3 | n=30 |
Following are more specific observations on each of the tools:
In an effort to streamline the Treasury Board submission process, TBS has considered and introduced a number of new measures to eliminate the need for a submission. For instance, during the TBS Executive Committee (EXCO) Retreat held in the fall of 2007, it was noted that delegated authorities have not kept pace with inflation or departmental capacity, resulting in many low-risk, transactional Treasury Board submissions.2 While increasing delegated authorities to be reflective of inflation has not removed the need for many submissions, for a few small agencies that are frequent submitters of many low-risk submissions, the need for more than 30 submissions can be eliminated annually.3 This will grant more time across TBS for high-risk submissions and strategic initiatives.4
Renewal of program terms and conditions (T&Cs) comprises a significant portion of Treasury Board business (700+ submissions from June 2000 to September 2007). It was proposed at the EXCO Retreat that ministers be authorized to extend expiring T&Cs for up to three years. This will allow Treasury Board more time to discuss high-risk or strategic business. Ministers will take responsibility for program T&Cs, giving their respective departments time to align T&Cs with new policy requirements and with the findings of strategic reviews.5
To qualify for increased delegated authorities, departments will have to demonstrate sound management practices. Of the 39 organizations reviewed, only 8 were recommended as potential candidates for increased delegated authorities. The reasons identified for not recommending organizations included the following: absent or poor track record, capacity issues in managing grants and contributions, low volume of submissions to have any workload impact, or existing thresholds or case-by-case review were deemed appropriate.6 To have increased delegated authorities, a federal organization must demonstrate that the first and second purposes of the submission process (oversight and management) can be completed in-house. In other words, the organization can manage itself successfully and oversight is not required.
It was also noted during the EXCO Retreat that TBS analysts spend a large amount of time reviewing poor quality draft submissions from departments. It is proposed that the chief financial officer (CFO) or delegate sign off on all draft Treasury Board submissions before TBS review to signal appropriate quality and completeness.7 This would assist analysts in their quality review and facilitation roles and would allow them to appropriately devote their time to analysis and not quality control. Furthermore, conditions would solely be used for high-risk submissions.
Name | Definition |
---|---|
Treasury Board | Is a statutory committee of Cabinet established under the Financial Administration Act. It consists of six ministers, including the President of the Treasury Board (Chairperson) and the Minister of Finance, who meet regularly to consider submissions sponsored by federal organizations. |
Cabinet | The Cabinet decides the government's priorities and policies, determines the legislation that will be presented to Parliament, and approves departmental spending and allocates funds. The Cabinet consists of all ministers who are appointed on the recommendation of the Prime Minister. |
Privy Council Office | A formal advisory body to the Crown appointed by the Governor General on the advice of the Prime Minister. The Clerk of the Privy Council is the most senior non-political official in the Government of Canada and provides professional, non-partisan support. |
Précis | Document prepared by the Secretariat and addressed to Treasury Board. It contains Secretariat recommendations on Treasury Board submissions. |
Expenditure Management Information System | The Expenditure Management Information System (EMIS) is an integrated and secure budget office system that supports the Treasury Board of Canada Secretariat (TBS) in fulfilling its expenditure management role. |
Change Agenda | Defined in Context |
TBS Program Sectors | These sectors, which support Treasury Board in its role as the government's management board and budget office, include the Economic Sector, the International Affairs, Security and Justice Sector, the Social and Cultural Sector, and the Government Operations Sector. They are the "single window" for departments to TBS on a wide range of issues, from Treasury Board submissions to Management Accountability Framework (MAF) assessments. |
Treasury Board Submission Centre | As part of the Strategic Communications and Ministerial Affairs Branch, the Treasury Board Submission Centre (the Centre) provides the Secretariat, the Treasury Board, and other federal government institutions with support services and expert advice on the processes and procedures for submissions (which includes maintaining and enforcing deadlines). |
TBS Boot Camps | Two-day orientation sessions that provide participants with a general overview of TBS roles, initiatives, and business. There are currently boot camps on 12 different topics, which range from "The Business of Supply and the Estimates Process" to "Crown Corporations." |
Management, Resources, and Results Structures | The Policy on Management, Resources, and Results Structures supports the development of a common, government-wide approach to planning and managing the relationship between resource expenditures and results while serving as a consistent and enduring foundation for collecting, managing, and reporting financial and non-financial information to Parliament. |
Strategic Reviews | Strategic reviews are departmental assessments of all direct program spending to ensure programs are managed effectively and efficiently. These reviews support a more rigorous results-based approach to managing taxpayer dollars responsibly and delivering effective and efficient programs that can better meet the priorities of Canadians. |
1. Source: A Guide to Preparing Treasury Board Submissions
2. Refers to all Treasury Board submissions except orders in council requiring Treasury Board recommendation.
3. The study did not assess Treasury Board submissions and associated précis documents for quality. Findings are based on qualitative data gathered from a survey and stakeholder interviews.
4. These issues are derived from the definitions of "relevance," "effectiveness," and "economy" found in Appendix A of the Treasury Board Policy on Evaluation.
5. A Guide to Preparing Treasury Board Submissions.
6. A limited review of submission process models used in other jurisdictions (see Methodology section) suggests that the role played by the Secretariat's assistant secretaries, whereby they present a federal organization's submission to Treasury Board, may be unique.
7. The Secretariat's program sectors are responsible for reviewing and providing advice on the management of federal organizations' financial resources.
8. A Guide to Preparing Treasury Board Submissions.
9. The term "Secretariat analysts" refers to both program analysts and COE analysts.
10. Other than Treasury Board, generally involves a policy committee.
11. It should be noted that when the Secretariat undertakes a Treasury Board submission on its own behalf, it is considered a federal organization like any other.
12. There is no formal definition of "complexity." The evaluation therefore relied on respondents' own criteria of "complex" and "standard." Note that strategic reviews (which can be "standard" or "complex") are not included in the table.
13. The Policy on the Management, Resources and Results Structures supports the development of a common, government-wide approach to planning and managing the relationship between resource expenditures and results while serving as a consistent and enduring foundation for collecting, managing, and reporting financial and non-financial information to Parliament. (http://www.tbs-sct.gc.ca/rma/mrrs-sgrr_e.asp)
14. A Guide to Preparing Treasury Board Submissions, p. 5
15. The Treasury Board submission and précis are then presented to Treasury Board ministers for consideration and approval.
16. While this question was part of the evaluation matrix, it was not covered in this report because (as discussed) it was determined that this was not an outcome of the Treasury Board submission process.
17. This question is not addressed in the findings of the report because the evaluation did not find any unexpected outcomes.
18. Note that when the report refers to "Secretariat analysts," it includes both program analysts and COE analysts if the context in which it is used is the Secretariat but only program analysts if the context is federal organizations (as program analysts are the primary contact for federal organizations). When this distinction is relevant, it is noted in the report.
19. The survey questionnaire was available on the Internet for a period of three weeks. Reminder emails were sent to respondents twice during the three-week period.
20. Responses and comments indicated that these respondents filled out the wrong survey; consequently, their responses were removed to ensure that the data was not contaminated.
21. Models in use in provinces were not reviewed.
22. Co-ordination at the Centre of Government: The Functions and Organization of the Government Office Comparative Analysis of OECD Countries, CEECs and Western Balkan Countries. Michael Ben-Gera. 2004.
23. Ibid. The ten countries are Australia, Austria, France, Germany, Iceland, Ireland, Netherlands, Norway, Spain, and Switzerland.
24. "The audit found that … a more consistent and inclusive approach to overtime costs and overtime forecasts, improvement to management and authorization of leave and overtime, … and development of formal and approved guidelines in the administration of leave and overtime would be beneficial."
25. The Program Activity Architecture of the Treasury Board of Canada Secretariat
26. A Guide to Preparing Treasury Board Submissions, p. 1
27. Ibid., p. 5
28. Feedback reviewed from the following program sector boot camps: February 2009, January 2008, August 2007, September 2004, March 2001, and September 2000.
29. "Appropriateness" refers to whether a submission put forward by a federal organization is necessary (i.e. required).
30. Interviewees also commented on the drawbacks of MCs not being required to undergo an early stage approval process from the Secretariat. One interviewee noted that the absence of Secretariat involvement can lead to problems when MCs include elements that may be contrary to Treasury Board policies.
31. Some federal organizations, however, indicated that they might choose to proceed with a submission even when the program analyst believes that it is not necessary.
32. Numbers may not add up to totals due to rounding.
33. This same question was not asked of federal organizations; therefore, the opinion of whether submissions were necessary is attributable only to Secretariat analysts.
34. From the time the program analyst receives the federal organization's first draft of the Treasury Board submission.
35. It is important to note that the time requirement may be significantly less in some cases, e.g. when the submission is a "top-up" to an existing program or when the submission is deemed to be of low risk and low complexity.
36. Some may argue, however, that flexibility in the Treasury Board submission process is an advantage in that it allows certain submissions, such as those related to Canada's Economic Action Plan, to be given priority handling when special circumstances warrant it.
37. According to guidelines in A Guide to Preparing Treasury Board Submissions
38. RDIMS is a Canadian federal government initiative to manage the full life cycle of any type of electronic document (email, correspondence, reports, etc.).
39. Budget Office Systems Renewal (BOSR) Project, Treasury Board of Canada Secretariat, 2007
40. http://www.tbs-sct.gc.ca/maf-crg/indicators-indicateurs/2008/policies-politiques/policies-politiques-eng.asp
41. Furthermore, scores between MAF Rounds IV and V are not fully comparable. The average MAF scores analyzed for the evaluation are for larger federal organizations.
42. Federal organizations' internal consultations
43. It should be noted that this percentage may not be representative of the level of effort of all COE analysts since only some COE analysts' duties include the review of Treasury Board submissions. Those COE analysts with significant duties relating to Treasury Board submissions may have been more likely to respond to the survey.
44. Numbers may not add to totals due to rounding.
45. Numbers may not add to totals due to rounding.
46. Data on the timing of Treasury Board submissions over the last 10 years show definite peaks in June, October, December, and March.
47. Factors contributing to the quality of Treasury Board submissions (according to survey): submissions appear not to have completed internal consultations (program analysts: 42.7%, COE analysts: 44.9%); submissions not allowing enough time for Secretariat analysts' review (program analysts: 42.5, COE analysts: 49.2%); submissions did not follow the Guide (program analysts: 43.3%, COE analysts: 37.3%); submissions lacking risk and mitigation strategies (program analysts: 40.5%, COE analysts: 47.5%); submissions requesting wrong authorities (program analysts: 39.8%, COE analysts: 38.1%); submissions without appropriate level of justification (program analysts: 45.3%, COE analysts: 44.1%); submissions containing inaccurate information (program analysts: 45.8%, COE analysts: 45.7%); submissions written in an unclear manner (program analysts: 48.6%, COE analysts: 39.4%).
48. While the evaluation team recognizes that turnover rates in federal organizations may also contribute to the quality of Treasury Board submissions and the efficiency of the submission process, this matter is outside the scope of the evaluation and therefore not addressed in the recommendation.
1. The data from this interview was not used, as Corporate Services was not identified as a submitting organization to be interviewed.
2. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.
3. Treasury Board of Canada Secretariat, Increased Delegated Authorities for Contributions, 2007.
4. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.
5. Ibid.
6. Treasury Board of Canada Secretariat, Increased Delegated Authorities for Contributions, 2007.
7. Treasury Board of Canada Secretariat, Roadmap for Enabling Change: New Risk-Based Approach to Departmental Business and Treasury Board Submissions, 2007.