Treasury Board of Canada Secretariat

review of Policy Monitoring 

October 28, 2004  

TBS Internal Audit and Evaluation Division
Ernst & Young, LLP

TABLE OF CONTENTS
1.0	EXECUTIVE SUMMARY
2.0	INTRODUCTION
2.1	Background
2.2	Review Purpose
2.3	Scope of Review
2.4 	Review Approach and Methodology
3.0 	REVIEW RESULTS
3.1	Summary of Leading and Effective Practices
3.2	Summary of Findings Related to Selected TB Policies
4.0	AREAS FOR IMPROVEMENT
4.1	Analysis of Effectiveness
4.2	Areas for Improved Policy Monitoring
MANAGEMENT RESPONSE
5.0	CONCLUSION
ANNEX A 	 List of Policies Included in Review
ANNEX B 	 List of Departments Interviewed

1.0	EXECUTIVE SUMMARY
The successful monitoring of Treasury Board (TB) policies is critical in today’s environment of rapid change and challenges in addressing policy compliance and effectiveness.  It is essential that effective policy monitoring practices be in place at the Treasury Board Secretariat (TBS) to ensure the success of TB policies.  This review was requested by the TBS Internal Audit and Evaluation Committee to provide TBS management with an independent study of leading practices in the area of policy monitoring.  The review was designed to assess the current status of policy monitoring within TBS and provide practical examples of leading practices.  The recommendations made in this report will provide guidance for the Policy Review and Reporting Project as it proceeds to streamline TB policies and reporting requirements.
Leading Practices Research
The review has researched leading practices for policy monitoring as documented in literature.  Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following:
•	Effective working relationships with departments to ensure provision of information to the central agency;
•	Strong intelligence gathering skills on policy outcomes and processes;
•	Capacity to understand, process, and utilize information based on robust internal analytical processes;
•	Internal information-sharing linking strategic priorities and policy development to monitoring information;
•	Clear understanding of the central agency’s objectives and the ability to promote these to departments; and
•	Ongoing evaluation of the information from departments and assessing its consistency with information prepared and processed by the central agency.
Proposed TBS Framework for Policy Monitoring (the “Framework”)  
The review also studied policy monitoring practices at TBS by reviewing seventeen TB policies.  Based on our findings, leading practices have been presented in a framework to provide a tool for TBS to effectively analyze policy outcomes and monitoring activities in a consistent and systematic manner.  
There are four levels of focus in the Framework:
1. Level of Risk – Risk is the overriding factor that impacts all elements of policy monitoring.  Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place.  The continued effectiveness of these procedures should be evaluated on an on-going basis.
2. Policy Objectives and Outcomes – Policies are designed to support the achievement of an organization’s objectives and the management of risks.  The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:
•	Policy objectives are clear and concise; 
•	Outcomes are well defined and measurable, with performance targets and indicators; and
•	Roles and responsibilities are well defined and communicated to departments.
3. Process Enablers – Policy monitoring “process enablers” are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS policy objectives.  Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes.
The Framework defines seven process enablers.  The one that appears to be the most effectively applied is TBS and departmental working groups and committees.  The process enabler that appears to be the least effectively applied at TBS is committed funding, due to insufficient resources.
4. Monitoring and Reporting Activities – Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up.  Information assists TBS and departments in setting priorities and compiling outcome gap analysis.  Structured reporting relationships are required between TBS and departments to report and analyze outcomes on a timely and systematic basis. 
The Framework defines ten policy monitoring and reporting activities.  The one that appears to be the most effectively applied is TBS’ ongoing advice to departments.  The activities that have been under-utilized or are the least effectively applied are the use of key performance indicators, ongoing reporting, analysis and follow-up by TBS, and dependence on internal audit and evaluation reports.
The Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study.  The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.
Summary of Findings Related to Selected TB Policies
Each element of the Framework was compared to the results from the sample of policies studied.  We grouped the sample of policies by level of risk, and charted the level of monitoring identified in comparison to a desired level.  From this exercise, we made three observations.
1.	Generally, TB policies identified as high risk need to enhance their level of monitoring and reporting activities;
2.	Generally, policies supporting medium risk areas have adequate levels of monitoring; and
3.	Generally policies seen as low risk areas are not receiving the minimum threshold levels of monitoring.
Areas for Improvement
Our study has revealed that certain elements described in the Framework have been adopted within TBS.  We also observed that policy areas generally understand the level of risk, a fundamental component of the Framework.  However there is an inconsistent application of the monitoring elements throughout TBS and certain practices are more effective as TBS monitoring tools.
The report recommends that TBS should adopt the Policy Monitoring Framework as a tool for a systematic approach to defining policy monitoring and reporting for policies that are developed in the future.  A portfolio of the Framework’s process enablers and policy monitoring and reporting activities should be selected and tailored to each policy area based on the level of risk.
Recommendations are also made for improvement in the use of nine of the elements of the Framework.  Of these, TBS should initially focus on the following four areas.
1. Key Performance Indicators  
We recommend that TBS continue to establish key performance indicators (KPIs) as a key requirement for policy development and monitoring.  KPIs should be practical and cost-effective to measure. 
2. TBS Committed Funding
We recommend a more effective use of TBS resources in conducting monitoring of policy outcomes.  Steps would include: 
•	Monitoring high-risk policy areas – Given the large number of TBS policies, an initial approach to adopt the Framework would be to focus resources on high-risk policy areas.  In most cases, structured monitoring may occur at the policy framework level rather than at the individual policy level. 
•	Key elements to monitor – Given the current TBS resource constraints, key performance indicators, policy outcomes, follow-up of departmental action items and implementation are the key elements to focus on.
3. Ongoing Reporting and Analysis
We recommend that TBS adopt a structured reporting approach to policy monitoring including: 
•	Reporting by policy outcomes (policy implementation, policy results);
•	Pre-determined reporting schedule leveraging existing departmental reporting tools and processes; 
•	Improved analytical processes and dissemination;
TBS providing more meaningful and value added feedback to departments comprising government wide trends and benchmarks as well as departmental comparisons.
4. Actionable Items 
We recommend that consideration be given to improving mechanisms for information sharing within TBS and to convert information into actionable items for correction including:
•	Departmental intelligence gathering from multiple TBS in-house sources through the TBS business planning team mechanism;
•	Structured department and TBS reporting of action items against policy outcomes;
•	Structured escalation of deficiencies in action items to the TBS executive level; and 
•	Ongoing reporting between the Secretary and departmental deputy heads, using such mechanisms as the Management Accountability Framework.

Management response
Due to the nature of the report, management has chosen to provide an overall response to the findings and recommendations.  The management response can be found on pages 30 - 32. 

2.0	INTRODUCTION
The successful monitoring of Treasury Board (TB) policies is critical in today’s environment of rapid change and challenges in addressing policy compliance and effectiveness.  It is essential that effective policy monitoring practices be in place at the Treasury Board Secretariat (TBS) to ensure the success of TB policies.  This review was requested by the TBS Internal Audit and Evaluation Committee to provide TBS management with an independent study of leading practices in the area of policy monitoring. 

2.1	Background
As the administrative arm of the Treasury Board (TB), the Treasury Board Secretariat (TBS) has a dual mandate: to support the Treasury Board as a committee of ministers, and to fulfill the statutory responsibilities of a central government agency.
Effective December 12, 2003, the functions and mandate of the Treasury Board Secretariat have been amended.  TBS is now tasked with providing advice and support to Treasury Board Ministers in their role of ensuring value for money as well as providing oversight of the financial management functions in departments and agencies.  The Secretariat also plays a key role in supporting the activities of the Expenditure Review Committee of Cabinet.
The Secretariat makes recommendations and provides advice to the Treasury Board on policies, directives, regulations and program expenditure proposals with respect to the management of the government's resources.  Its responsibilities for the general management of the government affect initiatives, issues and activities that cut across all policy sectors managed by federal departments and organizational entities (as reported in the Main Estimates).  The Secretariat is also responsible for the comptrollership function of government.
Under the broad authority of sections 5 to 13 of the Financial Administration Act, TBS supports the Treasury Board in its role as the general manager and employer of the Public Service.
TBS has recognized the need to update the policy suite to bring it in line with Results for Canadians: A Management Framework for the Government of Canada.  In September 2002, it launched a Policy and Reporting Review Project.  The Project issued a report in September 2003.  
TBS’ 2003 Departmental Performance Report indicated the Policy and Reporting Review Project will streamline TB policies and reporting requirements and lead to a smaller and more strategic set of policies and reporting requirements aligned with the Management Accountability Framework.
A component of the Policy and Reporting Review Project addresses TBS and departmental roles and responsibilities for effective policy monitoring.  This Policy Monitoring Review complements and expands on the practice of policy monitoring as it relates to TBS.
Policy monitoring can be defined as “the assessment of the functioning of policy, and the results achieved.  It addresses, as appropriate, issues of both policy application and compliance.”
Many TB policies have a requirement that departments themselves monitor implementation within their organizations with monitoring of the implementation of the policy across all departments conducted by TBS.  There is also a TB “Policy on Active Monitoring”, introduced in 2001, which requires monitoring across government.  However, we understand this policy is not meeting the needs for policy monitoring.
2.2	Review Purpose
It was stated in the June 2003 Policy Council Briefing that   “For results-based management to work, the policy suite must provide clear and appropriate direction on what actions and results TBS is holding Deputies and their officials accountable for.  In particular, it must set out respective responsibilities of departments and TBS for the monitoring and reporting of policy results.  Information gathered pursuant to reporting and monitoring activity should be used to inform decision-making at departmental, Secretariat and governmental levels”.
The Policy Monitoring Review was designed to assess the current status of policy monitoring within TBS and provide practical examples of leading practices.  The review is intended to provide assistance to management and is not an audit.  The recommendations made in this report will provide guidance for the Policy Review and Reporting Project as it proceeds to streamline TB policies and reporting requirements.
2.3	Scope of Review
The review focused on analysing the current activities undertaken by the TBS for monitoring TB policies to identify effective practices for:
•	Activities undertaken by the various sectors of TBS to monitor the implementation of and compliance with TB policies across government;
•	Activities undertaken by TBS to assist departments to monitor policies; and
•	Processes that will facilitate future monitoring activities.
The planning for this review, the selection of the TB policies considered in the review, and many of the interviews conducted were completed in the fall of 2003, prior to the announcements on December 12, that altered the functions and mandate of TBS.
2.4 	Review Approach and Methodology
Approach
The approach taken to conduct the review included the following steps:
•	Identified best practices by researching documented monitoring frameworks in other organizations;
•	Identified and documented the activities and processes used for monitoring policies in TBS; 
•	Identified differences between TBS’ policy monitoring activities and the best practices; and
•	Identified process and organizational enablers for successful policy monitoring.
Based on the above steps, practical recommendations were developed to guide the policy review process to ensure that revised policies facilitate monitoring activities.
Methodology
The review team selected 17 policies from the inventory of over 300 TB policies.  The sample of policies was chosen, with input from the business line leaders, from the three TBS business lines and included a range from low to high-risk policies and a cross-section of older and newer policies.  The policies included in the sample are listed in Annex A. 
For each of the seventeen policies in the sample, we interviewed one or more TBS policy centre representatives.  We discussed the major issues facing the policies, the policy monitoring and reporting activities and areas for improvement in policy monitoring.  Based on our discussions, we developed an assessment of the level of risk associated with a particular policy and confirmed this with the policy representative. 
In addition to interviewing TBS policy centre staff, interviews were conducted with departments to gain departmental perspectives on policy monitoring and reporting.  The departments in the sample included large and small organizations and are listed in Annex B.
We also interviewed representatives of departmental internal audit groups, one of the TBS program sectors, the TBS Centre of Excellence for Audit and the TBS Centre of Excellence for Evaluation.
The following three documents provided key information for the study:
•	Policy Council Briefing Note of June 2003 entitled “Monitoring and Reporting Requirements in TB Policies;
•	Active Monitoring Report: Report on Use and Impact of the Active Monitoring Policy; and
•	Presentation to the Results For Canadians Committee entitled “Understanding the TB Policy Suite” dated November 2001.

3.0 	REVIEW RESULTS
Introduction 
The review was designed to provide senior management with an assessment of leading practices in the area of policy monitoring that will act as a support tool to TBS’ Policy and Reporting Review Project.
Our research into external leading practices included “thought leadership” in the areas of corporate governance, performance monitoring and reporting, and management control frameworks.  We consulted the work of the Canadian Institute of Chartered Accountants (Study of Control Criteria), Conference Board of Canada, the Committee of Sponsoring Organizations of the Treadway Commission and author Dr. John Carver.  
Practices from New Zealand’s Pacific People’s Monitoring Framework provided valuable insight into monitoring frameworks.
Further, the review team has identified effective practices within TBS as a result of studying the sample of policies from each TBS business line.
The identified leading practices have been consolidated and are presented in the following sections of the report.  They are introduced in our proposed model - TBS Framework for Policy Monitoring (Section 3.1).  Each of the key monitoring elements of the framework is then defined.  The definitions are followed by our findings from the sample of TBS policies studied (Section 3.2).  In Section 4.0, we present our analysis of TBS current monitoring practices against the leading practices of the framework.  The areas for improvement identified in the comparison are set out for consideration.
3.1	Summary of Leading and Effective Practices
Our research indicates that the critical success factors for implementing effective monitoring and reporting include the following: 
•	Effective working relationships with departments to ensure provision of information to the central agency;
•	Strong intelligence gathering skills on policy outcomes and processes;
•	Capacity to understand, process, and utilize information based on robust internal analytical processes;
•	Internal information-sharing linking strategic priorities and policy development to monitoring information;
•	Clear understanding of the central agency’s objectives and the ability to promote these to departments; and
•	Ongoing evaluation of information received from departments and assessing its consistency with information prepared and processed by the central agency.
Based on this information, we developed the framework in Figure 1 to provide a tool for TBS to develop an effective approach to policy monitoring that can be implemented in a consistent and systematic manner.  It highlights the importance of obtaining information about policy performance from departments and to effectively analyze policy outcomes.
The four levels of focus of the TBS Framework for Policy Monitoring (the Framework) as illustrated in Figure 1 are: Level of Risk; Policy Objectives; Process Enablers; and Monitoring and Reporting Practices.

The following explanations describe the four levels of the framework.
1. Level of Risk  
An organization needs to identify its significant internal and external risks on an ongoing basis so that it can react to or initiate changes in an appropriate and timely manner.  Monitoring needs to be structured according to the risk of not meeting the desired policy outcomes.
Risk is the overriding factor that impacts all elements of policy monitoring.  Policies that have greater risk need to be emphasized to ensure they have effective monitoring procedures in place.  The continued effectiveness of these procedures should be evaluated on an on-going basis.
2. Policy Objectives and Outcomes
Policies are designed to support the achievement of an organization’s objectives and the management of risks.  The following elements are fundamental requirements for an effective policy to achieve its desired outcomes:
•	Policy objectives are clear and concise; 
•	Outcomes are well defined and measurable, with performance targets and indicators; and
•	Roles and responsibilities are well defined and communicated to departments.
3. Policy Monitoring Process Enablers
Policy monitoring “process enablers” are mechanisms that assist TBS and departments with information sharing and reporting in order to achieve TBS and departmental policy objectives. 
Policy monitoring is effective where process enablers are systematically applied to develop and implement policy and monitor outcomes.  Process enablers seek to:
•	Provide an information stream between stakeholders to support policy development in strategic areas;
•	Provide legislation that defines roles and responsibilities;
•	Provide mechanisms through which groups work collaboratively to meet policy objectives;
•	Encourage central agencies and departments to establish processes to monitor policy implementation; and
•	Ensure funding and staffing capacity is available to support policy effectiveness and monitoring.
4. Monitoring and Reporting Practices
Monitoring and reporting requires multiple approaches and activities for information gathering and follow-up.  Three policy performance areas requiring monitoring include:
•	Implementation Effectiveness – Strong relationships, the availability of subject matter experts and sharing of best practices between TBS and departments facilitate policy implementation and contribute to continuous improvement in the policy area.
•	Policy Compliance – Information gathered assists TBS and departments in monitoring policy compliance, identifying gaps and initiating corrective adjustments to TB policies.  
•	Outcome Monitoring – Information gathered assists TBS and departments in setting priorities and compiling outcome gap analysis.  Structured reporting relationships are required between TBS and departments to report and analyze outcomes on a timely and systematic basis.

TBS Framework for Policy Monitoring
This study identified seven leading “Process Enablers” and ten leading “Monitoring and Reporting” practices that have been incorporated into our proposed TBS Framework for Policy Monitoring illustrated in Figure 2.  The following sections further define the “Process Enablers” and “Monitoring and Reporting” elements of the Framework.

3.1.1 Policy Monitoring Process Enablers
The Framework’s seven leading process enablers and their desired characteristics are: 
1.  TBS and Departmental Executive Leadership/Sponsorship – TBS and departmental executives’ expectations and accountabilities for achieving policy objectives and monitoring performance are well understood and aligned.  TBS and departmental leadership teams are committed to and focused on policy improvement.  TBS and departmental leadership is well informed, and follows up on the results of actions taken.
2.  Effective Implementation – Policy development and implementation is effective where policy outcomes are well defined, policy schedule and milestone targets are established and these are agreed upon and followed-up by executives.  Support tools, such as training material, guidelines, checklists, and lessons learned accelerate the rollout of the policy.
3. Committed Funding – Adequate resources for monitoring are committed to provide skilled staff and tools for information gathering, key performance indicator monitoring and action item follow-up.  Monitoring activities are designed as an integral part of the organization (A-Base budgets) taking into consideration the risks to achieving objectives and the interrelationship with other monitoring elements.
4. Enabling Legislation – The existence of prescriptive legislation (e.g., Employment Equity Act) with well-defined requirements for annual reports to Parliament, audits, and roles of departmental officials is an effective enabler.  The legislation provides additional authority that focuses on compliance to government priorities.
5. Oversight by External Bodies – The existence of external bodies (e.g., Commissioner of Official Languages) empowered by legislation to conduct independent audits, and follow-up of complaints is an effective enabler.  External bodies provide the independent focus that can bring valuable feedback to TBS and departments.
6. Working Groups/Committees – Formalized working groups, chaired and co-chaired by primary stakeholders, are effective policy enablers.  These formal groups have well-defined deliverables for policy design, implementation and follow-up.  For optimal effectiveness, executive leadership teams from TBS and departments sponsor policy initiatives from design through to implementation and policy support.
7. Integration of Policy Monitoring with Key TBS Financial Processes – Transaction monitoring by TBS (e.g., Privacy Impact Assessments) is a detailed approach to monitoring specific policy requirements.  It is effective in identifying cases of non-compliance or issues identified during a submission review.  In order to be effective, sufficient TBS resources must be in place to handle the volume of submissions.
These seven process enablers represent a basket of effective practices to strengthen the achievement of the policy objectives.  In developing each policy, an optimal combination of these practices should be selected.  The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.

3.1.2	Monitoring and Reporting Practices
The Framework’s ten leading policy monitoring and reporting practices and their desired characteristics are:
1.  Key Performance Indicators Reporting (KPIs) – Organizations use a mixture of quantitative and qualitative performance targets to measure progress towards achieving the defined policy objectives.  These targets are measurable and aligned with the objectives.  Externally derived benchmarks are also used to help ensure targets are aligned with external best practices.  Where performance criteria have pre-established targets and are analyzed against policy outcomes, then the performance indicators provide early warning signals as to whether the targets have not been met.  It is critical to the success of KPIs that departmental KPI reporting is practical and supported by existing reporting systems.
2.  Executive Level Reporting – Executive level reporting and communicating of progress against policy outcomes enables TBS and departments to complete their priority setting and decision-making activities.  The enhancements that are being made to management reporting between TBS and departments through various initiatives, such as the Management Accountability Framework, place greater focus on outcomes and performance indicators.  The terms of executive performance management agreements also endorse achievement of policy outcomes.
3.  Strong Relationships with Departments – Developing and maintaining good working relationships between TBS and departments to ensure provision of information is necessary for ongoing monitoring.  TBS plays an effective advisory role with the departments where it contributes the following: 
•	Policy area subject matter expertise; 
•	Effective knowledge sharing, including posting best practices on websites, and providing training material, guidelines and checklists; and 
•	Both proactive and responsive guidance to departments in the application of policies.
4.  Ongoing Reporting and Analysis – Departments and TBS each have timely and reliable information to evaluate compliance to policy objectives and to measure against expectations, with performance primarily monitored by the departments.  Leading departments have the capacity to understand, process, and utilize information based on robust internal analytical processes.  Central agencies also monitor the information provided by departments and assess its consistency with information prepared and processed by the central agency.
5.  Audit and Evaluation Monitoring and Reporting – Monitoring is best conducted using multiple approaches including ongoing reporting and analysis.  Independent audits and evaluations conducted by internal audit and evaluation and by external bodies provide a fresh focus on the effectiveness of the policy as well as the effectiveness of monitoring.  However, audit and evaluation engagements are structured to the specific needs of the departments, as determined through risk assessments and consultation with senior management.  They are not driven by individual policy requirements.  Therefore, these activities are, at best, viewed as supplementary assessments of the adequacy of the monitoring and reporting processes in place and not as a primary tool for providing feedback on performance in meeting the policy objectives.
6.  Advice to Departments – A good understanding of policy objectives and priorities and the ability to promote these to departments is a critical success factor for effective policy development, deployment and monitoring.  The central agency positions itself as the “trusted advisor” or subject matter expert working collaboratively and in partnership with departments.  The central agency takes a proactive role in identifying and addressing issues in addition to reacting to problems as they arise.
7.  Intelligence Gathering – Monitoring the external environment provides valuable information on the state of the internal environment.  Changes in external conditions (such as changes in regulations, societal trends, and program and political changes) can have a significant effect on departments’ and TBS’ ability to meet their objectives. Intelligence gathering is facilitated by various tools and techniques such as:
•	Information sharing groups (TBS portfolio teams); and
•	Financial planning and reporting tools.
8.  Knowledge Sharing Tools and Processes – Sharing best practices and lessons learned from stakeholder departments and TBS promotes effective policy implementation.  Information obtained and shared provides a range of approaches to addressing policy concerns and the achievement of policy outcomes.  Building a culture of knowledge capture and sharing further enhances the TBS’ leadership and “trusted advisor” role with departments. 
9.  Communication – Open communication both creates and depends on trust.  A high level of trust encourages people to ensure that everything of importance is known by others.  Communication processes support common values and achievement of objectives.  For monitoring to be effective, there are communication processes capable of supporting two-way, open communication of timely, relevant and reliable information.  These include:
•	Broad Base Communication Practices:  Ongoing communication around policy issues, development and improvements (e.g., bulletins and websites).
•	Performance Reporting:  Structured information gathering (e.g., surveys, formal meetings) and dissemination of results from action plans and follow-up of policy outcomes.
10.  Actionable Items - A well established reporting and follow-up protocol between departments and TBS is in place to implement corrective action for policy outcome gaps.  
These ten policy monitoring and reporting activities represent a basket of effective practices to strengthen the feedback mechanisms for assessing the achievement of policy outcomes.  In developing each policy, an optimal combination of these practices should be selected.  The appropriate combination or portfolio will vary for each policy area based on the assessed level of risk and on other factors.

3.1.3	Adoption of the Framework for Policy Monitoring
The Policy Monitoring Framework has been presented to and discussed with the representatives of the TBS policy centres, and other TBS staff we interviewed in the study.  The Framework was well received and there was a general consensus that adoption of the Framework will lead to improved policy monitoring.
Recommendation
We recommend that TBS adopt the Policy Monitoring Framework as a tool for taking a systematic approach to defining the policy monitoring approach for policies that are developed in the future.
We also recommend that TBS adopt a portfolio of the Framework’s process enablers and policy monitoring and reporting practices that is tailored to each policy area and level of risk to improve policy monitoring.

See Management Response page 30-32
3.2	Summary of Findings Related to Selected TB Policies
Seventeen TB policies were selected for review with key TBS and departmental representatives responsible for the selected policy areas.  The purpose was not to critique or comment on specific policy issues.  Rather, the objective was to identify and discuss general trends and issues in policy monitoring including:
•	Level of risk – The sample of policies selected included higher risk and politically sensitive areas as well as routine and stable policy areas.  We discussed the level of risk associated with each policy with TBS and departmental policy centre representatives.  
•	Approach to policy monitoring – The approach to and level of monitoring activities varied across all policies and our analysis assessed effective levels of monitoring.  These also were discussed with key policy stakeholders.
Based on our analysis, we have grouped the seventeen policies by their level of risk and our assessment of the level of monitoring activity and presented the results in Figure 3.  The level of risk is presented on the horizontal axis (Low/Medium/High) and the level of monitoring is presented on the vertical axis (minimum/moderate/enhanced).  Also presented in Figure 3, is the “Desired State”, which is the recommended level of monitoring associated with each level of risk.  Effective policy monitoring requires a balance between TBS monitoring activities and the level of risk associated with the policy area.

Following are three observations with respect to the level of monitoring being conducted from the policies studied in the review.
1. Strategic Impact – Enhanced Level of Monitoring – Generally TBS policies identified as high risk areas need to enhance their level of monitoring and reporting activities to address their strategic significance.  These policies in a number of cases have less monitoring than policy areas with medium risk.
2. Operational Excellence – Moderate Level of Monitoring – Policies supporting medium risk areas generally have adequate levels of monitoring.  From our study, it would appear that some policy areas with medium risk are over monitored while other medium risk policies are not sufficiently monitored. 
4.	Operational Stability – Minimum Level of Monitoring – Policies that are seen as low risk areas require a minimum level of TBS monitoring to ensure stability is maintained.  Generally, low risk policies were not receiving minimum threshold levels of monitoring.

4.0	AREAS FOR IMPROVEMENT
Throughout the study, we observed that certain elements described in the Framework have been adopted within TBS.  We found that certain practices are more effective than others as TBS monitoring tools.  We also observed that policy areas generally understood the level of risk, a fundamental component of the Framework.  However there is an inconsistent application of the monitoring elements throughout TBS.

4.1	Analysis of Effectiveness
After completing our interviews with the representatives of the TBS policy centres and departments, we summarized our understanding of the effective monitoring practices, areas for consideration, level of risk, and constraints to monitoring for each of the policies in the sample.  We met again with the TBS policy centre representatives to confirm our understanding, to present the Framework and to discuss it in relation to the policy.
Next we completed an analysis by assessing the effectiveness of each of the Framework elements for each of the 13 policies in the sample that were identified as medium to high risk.  We combined the assessments of effectiveness for these individual policies and generated a relative effectiveness rating for each of the Framework elements.  The overall rating of effectiveness for any element is influenced by the number of policies in which the element is present and by the assessed effectiveness of the element in each of the policies.

We have illustrated the results of this analysis in the bar chart in Figure 4.  Each bar represents one of the leading practices defined previously in the report by Framework element and is measured against our assessment of a low, medium or high effectiveness rating.

There are a number of elements of the Framework that appear to be effectively used by TBS.  For example, as illustrated by the bar chart in Figure 4, “Working Groups and Committees” and “Advice to Departments” have been assessed as effective practices.  There are some other elements in the Framework, such as “Intelligence Gathering” and “Integration with Key TBS Financial Processes”, which we consider less important for TBS to address than others.  We will not discuss the elements in these two groups further in the report; however, summary observations, including TBS effective practices, are presented in a table in Section 4.2.10.
We observed that some of the other elements included in the Framework, such as “Committed Funding” are not consistently applied and provide opportunities for improvement.  We consider other elements of the Framework, such as “Key Performance Indicators” to be very important and require special attention.  In the next section, we set out our observations and discuss our recommendations for improvement for the eight Framework elements that fall into these two groups.

4.2	Areas for Improved Policy Monitoring
The Framework has defined nineteen elements for effective policy monitoring.  We have identified nine of these elements with areas for improvement, including eight that were selected from the chart in Figure 4, plus level of risk.  The nine elements are:
•	Level of Risk
•	Policy Objectives and Outcomes
•	Key Performance Indicators
•	Committed Funding
•	Ongoing Reporting and Analysis
•	Effective Implementation
•	TBS and Departmental Executive Leadership/Sponsorship
•	Actionable Items
•	Audit and Evaluation
These are described in the following sections.  Recommendations are included in the areas for consideration.
4.2.1	Level of Risk
Observation
There is an absence of systematic and formalized ranking and monitoring of the level of risk associated with the policies.  
Historically the number of policies has continued to grow in number without any significant elimination or consolidation of less relevant or outdated policies.  As a result, TBS has directed each of its business lines to review the number and content of policies and improve upon their effectiveness.
Areas for Consideration
Risk management is a one of the four levels of focus in the Framework for Policy Monitoring.  This is consistent with the TBS Management Accountability Framework and modern management practices.
We recommend that TBS establish a standard set of criteria for evaluating risks associated with each policy area.  Criteria would include the impact on departments and departmental risk areas.  Additional steps for consideration include:
•	Ensure that risk assessments for policies are reviewed on an ongoing basis;
•	Ensure that the level of policy risk determines the appropriate levels of monitoring; and
•	Ensure that the policy monitoring capacity (resources and skills) is aligned with the level of monitoring desired for moderate to high-risk areas in both TBS and departments.

4.2.2	Policy Objectives and Outcomes
Observation
Generally the policy objectives and monitoring requirements supporting the TBS policies we studied are broad statements without clarity of outcomes or success criteria.  New policies being developed are focusing attention on improved clarity around outcomes, key performance indicators and roles and responsibilities.  For example, the policy on Continuous Learning adopted a different policy format from traditional TBS policies by presenting departmental commitments, expectations, timeframes and providing a detailed description of roles and responsibilities. 
Areas for Consideration
As part of TBS’ policy renewal effort, we suggest creating improved policy objectives and outcomes that reflect the following:
Roles, responsibilities and accountabilities for policy outcomes of TBS and departments are well‑defined in the policy;
TBS provides accurate outcome analysis, identifying outcome gaps and trends across departments;
TBS capacity exists to prepare outcome gap reports, summaries and analysis.  Information will be aligned with results from other planning and reporting processes (e.g., Management Accountability Framework) prepared by TBS and departments; and
Policy outcomes including policy implementation schedules are clearly communicated and actionable.
4.2.3	Key Performance Indicators
Observation
The establishment and monitoring of key performance indicators that support policy outcomes was the most significant gap from the sample of policies studied.  Improvements in data and information capture infrastructure and tracking of performance indicators has been an area of increased focus for TBS policies under development (e.g., Employment Equity uses performance indicators to measure performance).
Areas for Consideration
We recommend that TBS continue with the establishment of key performance indicators (KPIs) as a key requirement for policy development and monitoring.  Steps for consideration include:
•	Performance criteria are pre-established and analyzed with policy outcomes;
•	Departmental KPI reporting is practical and systematic (minimize manual and ad hoc reporting requirements);
•	TBS assists departments to develop a standard set of KPIs per policy area; and
•	Where practical, KPIs are aligned to executive performance.

4.2.4	Committed Funding
Observation
A common issue raised by TBS interviewees was that limited resources adversely impacts TBS’ ability to conduct policy monitoring and reporting.  Furthermore, TBS has faced staff retention issues and its ability to attract top talent in policy areas has been a challenge.
In the case of most policies studied, A-Base funding provides resources to the policy centres within TBS.  Although we did not study the level of resources assigned by policy, it is apparent that the amount of resources and consequently the approach for monitoring varies considerably within TBS.  In some cases the variation in monitoring can be explained by the level of risk associated with the policy.  However, in others, the relationship between risk and resources was not apparent.
In the case of two policy areas (Modern Controllership - Integrated Risk Management and Federal Contaminated Sites Accelerated Action Plan- Contaminated Sites Management Policy) implementation is supported by project funding.
Areas for Consideration
We recommend that TBS assign resources to monitoring activities by adopting the following strategies:
•	Monitoring at the Policy Framework Level.  In the future, structured monitoring may occur at the policy framework level rather than at the individual policy level.  This will facilitate increased monitoring in higher risk areas.
•	Capacity/Resources for Monitoring.  Given the large number of TBS policies, an initial approach to adopt the Framework would be to focus resources on high-risk policy areas.
•	Key Monitoring Elements to Focus.  Given the current TBS resource constraints, key performance indicators, policy outcomes, follow-up of departmental action items and follow-up of implementation are important elements to focus on for the initial adoption of the Framework. 

4.2.5	Ongoing Reporting and Analysis 
Observation
TBS continues to evaluate its reporting requirements with departments through various initiatives including the Management Accountability Framework.
Reporting tends to be statistical with limited follow-up by TBS.  Reporting practices varied considerably within the sample of policies studied.  These include the following:
•	Annual reporting of statistics, financial indicators and volumes (Prevention and Resolution of Harassment, Access to Information, Contracting, Contaminated Sites, Continuous Learning);
•	Formal TBS analysis and feedback to departments (Employment Equity, Official Languages); 
•	Policy implementation status reporting (Government Security Policy, Integrated Risk Management, Transfer Payments); and
•	Action item tracking and follow-up (Contracting).
Qualitative analysis was minimal and was generally attributed to lack of TBS resources.
Areas for Consideration
We recommend adoption of additional ways to enhance monitoring and reporting including the following:
•	Structured Reporting Approach – Adoption of a structured approach to reporting for each policy framework would include the following elements:
o	Reporting by policy outcomes (policy implementation, policy results);
o	Pre-determined reporting schedule (quarterly, yearly); and
o	Leveraging existing departmental reporting tools and processes.
•	Analytical Processes and Dissemination – TBS has a significant number of data gathering processes and information sources within its organization (Program Sectors, Centres of Excellence, Policy Centres, departmental executive reporting).  More meaningful and value added feedback to departments would include leading practices, government-wide trends, analysis, departmental comparisons and benchmark reporting.

4.2.6	Effective Implementation 
Observation
Our study identified a number of effective practices for policy implementation which include the following:
•	Data Gathering on Progress of Policy Implementation – Policies, once rolled-out, require ongoing post implementation support and monitoring.  For example, Integrated Risk Management will continue to conduct surveys gather information and prepare TBS reports on implementation progress.
•	Implementation Tools – Certain sampled policies have supported implementations with a number of tools and techniques:
o	Communication tools/website (Integrated Risk Management); 
o	Implementation guidelines, checklists, audit guidelines and training material (Privacy Impact Assessment, Integrated Risk Management, Contracting); and
o	Mid-term reports (Government Security Policy and Continuous Learning).
•	Pilot implementation – A number of policies have been designed, developed and implemented initially as a pilot in key stakeholder departments, followed by a broader rollout to other departments (Integrated Risk Management Framework, Privacy Impact Assessments).  This approach allows for the application of lessons learned, and refinements to the policies, support tools and guidelines in an accelerated manner prior to the full rollout of policy.
Areas for Consideration
In addition to the effective TBS implementation tools referred to above, we recommend TBS consider more consistent adoption of the following practices to assist in monitoring policy implementation:
•	Implementation Working Team – A sub-team to the leadership team is involved from design through implementation and support.
•	Well-defined and Timely Implementation Milestones – Implementation targets are clearly defined in the policy outcomes.  These will include the following:
o	Establishment of key performance indicators supporting policy outcomes and objectives; and
o	Departmental implementation milestones, which include tailoring the policy to address department specific issues, departmental training and reporting on policy outcomes.
•	Monitoring Policy Implementation – Progress is measured against the milestones including the following:
o	Ongoing review of implementation progress by department; and
o	Independent audit/evaluation of implementation and effectiveness of policy at a predetermined milestone (e.g., mid-term report). 
•	Implementation Support – Roll-out of the policy is accelerated with implementation supported with the following change management elements:
o	Training material;
o	Guidelines and checklists to assist departments with implementation;
o	Lessons learned and best practices from pilot and early adopters of policy;
o	Provision for TBS subject matter experts available to support implementation.

4.2.7	TBS and Departmental Executive Leadership/Sponsorship
Observation
From the policies studied, we observed several practices demonstrating executive leadership from TBS and key stakeholder departments that supported policy direction, development, and monitoring.  The Clerk of the Privy Council also provides leadership in some areas in terms of establishing government-wide management priorities.  The leading practices we observed include:
•	Performance management agreements require the achievement of policy outcomes (Integrated Risk Management Framework, Employment Equity);
•	TBS Secretary conducts follow-up of compliance to policy implementation and deliverables (Integrated Risk Management);
•	TBS Secretary and deputy heads reporting through the Management Accountability Framework process; and
•	TBS executive sponsors are active in the design and implementation of policy (Privacy Impact Assessments).
Areas for Consideration
For high-risk policy areas, we recommend that TBS develop an escalation protocol and procedures within TBS and with departments to report performance exceptions to TBS executives.  This will require establishment of ongoing reporting and analysis of policy outcome gaps.

4.2.8	Actionable Items 
Observation
The study revealed that TBS is challenged by resource constraints impacting active follow-up and monitoring of policy outcomes and performance.  Follow-up and reporting does not appear to be consistent and systematic.
Areas for Consideration
We recommend that consideration be given to improve the mechanisms for information sharing within TBS and to convert information into actionable items for correction including:
•	Departmental intelligence gathering from multiple TBS in-house sources such as the TBS portfolio teams;
•	Structured department and TBS reporting of action items against policy outcomes;
•	Structured escalation of deficiencies in action items to TBS executive level; and
•	Ongoing reporting between Secretary and departmental deputy heads.
4.2.9	Audit and Evaluation Monitoring and Reporting
Observation
A majority of the policies studied make reference to reliance on the conduct of departmental internal audits as the key, and in many cases, sole identified monitoring mechanism.  However, these specific responsibilities are not consistent with the role of internal audit.  The Policy on Internal Audit affirms the function of internal audit as a provider of assurance services to departmental management.  Audit engagements are structured to the specific needs of the organization, as determined through risk assessments and consultation with senior management.  While internal audits provide assurance that the appropriate monitoring and control frameworks are in place, they should not be considered the primary mechanism for monitoring and reporting on individual policies.  Internal audit plans are department focused and therefore do not provide consistent coverage across the government.
Evaluations are distinct from audits.  While audit provides assurance services, evaluation is a management tool that has two main purposes:
•	To help managers design or improve the design of policies, programs and initiatives; and
•	To provide, where appropriate, periodic assessments of policy or program effectiveness, of impacts both intended and unintended, and of alternative ways of achieving expected results.
There are nine policies in the study sample with a specific requirement for departmental internal audits or evaluations of the policy.  However, Contracting, Transfer Payments and Government Security Policy are the only policies where we found that internal audits or evaluations are being completed.
We observed that information from departmental audits is not easily shared and disseminated in a meaningful way within TBS.  Internal audit reports are circulated across TBS, and are reviewed independently in many areas, but there does not appear to be a centralized approach to assessing the impact of the information contained the audit reports.
Area for Consideration
We recommend that the requirement for internal audits of compliance of specific policies be discontinued as a generic policy requirement since it does not reflect the way in which internal auditing is planned and conducted.  The risk-based audit approach undertaken by TBS and departments does not align with the specific policy monitoring required for TBS and departments.  Internal audit should be viewed as a provider of assurance over and above the other monitoring and reporting mechanisms adopted for policies.  If appropriate, the policy could refer to the role of internal audit by making reference to the Policy on Internal Audit.
We also recommend that consideration be given to identify emerging risks and share audit results including:
•	Integration of Policy Centre Analysis with Centres of Excellence Reviews of Audits and Evaluations – Formal monitoring of policy risk is conducted by Policy Centres and is communicated to the TBS Centre of Excellence for monitoring purposes.
•	Knowledge sharing of departmental audit and evaluation findings – Ongoing communication of audit results is channeled through TBS Centres of Excellence.  Formal analysis and assessment is shared with Policy Centres and Program Sectors through existing TBS channels (e.g., portfolio teams and TBS communities of interest).
•	Collaborative Approach to Follow-up of Results – Follow-up of departmental audit plans and results and identification of emerging areas of policy risk is a shared responsibility of Centres of Excellence, Program Sectors and Policy Centres.  TBS portfolio teams may be a key enabler that can be re-established to address issues and strategies for follow-up.

4.2.10 Other Observations
In developing the Framework, a number of effective TBS monitoring activities from select policies were identified and are included as key elements. These are briefly listed in the table below.   
Element of Framework	TBS Policies	TBS Effective Practice	
Existing Legislation
External Bodies	Employment Equity 
Official Language
Access to Information	•	Audit by External Body (legislative)
•	Annual Reporting to Parliament
•	Performance review and “report cards”	
Effective Working Groups/Committees	Federal Contaminated Sites Management Policy
Integrated Risk Management (IRM)
Official Languages	•	Contaminated Sites Working Committee
•	Implementation Council (IRM)
•	Departmental Advisory Committee on Official Languages	
Integration with Key TBS Financial Processes	Contaminated Sites Management Policy
Privacy Impact Assessment	•	Program funding from Federal Contaminated Sites Accelerated Action Plan
•	TB Project Submissions	
Executive Level Reporting	Employment Equity 
Official Language
Access to Information	•	TBS Annual Reports to Parliament
•	TBS Reports to President of Treasury Board
•	Departmental reports to Parliament	
Communications	Integrated Risk Management	•	Websites	
Intelligence gathering	Various	•	TBS Program/Policy Forum	
Knowledge Sharing Tools	Privacy Impact Assessment
Integrated Risk Management
Contracting	•	Best practices 
•	Training material
•	Implementation guidelines
•	Audit Criteria	
Advice to departments	Government Security Policy	•	TBS Subject Matter Expertise	
Strong Relationships with departments	Contaminated Sites Management Policy
Integrated Risk Management	•	Smaller communities and stakeholders

MANAGEMENT RESPONSE
The senior management of Treasury Board Secretariat (TBS) acknowledges the necessity of having effective policy monitoring practices in place at TBS and generally agrees with the recommendations stemming from this review, which aim to improve these practices at TBS.  
Leading practices in policy monitoring activities at TBS were identified and presented in a Policy Monitoring Framework (“the Framework”) as a tool for TBS to achieve a systematic approach to defining policy monitoring and reporting for policies that are developed in the future.  It was suggested that adoption of the Framework would lead to improved policy monitoring.  
We agree to take steps towards adopting the critical elements of the proposed Policy Monitoring Framework (“the Framework”) outlined in the report including, as a priority, four of the nine recommended areas of focus plus one additional area:
•	Key Performance Indicators
•	TBS Committed Funding
•	Ongoing Reporting and Analysis
•	Actionable Items, and
•	Levels of Risk.
On the issue of risk, we agree with the general proposals for evaluating the risks associated with each policy area, including the needs for periodic risk assessments to determine appropriate policy monitoring levels, and for ensuring the alignment of necessary capacity (resources and skills) to accomplish policy monitoring.  Specific attention will be paid to the way in which high-risk policies are monitored, and how structured monitoring can be built in to the policy framework. 
In the context of the present TBS Executive review of strategic business re-engineering initiatives (of which the TB Management Policy Review is one), we will review the internal distribution of resources to support the necessary monitoring of high-risk policies, and recommend necessary incremental investments or resource reallocations to that end.  These decisions will be made during this fiscal year.
As recommended by the Review of Policy Monitoring, and consistent with the principles of the current TB Management Policy Review, we will continue to consider and implement improvements in the areas of:
•	Policy Objectives and Outcomes
•	Effective Implementation
•	TBS and Departmental Executive Leadership/Sponsorship, and
•	Audit and Evaluation.
In general, we will ensure that policy monitoring is conducted in a way consistent with the new approach to strategic oversight and risk assessment being adopted by TBS – including through related assessments of departmental and government-wide management and expenditure performance, based on the Management Accountability Framework.  We will address the identified requirements for policy monitoring during the course of the TB Management Policy Review, whose main timelines for implementation are as follows:
•	Content review for procurement and real property policies – by March 31, 2005
•	Content review for all other policies – by March 31, 2006
•	Full implementation for all policies – by March 31, 2007.
General Remarks
The recommendations made in the review are meant to provide guidance to the TB Management Policy Review, as it proceeds to streamline TB policies and reporting requirements.  Through the work of the TB Management Policy Review, a coherent and consistent approach to these recommendations will be taken across the policy suite.  
The TB Management Policy Review will examine ways in which policy monitoring and reporting requirements can be better linked to the Management Accountability Framework and the Expenditure Management Information System.  This will help to ensure better knowledge capacity, better knowledge transfer and less redundancy in the reporting of results.
The review suggests that “the requirement for internal audits of compliance of specific policies be discontinued as a general policy requirement since it does not reflect the way in which internal auditing is planned and conducted.”  Recognizing that risk-based audit planning practices are accepted across government, we appreciate the argument for discontinuing general policy requirements for internal audits of compliance within policies.  We will look at possibilities for discontinuing the audit requirements articulated in specific policies, on a policy-by-policy basis.  However, existing requirements will not be discontinued outright until other options for providing similar oversight capacity are identified and in place.  Until then, the requirement for internal audits of compliance with specific policies will remain.
We would suggest that specific audit requirements within policy instruments should be contemplated for circumstances of generalized risk related to the policy, and in consultation with TBS internal audit policy experts.  In the context of broader reviews (e.g. to strengthen compliance and sanctions related to the Financial Administration Act), we are presently exploring the range of mechanisms for ensuring effective compliance with policy and legislation.  The resulting recommendations will influence how we address audit requirements in the TB policy suite.

5.0	CONCLUSION
Our study has revealed that TBS conducts effective policy monitoring in certain policy areas.  We also observed that policy areas generally understood the level of risk, a fundamental component of the Framework.  However there is an inconsistent application of the monitoring elements throughout TBS and certain practices are more effective as TBS monitoring tools.
The recommendations within the report are:
1.	TBS should adopt the Policy Monitoring Framework as a tool for taking a systematic approach to defining the policy monitoring and reporting approach for policies that are developed in the future.
2.	A portfolio of the Framework’s process enablers and monitoring and reporting practices, tailored to each policy area and level of risk, should be adopted to improve policy monitoring.  Generally TBS policies identified as high risk areas need to enhance their level of monitoring and reporting activities to address their strategic significance.
3.	The following leading practices included in the Framework are areas that should be addressed to improve monitoring and reporting for TB policies:

Key Performance Indicators
Level of Risk
Committed Funding
Ongoing Reporting and Analysis
Policy Objectives and Outcomes
Effective Implementation 
Actionable Items
Audit and Evaluation
TBS and Departmental Executive Leadership/Sponsorship
A more detailed description of the recommendation for each practice is included in the body of the report.
The areas that TBS should focus on initially to strengthen the achievement and monitoring of policy outcomes are:
•	Establish key performance indicators;
•	Strengthen resources committed to monitoring activities;
•	Establish ongoing reporting and analysis; and
•	Follow up on actionable items.				

ANNEX A 		List of Policies Included in Review 

Employment Equity Policy
Official Languages 	- Language of Work in “Bilingual “ Regions
- Communications with the Public
Access to Information
Federal Contaminated Sites Management Policy
Privacy Impact Assessments Policy
Government Security Policy
Integrated Risk Management Framework including Risk Management Policy
Policy on the Prevention and Resolution of Harassment in the Workplace
Transfer Payment Policy
Contracting Policy
A policy for Continuous Learning in the Public Service of Canada
Policy on Delegation of Authorities
Policy of Delegation of Organizational Authority
Policy on Electronic Authorization and Authentication
Smoking in the Workplace
Telework Policy

ANNEX B 		List of Departments Interviewed

Agriculture & Agri-food Canada 
Canada Revenue Agency
Canadian Heritage
Canadian Institutes of Health Research
Citizenship & Immigration Canada
Correctional Service Canada
Health Canada
Human Resources Development Canada
Indian and Northern Affairs
Industry Canada
National Defence
Public Works and Government Services Canada
Transport Canada