Section 2: Analytical Framework for Selecting Instruments

This section presents an analytical framework to facilitate a disciplined approach to assessing, selecting, and implementing instruments. Departments and agencies may use the framework as is or as a template for developing their own framework for their respective areas of responsibility.

The framework sketches out a sequence of enquiry, specifies a methodological foundation, and provides guidance for each step in the instrument choice process.

Analytical Framework for Selecting Instruments

Analytical Framework for Selecting Instruments – Text version

Benefits of using an analytical framework

Several benefits flow from the use of an analytical framework for selecting instruments:

Overarching rationales for the framework


Seeking an Optimal Policy Mix

The Conference Board of Canada considers stakeholder participation crucial to identifying issues and risks (problem identification), assessing and selecting instruments, and evaluating impacts. The Board proposes the following approach for environmental policy making:

  • Develop objective: Consider the needs and objectives of all stakeholders [...]. Stakeholders include those contributing to the problem, those affected by the problem, and those with the potential to help address the problem.
  • Select instruments: Consider all stakeholders, particularly in terms of their rights, roles, and responsibilities.
  • Evaluate impacts: Consider all stakeholders who may be positively or negatively affected.

Conference Board of Canada, The Optimal Policy Mix: Matching Ends and Means in Environmental Policy Making, 2000.

The following framework is based on two overarching rationales:

The framework is not intended to be a sequential roadmap of where and how officials should assess instruments to achieve public policy objectives. The process is inherently iterative in that the accumulation of information and knowledge concerning a problem or situation and the objectives that the government is aiming to achieve will require officials to repeatedly revisit each of the steps in the framework.

Given the complexity of policy making, ongoing consultation with actors and institutions that can have an effect on the risks and objectives being addressed is a crucial component of this iterative process. Consultation not only improves the government's instrument selection capacity, but also increases Canadians' trust in government and encourages knowledge sharing among government, industry, and non-governmental organizations.

Step 1: Identify and define the problem: issues, risks, causal factors

The first step in any policy analysis is to identify and define the key features and sources of the problem. The knowledge generated at this stage is crucial to making good decisions concerning policy priorities, instrument choice, performance expectations, and (if "rules" are part of the instrument mix) strategies to achieve conforming behaviour.

Defining the Problem

How a problem is defined influences the selection of instrument(s) to address that problem. A simple example would be that of panhandling in the streets. If panhandling is defined as a nuisance and a potential danger, one option is to use law to prohibit the activity. However, if panhandling is seen as part of a larger problem such as poverty, lack of education, disability, etc., the intervention instruments will be different and may include outreach programs, workplace training, and/or other types of community support programs.

Increasingly, the problems that governments are asked to deal with are characterized in terms of "risk." Governments have to make decisions about whether and how to intervene in situations involving a high degree of uncertainty regarding what is going on now and what may happen in the future. Risk, which can have both positive and negative connotations, is a concept that can help decision makers cope with the uncertainty inherent in governance activities, discipline public policy analysis, and foster more rational approaches to instrument choice.

Government intervention in response to a problem cannot change the past or even the present; it can only change the future. Moreover, government's ability to manage risks in the near term is often quite limited in that the die is already cast and a government's ability to devise and activate an appropriate mix of instruments is subject to significant time delays. The further a government's action extends into the future, the greater is the government's ability to influence the level of risk, but also the greater is the uncertainty about how the underlying systems that determine the risk will evolve.

To appropriately address a public policy issue, government needs a sense of where the problem is headed, how far and how fast–the dynamics of risk. Basing decisions on "snapshot" assessments of past or current risk makes sense if the risk is static and likely to remain that way throughout the time horizon relevant for public policy choices and program implementation. However, the risks that government must address are usually the result of complex interactions of social, economic, and environmental systems. These systems themselves are in a constant state of flux, which means that the risk could change over time and perhaps in ways that are not intuitively obvious.

Playground Injuries

Playground Injuries – Text version

In most circumstances, risks are not constant and are likely to change over time, both in an absolute sense and relative to each other. This should be kept in mind during the problem analysis stage. Examining how the risk evolved in the past can provide important insights into how it may behave in the future, but investigating why it has changed may be even more important. Because correlations or linear projections of past trends do not always predict the future path, it may be prudent to make the effort to understand the causal influences that determine risk and how they function together as a system. Understanding why risk has changed in the past is an essential foundation for informed policy priority setting and for selection of instruments that will make a difference in how the risk evolves in the future. Developing knowledge regarding the systemic determinants of risk is the first step in any rigorous risk analysis and, in one form or another, has always been the hallmark of exemplary policy analysis.

In keeping with the Cabinet Directive on Streamlining Regulation and the Canadian Cost-Benefit Analysis Guide, a dynamic risk analysis methodology could be an integral step in the development of a rational approach to instrument choice. In reality, dynamic risk analysis is just good, disciplined public policy analysis, recast in "integrated risk management" terms.

Dynamic risk analysis involves describing how the risks have evolved over time, why the risks have occurred, the current level of risk, and how the risks are likely to evolve in the future with and without government intervention. What is most important is understanding the way in which the risks are evolving and the reasons for that evolution. In most cases, the creation of a logic model can help comprehension of how the causal determinants of risk affect ultimate outcomes.

Dynamic risk analysis also explicitly and transparently acknowledges the uncertainty inherent in public policy analysis, including limitations on the government's ability to accurately assess risk levels over time, understand the complex interactions of causal influences, and predict how intervention may change the trajectory of risk in the future.

The dynamic risk analysis approach provides a structured, highly disciplined, more realistic basis for performance assessments of intervention, with a focus on continuous learning and improvement in both policy design and program implementation.

Step 2: Set objective(s)

The second step involves articulating the policy goal(s) and desired outcomes. There may be several policy goals, each with a variety of outcomes. While policy goals and outcomes go hand in hand, they are not quite the same. For example, a goal might be to make a particular activity safer, while the desired outcome might be a 30-per-cent reduction in the rate of injury.

Goal versus Outcome

Goal versus Outcome – Text version

Setting goals and outcomes involves addressing the question of which risks are worth reducing and to what levels, while trying to balance the risks, costs, and benefits of action. Further guidance on balancing risks, costs, and benefits can be found in the Canadian Cost-Benefit Analysis Guide. This process will help to identify the instruments that will best manage the risks and will ensure that policy considerations drive the choice of instrument (ends drive the means), rather than the other way around.

A policy goal articulates desired risk management outcomes that would be acceptable in the public interest. The benefit incurred through intervention can be represented by the gap between the projected evolution of risk (i.e. in the absence of intervention) and the desired risk management performance that will be achieved over time as a consequence of the policy intervention. Performance indicators (discussed in Step VI) provide an indication of the effectiveness of an instrument or mix of instruments in achieving the desired risk management policy goal on an ongoing basis.

One avenue that should always be considered is that of taking no action. The idea that government needs to act assumes there must be a response to every situation. A rational approach to instrument choice includes the option of deliberately doing nothing. Maintaining the status quo or refraining from intervening may appear to be a non-decision, but when properly evaluated and applied in the right circumstances, it is a legitimate response known as a "static response."

As mentioned previously, adjustments may have to be made to the objectives as information and analysis evolve over the next steps in the process.

Step 3: Identify potential intervention points

When considering a problem and objectives, it may be possible to identify different points at which policy intervention could occur. Taking into account the actors and institutions that could have an effect on the risks or objectives of the problem or situation, there may be certain critical points at which government intervention would be most effective. There are often two frames of reference for specifying the "point" at which government intervenes in the economy or society: the specific point in time in a sequence of events and/or the particular physical target that can be distinguished from other targets. It may be necessary to revisit the objectives to determine the possible outcomes of different points of intervention.

Hazard Analysis and Critical Control Point System

The Hazard Analysis and Critical Control Point System (HACCP) is a food safety system in which critical points in the food-processing chain are identified and controls put in place to ensure that food safety hazards are eliminated. It differs from random sampling as a food safety system in that it focusses on preventing problems before they occur rather than trying to detect failures through end-product testing.

Although HACCP is an industry self-inspection program, it illustrates how government intervention points can be internalized within a program. The U.S. Food and Drug Administration, for example, notes seven steps that seafood companies are to follow to ensure seafood safety.  Each processor must:

  1. identify potential hazards specific to the seafood and determine how to minimize these risks
  2. identify the critical risk control points of seafood, such as cooking or storage, so as to minimize or eliminate the risk
  3. establish preventive measures with critical limits for each control point
  4. establish procedures to monitor the critical control points
  5. establish corrective action to be taken when the critical limits for each control point are not met
  6. establish means by which the system can be verified
  7. establish effective recordkeeping

The FDA's HACCP regulations also require seafood processors to write and comply with sanitation standards specifying the safeness of the water used in food preparation, the cleanliness of surfaces and all instruments that come in contact with the food, and proper maintenance of the food processing equipment.

In Canada, the Canadian Food Inspection Agency uses the HACCP to carry out its Food Safety Enhancement Program.

Step 4: Identify actors/institutions that can have an effect on risks or objectives

The fourth step involves identifying the actors and institutions that should be involved in addressing the problem or situation. The government's Good Governance Guidelines urge officials to explore the use of collaborative arrangements with communities, the voluntary and private sectors, and our partners in Confederation. Many of today's societal problems and issues are complex, require specialized knowledge, and are expensive to fully address. It is therefore advantageous to work with other actors and institutions to address public policy problems rather than simply taking a government-centred instrument choice perspective. Hence, it is important to bear in mind that instrument choice by government is a subset of a larger constellation of actors, institutions, instruments, and approaches that, taken together, constitute a more "distributive governance" method of addressing public policy issues. Harnessing the resources and expertise of these external actors, institutions, and processes requires a wider range of non-conventional approaches. Answering the following questions will assist officials in identifying the actors and institutions that should be involved in addressing a situation or problem.

Questions To Consider for Identifying Actors/Institutions

Questions To Consider for Identifying Actors/Institutions – Text version

Consultation Image of a Group of people

To reinforce both of the overarching rationales, depending on the case at hand, consultations with actors and institutions can be carried out at any time during the policy-making process. Because consultation itself is an iterative process within the analytical framework, it is not identified as a particular step in the process. The guidance regarding regulatory consultations with actors and institutions provided in Guidelines for Effective Regulatory Consultations, available at Regulatory Affairs http://www.tbs-sct.gc.ca/ri-qr, is also useful for consultations pertaining to problems or situations in which direct regulation may not be the optimal policy instrument. Consultations encourage knowledge sharing, promote transparency and trust in government action, and can enhance public policy performance results.

Consulting key actors and institutions regarding the choice of instruments can be as important as consulting them on policy issues. When using a mix of instruments, a number of actors and institutions will likely be involved in the implementation phase. This may mean that they will need to assume new roles and responsibilities to further public policy goals. Consideration of all needs, rights, roles, and responsibilities of all of the actors and institutions at the beginning of the policy/instrument selection process can improve the quality of public policy decisions.

The following questions should be considered for effective consultations throughout the process of assessing and selecting instruments.

Questions To Consider for Effective Consultations

Questions To Consider for Effective Consultations – Text version

Step 5: Considerations in assessing and selecting instruments

This step involves taking into account a number of considerations relevant to the assessment and selection of instruments. These considerations include effectiveness, legality, conformity and compliance, and accountability.

There are three dimensions to establishing the considerations for evaluating instruments. The first involves identifying the considerations that relate to managing the problems/risks found in Step I. These would include, for example, technical, legal, implementation (manageability), and conformity/compliance risks. The second dimension entails identifying and taking full advantage of opportunities for coordinating the types and/or use of instruments with other governments and agencies, both domestically and internationally. The third involves identifying considerations that reflect certain values such as economic efficiency, fairness, individual liberty, political acceptability, and accountability.

The following are considerations that are commonly used in assessing and selecting instruments.

Effectiveness: the expected capacity of the instrument or mix of instruments to achieve the intended policy objective

Will the instrument or mix of instruments do the job? Could it cause unintended or unwanted results? If it could have negative results, the policy objectives may have to be adjusted and other measures to achieve those goals considered.

Legality, individual liberty, and state coercion: the need to ensure that the government has met all legal requirements and has the authority to put the instrument into place

Legal scrutiny also includes ensuring that legal obligations have been met. These obligations are stipulated in international agreements to which the Government of Canada is a party. They include, for example, the Canadian Agreement on Internal Trade, the International Convention for the Safety of Life at Sea, the North American Free Trade Agreement, and the Agreement Establishing the World Trade Organization. A related consideration is the degree to which the use of the instrument(s) constrains the inherent legal rights of individuals. Many of the instruments available to government, particularly those that involve the use of rules, achieve their objectives by constraining the rights of individuals to engage in activities that would otherwise be permissible. A related aspect is the degree of state coercion (for example, enforcement activity) that would likely be required to implement the instrument and achieve the risk management objectives.

Conformity and compliance: the expected change in the behaviour of actors and institutions following implementation of the instrument(s)

Some types of problems, depending on their inherent risks, may require monitoring, sanctioning, and enforcement activities, while others may not. In cases of low risk or where the risks do not involve irreversible or critical harm to Canadians or the environment, it may be preferable to use instruments or mixes of instruments that encourage or promote, rather than require, a change in behaviour. Issues of consistency and coordination of policy intervention across jurisdictions and of interdependency between actors and institutions must also be considered.

The Netherlands' Table of Eleven

Maximizing conformity and compliance should be considered at the design phase when choosing a suitable mix of instruments. The Netherlands has created a checklist that helps track conformity and compliance issues throughout the policy development process. The checklist evaluates proposals against a table of 11 key determinants of compliance:

  1. Knowledge of rules: target group familiarity with policies, laws, and regulations
  2. Cost/benefit considerations: material and non-material advantages and disadvantages arising from violating or observing policies, laws, and regulations
  3. Level of acceptance: the extent to which the target group (generally) accepts the policy, laws, and regulations
  4. Normative commitment: innate target group willingness to comply or habit of complying with policy, laws, and regulations
  5. Informal control: the possibility that non-compliant behaviour by the target groups will be detected and disapproved of by third parties (i.e. non-government authorities) and possibility–and severity–of sanctions imposed by third parties (e.g. loss of customers/contractors, loss of reputation)

    Enforcement dimensions (the influence of enforcement on compliance)

  6. Informal report probability: the likelihood that an offence will come to light other than during an official investigation and may be officially reported (whistle blowing)
  7. Probability of inspection: risk of an administrative (paper) or substantive (physical) audit/inspection by official authorities
  8. Detection probability: the likelihood that an offence will be detected during an administrative audit or substantive investigation by official authorities (probability of non-compliant behaviour being uncovered in the event of some kind of scrutiny)
  9. Selectivity: the (increased) chance of inspection and detection as a result of risk analysis and targeting of firms, persons, or areas (i.e. extent to which inspectors succeed in checking offenders more often than those who abide by the law)

    Sanction dimensions (the influence of sanctions on compliance)

  10. Sanction probability: the likelihood of a sanction being imposed if an offence has been detected through inspection and criminal investigation
  11. Sanction severity: the severity and type of sanction and associated adverse effects (e.g. loss of respect and reputation)

Economic implications: the expected changes to the net social benefits and corresponding distributional impacts from implementing an instrument or a mix of instruments

Determination of Legitimacy

The question of legitimacy must be confronted when considering and assessing which instrument(s) will be used to achieve public policy objectives. The values held by Canadians should be reflected in the criteria for selecting instruments. In a study for Quebec's Société de l'assurance automobile, Pierre Issalys identified and defined five criteria specific to legitimacy:

  1. Political control–Parliament (to which government authority is subordinate), through its representatives, is the embodiment of the collective freedom of society.
  2. Public space–The ability of Canadians to hold government responsible for its actions, either directly or through its representatives, must be facilitated and maintained by the rule of law.
  3. Fairness–Government action must respect the rights of individuals and be as consistent and transparent as possible.
  4. Technical–Public policy should, wherever possible, be motivated by sound empirical foundations.
  5. Economy of means–Government resources should be efficiently allocated in order to meet public policy objectives.

Consult the Canadian Cost-Benefit Analysis Guide for further details. This consideration includes not only the costs and benefits to Canadians, government, and industry, but also issues such as who may win or who may lose, and how those disadvantaged by government action will fare. Increasing the flexibility of the instrument or mix of instruments and reducing the paperwork burden can help to minimize costs. Choosing the appropriate instrument or mix of instruments can help to maximize the potential benefits of an action, whether the benefits gained are tangible (e.g. product value added) or intangible (e.g. increased safety).

Fairness: the acceptability of potential economic and social distributional consequences arising from the use of an instrument or mix of instruments on different social groups, business sectors, and/or regions of the country

The internalization of externalities should also be considered as a distributional factor (i.e. the "polluter pays" principle is applied when considering policy intervention).

Political accountability, transparency, and legitimacy: the ability of government to demonstrate the effectiveness of an instrument or mix of instruments, and the support of government departments and Canadians for the use of these instruments and their policy objectives

Jurisdictional compatibility with other instruments already in use or proposed by other governance institutions, and not only government authorities, should also be taken into account. One of the key prerequisites for accountability is transparency in both the policy development process and the administration of government programs. Transparency remains an important consideration (perhaps even more so) if the instrument mix includes measures not wholly within government control, such as industry self-regulation or delegated authority.

Sample Instrument Assessment Matrix

A matrix similar to the one below can be used as an aid in assessing the suitability of various instruments.

 Criteria Import- ance
Rating
(H, M, L)
Regul- atory
Instru- ments
Economic
Instru- ments
Voluntary
Instru- ments
Inform- ation
Instru- ments
1 2 3 4 5 6 7 8 9 10 11 12

Effectiveness

  • magnitude of risk mitigation over time
  • speed with which risk mitigation objectives are achieved
  • sustainability of risk management performance
  • severity of side effects
                         

Legality and Legal Obligations

  • consistency with Constitution and statutes
  • international trade agreements
  • international social agreements
                         

Individual Liberty and State Coercion

  • severity of constraints on individual liberties
  • anticipated need for application of state coercion
  • comparative balance between constraints and enabling effects of instruments
                         

Conformity and Compliance

  • accessibility of rules/objectives for actors and institutions
  • informal report probability
  • inspection probability
  • detection probability (from Netherlands example)
                         

Economic Implications

  • regulatory burden
  • reduction of compliance costs
  • market structure
  • rate of cost abatement
  • price sensitivity
  • potential for value-added opportunities
                         

Fairness

  • distributional impacts across social groups, industry sectors, SMEs, regions
  • internalization of externalities
                         

Political Accountability and Legitimacy

  • expected buy-in from government
  • expected support from Canadians
  • transparency of process and activities
  • accessibility of information on activities and instrument performance
                         

Step 6: Set performance indicators

Performance indicators are essential for determining whether the instrument or mix of instruments is performing as intended and is achieving and sustaining the desired risk management outcomes. Such indicators should be tentatively identified before the details of the intervention strategy are established, as they can help shape the final product. A single or small set (i.e. "dashboard") of high-level indicators should be specified to track overall risk management performance in the policy area (e.g. the number of deaths per year due to playground-related injuries and the number of serious injuries per year). Performance measures in policy sub-areas should also be identified (e.g. manufacturer defect rate for playground equipment and level of on-site adult/parental supervision). These measures will likely correspond to causal risk factors influencing the achievement of the policy goal (as identified in Step I) or will relate to the criteria used for selecting the instruments (as described in Step V). To ensure that the performance measures are meaningful (e.g. understandable, relevant, and comparable), it is valuable to engage departmental planners or evaluators in the policy process to leverage in-house expertise.

If a dynamic risk analysis approach is being used, it is preferable to avoid using "snapshot" performance monitoring and reporting, which may provide only a fragmented or disaggregated and time-limited picture of performance. Instead, the focus should be on tracking emerging trends in both risk-related indicators and the determinants of risk. This approach should be a foundation principle in any performance management regime.

The goal of performance monitoring and reporting should be to assess the effectiveness of the instrument(s) against the performance expectations identified in Step II and determine the reasons for any divergence. If a divergence does occur, it may be due to a problem in program implementation or in the policy development process (e.g. overestimating the power of instruments to mitigate a risk; overestimating how quickly they would take effect; or not recognizing elements of social, economic, or environmental systems that would counteract the beneficial impacts of the instruments).

The logic model established in "identifying and defining the problem" (Step I) will help to provide a sound understanding of the causal determinants of the risk so that government may be able to rely on and respond to changes in indicators relating to these factors over time (e.g. the level of awareness, compliance, and safe behaviour of the target audience). Often, these dynamics will be detectable before changes in the ultimate outcomes (e.g. a fatality or injury) due to the instruments employed.

Logic model

Logic model – Text version

Step 7: Implement instrument or mix of instruments

Implementation involves applying the instrument or mix of instruments to manage the risks over time and achieve policy objectives. This usually requires an implementation project plan with the following components:

Where the chosen intervention strategy and use of selected instruments involve a modification of existing policy and program approaches, it may be useful to first carry out some form of "gap analysis" identifying the differences between the existing and proposed approaches. The differences will largely determine the scope and content of the implementation project plan.

Gap Analysis to Identify the Differences Between the Existing and Proposed Approaches

Gap Analysis to Identify the Differences Between the Existing and Proposed Approaches – Text version

Date modified: