Start with the MRRS PMF: Review the MRRS PMF that the department developed in accordance with the Policy on MRRS. Include, as appropriate, the performance indicators from the MRRS PMF in the PM Strategy Framework.

The PM Strategy Framework should not be developed in isolation. In accordance with the Policy on MRRS, all programs represented in a department's PAA must contribute to its strategic outcome(s). As such, when developing the PM Strategy Framework, the performance indicators should complement those already established in the departmental PMF.

Identify performance indicators: Develop or select at least one performance indicator for each output and each outcome (immediate, intermediate and ultimate) that has been identified in the program logic model.

Keep in mind that, in addition to day-to-day program monitoring, the performance indicators will also be used for evaluation purposes. As such, it is recommended that program managers also consider the core issues See footnote ¹² for evaluation (i.e. relevance and performance) and consult with heads of evaluation when developing the performance indicators.

There are two types of indicators: quantitative and qualitative.

• Quantitative performance indicators are composed of a number and a unit. The number indicates the magnitude (how much) and the unit gives the number its meaning (what), e.g. the number of written complaints received.
• Qualitative indicators are expressed in expository form, e.g. assessment of research quality. As much as possible, qualitative indicators should be condensed into a rating scale, e.g. research quality is rated as "excellent," "average" or "below average." This will allow for comparability over time.
• Choosing appropriate performance indicators is essential for effectively evaluating and monitoring a program's progress. Appropriate indicators are characterized as follows:
• Valid—The indicators measure what they intend to measure.
• Reliable—The data collected should be the same if collected repeatedly under the same conditions at the same point in time.
• Affordable—Cost-effective data collection (and analysis) methods can be developed.
• Available—Data for the indicators are readily and consistently available to track changes in the indicator.
• Relevant—The indicator clearly links back to the program outcomes.

Keep the number of performance indicators to a manageable size. A small set of good performance indicators are more likely to be implemented than a long list of indicators.

Identify data sources: Identify the data sources and the system that will be used for data management.

There are a number of possible data sources:

• Administrative data—Information that is already being collected through program files or databases or could be collected with adjustments to regular programming processes or funding agreements;
• Primary performance data—Specialized data collection exercises, such as focus groups, expert panels or surveys, in which information needs to be collected; and
• Secondary data—Information that has been collected for other purposes, such as national statistics on health or economic status (e.g. Statistics Canada data).

Use readily available information. Take advantage of any available sources of information at your disposal.

Define frequency and responsibility for data collection: Identify frequency for data collection and the person(s) or office responsible for the activity.

Describe how often performance information will be gathered. Depending on the performance indicator, it may make sense to collect data on an ongoing, monthly, quarterly, annual or other basis.

When planning the frequency and scheduling of data collection, an important factor to consider is management's need for timely information for decision making.

In assigning responsibility, it is important to take into account not only which parties have the easiest access to the data or sources of data but also the capacities and system that will need to be put in place to facilitate data collection.

Establish baselines, targets and timelines.

Performance data can only be used for monitoring and evaluation if there is something to which it can be compared. A baseline serves as the starting point for comparison. Performance targets are then required to assess the relative contribution of the program and ensure that appropriate information is being collected.

Baseline data for each indicator should be provided by the program when the PM Strategy Framework is developed. Often, and particularly for indicators related to higher-level outcomes, this information will have already been collected by program managers as part of their initial needs assessment and to support the development of their business case.

Targets are either qualitative or quantitative, depending on the indicator. Some sources and reference points for identifying targets include:

• baseline data based on past performance or previous iterations of the program;
• the achievements of other, similar programs that are considered leaders in the field (benchmarking);
• generally accepted industry or business standards; and
• publicly stated targets (e.g. set by the government, the federal budget).

If a target cannot be established or if the program is unable to establish baseline data at the outset of the program, explicit timelines for when these will be established as well as who is responsible for establishing them should be stated.

Consult and verify: As a final step, program managers should consult  See footnote ¹³ with heads of evaluation and other specialists in performance measurement to validate the performance indicators and confirm that the resources required to collect data are budgeted for. Consultation with other relevant program stakeholders (e.g. information management personnel) is also encouraged.

• This step complies with the Directive on the Evaluation Function, which holds heads of evaluation responsible for reviewing and providing advice on the PM Strategies for all new and ongoing direct program spending, including all ongoing programs of grants and contributions, to ensure that they effectively support an evaluation of relevance and performance (Section 6.1.4, subsection a).
• It is highly recommended that program managers work with the head of evaluation to ensure that the performance indicators selected will provide adequate data to support an evaluation of the program. The choice of consultation mechanism remains at the discretion of the program manager and head of evaluation.

Key questions to be considered during the consultation and verification stage include:

• Does the selection of indicators align with and complement the indicators in the departmental PMF?
• Are the indicators valid measures of the outputs and outcomes (i.e. do they measure what they intend to measure)?
• Are the indicators reliable (i.e. would the data recorded be the same if collected repeatedly under the same conditions at the same point in time)?
• Is it likely that the required data will be provided in a timely manner?
• Is the cost reasonable and clearly budgeted for?

If the answer to any of the above questions is "no," adjustments need to be made to the indicators or to the resources allocated for implementation of the PM Strategy.