ARCHIVED - Peer Review Process

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.

An Enhanced Framework for the Management of Information Technology Projects

PEER-PS-001 Peer Review Process

May 16, 1997
Chief Information Officer Branch
Treasury Board of Canada Secretariat

1. Introduction

The purpose of peer reviews is to remove defects from work products early and efficiently. Peer reviews also serve to provide a better understanding of work products and of the defects that can be prevented.

A peer review is a systematic examination of work products by the author's peers to identify defects and areas where changes are needed.

1.1 Goals

Goal 1 Peer review activities are planned.

Goal 2 Defects in work products are identified and removed.

1.2 Scope

The peer review process is applicable to any business unit requiring a process to review work products related to software or any other business.

1.3 Document Structure

This document is structured as follows;

Section 2 provides the reader with a general overview of the Peer Review Process.

Appendix A contains the bibliography.

Appendix B contains the procedure, checklists, and templates used in the Peer Review Process.

Review Procedure
Checklists
- Review Package Cover Sheet
- Facilitator Checklist
- Exit Criteria
Templates:
- Meeting Notice
- Preparation Report
- Defect List
- Defect Summary
- Review Certification Report

2. Overview

2.1 What to Review

Projects identify the work products to be reviewed. Specific reviews can be mandated in published methodologies or organizational policies.

Reviews should be considered, but not limited, to the following work products:

Software Development Process

requirements
project plans
logical design
physical design
code
test plans
user documentation

Software Process Improvement

policies
processes
procedures
standards
templates
checklists

2.2 Check Procedures

Entry and exit criteria are critical for conducting a review. Prior to beginning a review, senior project personnel must establish entry and exit criteria for each type of work product to be reviewed. Criteria may exist from formal methodologies and other groups within the organization. Generic criteria can be modified for specific products.

Entry Criteria

Entry criteria represents a list of required activities that must be performed and the supporting materials and all documents that must be available prior to the review.

Entry criteria for a policy should include the standard to create a policy, approved references on the subject covered in the policy, and departmental business plans.

Entry criteria for code should include the physical design specification from which the code was generated, a clean compile listing, test environment materials, automated code audit reports, and unit test scripts and cases.

The available entry criteria and Review Package Cover Sheet (see appendixes) is used by the facilitator to determine if the work product is ready for review.

Exit Criteria

Exit criteria is a list of the requirements that must be met to pass the review.

Exit criteria for a policy will include such items as: do the policy meet established standards for creating a policy, does the policy endorse the departmental and branch vision, does the policy endorse the approved departmental frameworks and methodologies, etc. (reference appendixes).

Exit criteria for code will include such items as: does the code meet established quality standards, does the code perform the functions required in the physical design, etc.

2.3 Defect Classification

Classifying defects provides meaningful data for analysing defects found in reviews and the opportunity for identifying and removing the root cause of the defects. This type of process improvement results in overall cost savings and improved product quality.

Defects should be classified by origin, type, class, and severity.

Origin

Where the defect was created. This could be a Software Development Life Cycle (SDLC) phase or some other phase of a project.

Software Development Life Cycle (SDLC)

requirements
functional
detailed
code

Other

external documentation
bad fix

Type

Type indicates defect types listed in the exit criteria.

Class

Class indicates whether the defect was missing, wrong, or extra.

missing requirement
wrong requirement not met
extra requirement not requested, but was present in the product
unclear, ambiguous
unknown cannot be categorized

Severity

Severity has 2 levels

major, those that cause incorrect results
minor, all those that are not major

Approved defect classification do not currently exist.

Process Improvement (PI) teams, methodologists, Centres Of Expertise (COE), and project teams can work with the organization's measurement groups to define defect classifications for the organization.

2.4 Roles

Facilitator

Trained to organize, lead, and control the review process and to oversee any necessary follow-up.

(1) Should NOT be a member of the project team.
(2) Organizes the review: in consultation with the author and/or project leader selects the participants, verifies distribution of the review materials, schedules the overview, review, and required follow-up sessions.
(3) Leads the review: ensures all participants are prepared, encourages participation, focuses on finding defects, controls the flow and direction, and maintains objectivity.
(4) Control the review: enforces adherence to the entry and exit criteria, seeks consensus on defects, makes final decision on disagreements, directs the recording and categorizing of defects, summarizes review results and keeps to a time limit of one to two hours per review.
(5) Ensures the author completes the follow-up tasks.

Reader

Responsible to set the pace of the review by paraphrasing or reading the work product being reviewed.

(1) Must not be the facilitator or author.
(2) Thoroughly familiar with the material to be reviewed.
(3) Objectively presents the work product.
(4) Paraphrases or reads the material line by line or paragraph by paragraph, pacing the reading for effective clarity and comprehension.

Recorder

Responsible to record defects and summarize the review results. Must have ample time to note defects since this is the only information the author will have to find and correct defects. Avoid using abbreviations or shorthand that may not be understood by other team members.

(1) Can be the same individual as the facilitator, but cannot be the reader or author.
(2) Records every defect found.
(3) Presents the defect list for consensus by all participants in the review.
(4) Classifies the defects, as directed by the reviewers, by type, class and severity based on predetermined criteria.

Author

Originator of the work product being reviewed.

(1) May act as reviewer during the review meeting.
(2) Determines when the work product is ready for review.
(3) Assist the facilitator in selecting the participants for the review team.
(4) Meets all entry criteria outlined in the appropriate review package cover sheet.
(5) Provides an overview of the material prior to the review for clarification, if requested.
(6) Provides clarification of the review material during the review, if requested.
(7) Corrects the defects and presents the finished rework to the facilitator for sign-off.

Reviewers

Trained individuals who can effectively contribute to the objectives of the review meeting.

(1) Facilitator, reader, and recorder can also be reviewers.
(2) Must be prepared for the review by carefully examining and understanding the material prior to the review.
(3) Maintains objectivity towards the work product.
(4) Reviews the work product.
(5) Records all preparation time.
(6) Presents potential defects and problems encountered before and during the review meeting.

2.5 Review Process

The review process contains 5 steps, each distinctive and essential to the successful outcome of the overall process.

(1) Planning
(2) Overview Meeting (optional)
(3) Preparation
(4) Review Meeting
(5) Rework/Follow-up

Planning

To initiate, organize, and schedule the review, define the participant roles, and define how defects will be classified.

Overview Meeting (Optional)

To acquaint all reviewers with the work product to be reviewed and to minimize individual preparation time. An overview is optional and is performed if the:

work product is lengthy, complex or new,
review process is new,
participants are new to the review process.

Preparation

To provide time for each review participant to acquire a thorough understanding of the product to be reviewed and identify any defects found (per exit criteria).

Review Meeting

To find defects in the work product, but not correct those defects or suggest alternatives for correction.

Rework/Follow-up

To complete rework required, obtain a sign-off or initiate a re-review and capture review results.

2.6 Review Results

The Facilitator, at the end of review meeting, determines the state of the review and what follow-up is required.

The Facilitator selects one of the following states:

No defects found – product has passed the review
Minor work required – minor defects found and a re-review is not required. The author will correct all defects and the facilitator will verify the results.
Major work required – major defects were found and a re-review is required.

Appendix A – Bibliography

Bibliography

Quality Practices Manual. Quality Assurance Institute, Orlando, Florida, U.S.A. January 1995.

Capability Maturity Model for Software, Version 1.1. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, U.S.A. February 1993.

Appendix B – Procedure, Checklists and Templates

Review Procedure

A step-by-step method to conduct a peer review. Used by the Participants throughout the review cycle.

STEP 1: Planning

To initiate, organize, and schedule the review, define the participant roles, and define how defects will be classified.

Participant		Task
Author	(1)	Initiates the review process. Contacts the facilitator, appointed by project management, to indicate the work product is ready for review. Provides basic information on project and work product to be reviewed.
	(2)	Verifies a Review Package Cover Sheet exists for the work product. If it does not exist: With the project leader and facilitator, creates a Review Package Cover Sheet for the work product. Forwards a copy of the materials listed in the Review Package Cover Sheet to the facilitator.
Facilitator	(3)	Begins activities listed in Facilitator Checklist.
	(4)	Determines if the product is ready for review based on the Review Package Cover Sheet.
	(5)	Selects reviewers in consultation with the author and/or the project leader and assigns the roles of reader and recorder.
	(6)	Estimates review preparation time. documents: 10-15 pages per hour code: 100-200 statements per hour
	(7)	Schedules the review meeting and sends Review Meeting Notice to participants.
	(8)	Determines with author and project leader if overview is required (e.g. if the product is lengthy or complex)
	(9)	Co-ordinates the distribution of the review material including the Review Meeting Notice.

STEP 2: Overview Meeting (Optional but suggested)

To acquaint all acquaint all reviewers with the work product to be reviewed and to minimize individual preparation time. An overview is optional and is performed if the work product is lengthy, complex or new, if the review process is new, or if the participants are new to the review process

Participant		Task
Facilitator	(1)	Requests the author to present an overview of the work product.
Author	(2)	Organizes, schedules and presents the overview.
Reviewers	(3)	Attends the overview to gain an understanding of the product, not to find defects.

STEP 3: Preparation

To provide time for each review participant to acquire a thorough understanding of the product to be reviewed and identify any defects found (as per exit criteria)

Participant		Task
Reviewers	(1)	Familiarize themselves with the review material.
	(2)	Using the Exit Criteria Checklist, record all defects found and time spent on the Review Preparation Report and Review Defect List.

STEP 4: Review Meeting

To find defects in the work product, but not correct those defects or suggest alternatives for correction.

Participant		Task
Facilitator	(1)	Introduces participants and identifies roles.
	(2)	Re-states objective of the review.
	(3)	Verifies reviewer's readiness by asking time spent in preparation and whether or not all material was reviewed prior to the meeting (as indicated on each reviewer's Review Preparation Report). If any of the participants are not prepared, the facilitator must decide whether to continue with the review, or reschedule it for further preparation.
Reader	(4)	Paraphrases or reads the material.
Reviewers	(5)	State potential defects found, discuss the defect stated and reach a consensus on whether or not the defect exists.
Recorder	(6)	Records defects found by origin, type, class, and severity on the Review Defect List.
	(7)	Classifies each defect found with concurrence from all reviewers.
	(8)	Prepares the Review Defect Summary.
Author	(9)	Provides clarification of the work product, when requested.
Facilitator	(10)	Calls the review to an end if a high number of defects are found early in the review meeting indicating the work product was not ready for review. (The author is responsible for re-initiating a review, through the facilitator, once the work product is ready)
	(11)	Determines the disposition of the review and any follow-up required.
	(12)	Approves the Review Defect List and the Review Defect Summary and forwards a copy to the author and Project Software Quality Assurance.
	(13)	Sign-off on the Defect Certification Report if no defects were found.

STEP 5: Rework/Follow-up

To complete rework required, obtain a sign-off or initiate a re-review and capture review results.

Participant		Task
Author	(1)	Completes all rework to correct defects found as a result of the review.
	(2)	Re-initiates the review process if the inspection ended with a disposition of major rework required.
	(3)	Contacts the facilitator to approve the rework, if the review ended with a disposition of minor rework required.
Facilitator	(4)	Reviews all rework completed and signs-off on Review Certification Report when all defects have been corrected.
Recorder	(5)	Summarizes defect data and ensures entry into an Inspection Defect Database.

Review Package Cover Sheet

Used by the Facilitator to determine if the work product is ready for review.

Unique for each type of artifact.

_____	Review Meeting Notice.
_____	Copy of work product.
_____	Exit Criteria Checklist Form.
_____	If re-review, defect List from prior review.
_____	Applicable Work Product Standards.
_____	Standards.
_____	Departmental Planning Documents.
_____	Others.

Facilitator Checklist

Used by the Facilitator throughout the review cycle.

_____	Check that entry criteria (Review Package Cover Sheet) have been met.
_____	Meet with author and team leader to select qualified review participants and assign roles.
_____	Determine need for an overview session.
_____	Schedule Review Meeting.
_____	Complete Review Meeting Notice.
_____	Gather materials from author and distribute to review participants.
_____	Talk with reviewers to assure preparation time.
_____	Complete self-preparation of material for review.
_____	Conduct Review Meeting.
_____	Ensure completion and distribution of Review Defect List and Review Defect Summary.
_____	Verify conditional completion (Facilitator review or re-inspection).
_____	Complete Review Certification Report.

Exit Criteria

Used by review Participants to determine if the work product passes the review.

Defect Type	Criteria
DC		DOCUMENTATION
	______	Documentation complete
EN		ENGLISH READABILITY
	______	Format and content correct
MN		MAINTAINABILITY
	______	Artifact maintainable and extendible
ST		STANDARDS
	______	Adherence to Quality/Project Standards and Practices

Meeting Notice

Distributed to Reviewers by the Facilitator.

Project Name:			Current Date:
Name of item being reviewed:
Item version identification:
Material Size (Lines/Pages):			Expected preparation time:
Facilitator:
Review Type:	Review
	Re-review
THIS REVIEW HAS BEEN SCHEDULED FOR:
DATE:
TIME:
LOCATION:
DURATION:
THE FOLLOWING INDIVIDUALS ARE SCHEDULED TO PARTICIPATE
NAME		PHONE		ROLE





COMMENTS:

Preparation Report

Used by Reviewers during preparation to record defects.

Project Name:			Current Date:
Name of item being reviewed:
Item version identification:
Material Size (Lines/Pages)
Expect Preparation Time:
Preparation Log:
DATE		TIME SPENT



DEFECT LIST:
LOCATION	DEFECT DESCRIPTION			EXIT CRITERIA VIOLATED

Note: For additional defects use reverse side of form.

Defect List

Used by Recorder during the review to record defects.

Project Name:					Current Date:
Name of item being reviewed:
Item version identification:
Facilitator:
Phone:
Review Type:			Review			Release #
			Re-review			Product Type:

DEFECT TYPES: These are examples of defect types. Standard defect types do not currently exist. Until standard defect types are defined, senior project personnel should establish defect types for each type of work product to be inspected.
CM	Comments	LO		Logic			PF	Performance
DA	Data	LR		Linkage Req.			RQ	Requirements
DC	Documentation	MN		Maintainability			SC	Specs. Clarification
EN	English Readability	MS		Messages/Return			ST	Standards
IF	Interface	OT		Other			TP	Test Plan
LD	Logical Design	PD		Physical Design

DEFECT CLASS:

E Extra
M Missing
W Wrong
A Ambiguous
U Unknown

LOCATION	DEFECT DESCRIPTION	ORIGIN	TYPE	CLASS	SEVERITY

Defect Summary

Prepared by Recorder at the end of the review meeting.

Project Name:		Current Date:
Name of item being reviewed:
Item version identification:
Facilitator:
Phone:
Review Type:	Review		Re-review:

	MINOR DEFECT CLASS
DEFECT TYPES		M	W	E	A	U	Total

CM Comments
DA Data
DC Documentation
EN English Readability
IF Interfaces
LD Logical Design
LO Logic
LR Linkage Reqs.
MN Maintainability
MS Message/Return
OT Other
PD Physical Design
PF Performance
RQ Requirements
SC Spec Clarification
St Standards
TP Test Plan
TOTALS

M Missing
W Wrong
E Extra
A Ambiguous
U Unclear

Project Name:		Current Date:
Name of item being reviewed:
Item version identification:
Facilitator:
Phone:
Review Type:	Review		Re-review:

	MAJOR DEFECT CLASS
DEFECT TYPES		M	W	E	A	U	Total

CM Comments
DA Data
DC Documentation
EN English Readability
IF Interfaces
LD Logical Design
LO Logic
LR Linkage Reqs.
MN Maintainability
MS Message/Return
OT Other
PD Physical Design
PF Performance
RQ Requirements
SC Spec Clarification
St Standards
TP Test Plan
TOTALS

M Missing
W Wrong
E Extra
A Ambiguous
U Unclear

Review Certification Report

Prepared by Facilitator when work product has passed review. Copy forwarded to project Software Quality Assurance (SQA) group.

Project Name:	Review Date:
Name of item being reviewed:
Item version identification:
The following people have reviewed the named Item and agreed that all technical, contractual, quality, etc. requirements and review criteria have been satisfied.
Facilitator:
Recorder:
Reader:
Author:
Software Quality Representative:
Inspectors




Facilitator signature/date