Public Works and Government Services Canada	Travaux publics et Services gouvernementaux Canada	

Application Management Services Sector
Application Environment Services
AMS Software Process
Improvement Program
PEER REVIEW PROCESS

Document ID: PRREVIEW.DOC V0.1
Status: Draft
16 May 1997

Prepared by:

Date signed:	Software Engineering Process Group (SEPG) 		Accepted by:

Date signed:	Management Steering Group 
(MSG)

Revision Record Sheet
Revision Number	Date of Issue	Author(s)	Brief Description of Change	
0.1	16 May 1997	PWGSC	First Draft	

Table of Contents
1.	Introduction	1
1.1	Goals	1
1.2	Scope	1
1.3	Document Structure	1
2.	Overview	2
2.1	What to Review	2
2.2	Check Procedures	2
2.3	Defect Classification	3
2.4	Roles	4
2.5	Review Process	6
2.6	Review Results	7
Appendix A – Bibliography	8
Bibliography	9
Appendix B – Procedure, Checklists and Templates	10
Review Procedure	11
Review Package Cover Sheet	14
Facilitator Checklist	15
Exit Criteria	16
Meeting Notice	17
Preparation Report	18
Defect List	19
Defect Summary	21
Review Certification Report	22

1.	Introduction
The purpose of peer reviews is to remove defects from work products early and efficiently. Peer reviews also serve to provide a better understanding of work products and of the defects that can be prevented.
A peer review is a systematic examination of work products by the author’s peers to identify defects and areas where changes are needed.
1.1	Goals
Goal 1	Peer review activities are planned.
Goal 2	Defects in work products are identified and removed.
1.2	Scope
The peer review process is applicable to any business unit requiring a process to review work products related to software or any other business.
1.3	Document Structure
This document is structured as follows;
Section 2 provides the reader with a general overview of the Peer Review Process.
Appendix A contains the bibliography.
Appendix B contains the procedure, checklists, and templates used in the Peer Review Process.
•	Review Procedure
•	Checklists:
−	Review Package Cover Sheet
−	Facilitator Checklist
−	Exit Criteria
•	Templates:
−	Meeting Notice
−	Preparation Report
−	Defect List
−	Defect Summary
−	Review Certification Report

2.	Overview
2.1	What to Review
Projects identify the work products to be reviewed. Specific reviews can be mandated in published methodologies or organizational policies.
Reviews should be considered, but not limited, to the following work products:
Software Development Process
•	requirements
•	project plans
•	logical design
•	physical design
•	code
•	test plans
•	user documentation
Software Process Improvement
•	policies
•	processes
•	procedures
•	standards
•	templates
•	checklists
2.2	Check Procedures
Entry and exit criteria are critical for conducting a review. Prior to beginning a review, senior project personnel must establish entry and exit criteria for each type of work product to be reviewed. Criteria may exist from formal methodologies and other groups within the organization. Generic criteria can be modified for specific products.
Entry Criteria
Entry criteria represents a list of required activities that must be performed and the supporting materials and all documents that must be available prior to the review.
Entry criteria for a policy should include the standard to create a policy, approved references on the subject covered in the policy, and departmental business plans.
Entry criteria for code should include the physical design specification from which the code was generated, a clean compile listing, test environment materials, automated code audit reports, and unit test scripts and cases.
The available entry criteria and Review Package Cover Sheet (see appendixes) is used by the facilitator to determine if the work product is ready for review.
Exit Criteria
Exit criteria is a list of the requirements that must be met to pass the review.
Exit criteria for a policy will include such items as: do the policy meet established standards for creating a policy, does the policy endorse the departmental and branch vision, does the policy endorse the approved departmental frameworks and methodologies, etc. (reference appendixes).
Exit criteria for code will include such items as: does the code meet established quality standards, does the code perform the functions required in the physical design, etc.
2.3	Defect Classification
Classifying defects provides meaningful data for analysing defects found in reviews and the opportunity for identifying and removing the root cause of the defects. This type of process improvement results in overall cost savings and improved product quality.
Defects should be classified by origin, type, class, and severity.
Origin
Where the defect was created. This could be a Software Development Life Cycle (SDLC) phase or some other phase of a project.
Software Development Life Cycle (SDLC)
•	requirements
•	functional
•	detailed
•	code
Other
•	external documentation
•	bad fix
Type
Type indicates defect types listed in the exit criteria.
Class
Class indicates whether the defect was missing, wrong, or extra.
•	missing requirement
•	wrong requirement not met
•	extra requirement not requested, but was present in the product
•	unclear, ambiguous
•	unknown cannot be categorized
Severity
Severity has 2 levels
•	major, those that cause incorrect results
•	minor, all those that are not major
Approved defect classification do not currently exist.
Process Improvement (PI) teams, methodologists, Centres Of Expertise (COE), and project teams can work with the organization’s measurement groups to define defect classifications for the organization.
2.4	Roles
Facilitator
Trained to organize, lead, and control the review process and to oversee any necessary follow‑up.
(1)	Should NOT be a member of the project team.
(2)	Organizes the review: in consultation with the author and/or project leader selects the participants, verifies distribution of the review materials, schedules the overview, review, and required follow‑up sessions.
(3)	Leads the review: ensures all participants are prepared, encourages participation, focuses on finding defects, controls the flow and direction, and maintains objectivity.
(4)	Control the review: enforces adherence to the entry and exit criteria, seeks consensus on defects, makes final decision on disagreements, directs the recording and categorizing of defects, summarizes review results and keeps to a time limit of one to two hours per review.
(5)	Ensures the author completes the follow‑up tasks.
Reader
Responsible to set the pace of the review by paraphrasing or reading the work product being reviewed.
(1)	Must not be the facilitator or author.
(2)	Thoroughly familiar with the material to be reviewed.
(3)	Objectively presents the work product.
(4)	Paraphrases or reads the material line by line or paragraph by paragraph, pacing the reading for effective clarity and comprehension.
Recorder
Responsible to record defects and summarize the review results. Must have ample time to note defects since this is the only information the author will have to find and correct defects. Avoid using abbreviations or shorthand that may not be understood by other team members.
(1)	Can be the same individual as the facilitator, but cannot be the reader or author.
(2)	Records every defect found.
(3)	Presents the defect list for consensus by all participants in the review.
(4)	Classifies the defects, as directed by the reviewers, by type, class and severity based on predetermined criteria.
Author
Originator of the work product being reviewed.
(1)	May act as reviewer during the review meeting.
(2)	Determines when the work product is ready for review.
(3)	Assist the facilitator in selecting the participants for the review team.
(4)	Meets all entry criteria outlined in the appropriate review package cover sheet.
(5)	Provides an overview of the material prior to the review for clarification, if requested.
(6)	Provides clarification of the review material during the review, if requested.
(7)	Corrects the defects and presents the finished rework to the facilitator for sign‑off.
Reviewers
Trained individuals who can effectively contribute to the objectives of the review meeting.
(1)	Facilitator, reader, and recorder can also be reviewers.
(2)	Must be prepared for the review by carefully examining and understanding the material prior to the review.
(3)	Maintains objectivity towards the work product.
(4)	Reviews the work product.
(5)	Records all preparation time.
(6)	Presents potential defects and problems encountered before and during the review meeting.
2.5	Review Process
The review process contains 5 steps, each distinctive and essential to the successful outcome of the overall process.
(1)	Planning
(2)	Overview Meeting (optional)
(3)	Preparation
(4)	Review Meeting
(5)	Rework/Follow‑up
Planning
To initiate, organize, and schedule the review, define the participant roles, and define how defects will be classified.
Overview Meeting (Optional)
To acquaint all reviewers with the work product to be reviewed and to minimize individual preparation time. An overview is optional and is performed if the:
•	work product is lengthy, complex or new,
•	review process is new,
•	participants are new to the review process.
Preparation
To provide time for each review participant to acquire a thorough understanding of the product to be reviewed and identify any defects found (per exit criteria).
Review Meeting
To find defects in the work product, but not correct those defects or suggest alternatives for correction.
Rework/Follow‑up
To complete rework required, obtain a sign‑off or initiate a re‑review and capture review results.
2.6	Review Results
The Facilitator, at the end of review meeting, determines the state of the review and what follow‑up is required.
The Facilitator selects one of the following states:
•	No defects found – product has passed the review
•	Minor work required – minor defects found and a re‑review is not required. The author will correct all defects and the facilitator will verify the results.
•	Major work required – major defects were found and a re‑review is required.

Appendix A – Bibliography

Bibliography
Quality Practices Manual. Quality Assurance Institute, Orlando, Florida, U.S.A. January 1995.
Capability Maturity Model for Software, Version 1.1. Software Engineering Institute, Carnegie Mellon University, Pittsburgh, Pennsylvania, U.S.A. February 1993.

Appendix B – Procedure, Checklists and Templates

Review Procedure
A step‑by‑step method to conduct a peer review. Used by the Participants throughout the review cycle.
STEP 1: Planning
To initiate, organize, and schedule the review, define the participant roles, and define how defects will be classified.
Participant		Task	
Author	(1)	Initiates the review process.
Contacts the facilitator, appointed by project management, to indicate the work product is ready for review. Provides basic information on project and work product to be reviewed. 	
	(2)	Verifies a Review Package Cover Sheet exists for the work product. 
If it does not exist: With the project leader and facilitator, creates a Review Package Cover Sheet for the work product .
Forwards a copy of the materials listed in the Review Package Cover Sheet to the facilitator. 	
Facilitator	(3)	Begins activities listed in Facilitator Checklist. 	
	(4)	Determines if the product is ready for review based on the Review Package Cover Sheet. 	
	(5)	Selects reviewers in consultation with the author and/or the project leader and assigns the roles of reader and recorder.	
	(6)	Estimates review preparation time.
•	documents: 10‑15 pages per hour
•	code: 100‑200 statements per hour	
	(7)	Schedules the review meeting and sends Review Meeting Notice to participants. 	
	(8)	Determines with author and project leader if overview is required (e.g. if the product is lengthy or complex) 	
	(9)	Co‑ordinates the distribution of the review material including the Review Meeting Notice. 	

STEP 2: Overview Meeting (Optional but suggested)
To acquaint all acquaint all reviewers with the work product to be reviewed and to minimize individual preparation time. An overview is optional and is performed if the work product is lengthy, complex or new, if the review process is new, or if the participants are new to the review process
Participant		Task	
Facilitator	(1)	Requests the author to present an overview of the work product.	
Author	(2)	Organizes, schedules and presents the overview.	
Reviewers	(3)	Attends the overview to gain an understanding of the product, not to find defects.	

STEP 3: Preparation
To provide time for each review participant to acquire a thorough understanding of the product to be reviewed and identify any defects found (as per exit criteria)
Participant		Task	
Reviewers	(1)	Familiarize themselves with the review material. 	
	(2)	Using the Exit Criteria Checklist, record all defects found and time spent on the Review Preparation Report and Review Defect List. 	

STEP 4: Review Meeting
To find defects in the work product, but not correct those defects or suggest alternatives for correction.
Participant		Task	
Facilitator	(1)	Introduces participants and identifies roles.	
	(2)	Re‑states objective of the review. 	
	(3)	Verifies reviewer’s readiness by asking time spent in preparation and whether or not all material was reviewed prior to the meeting (as indicated on each reviewer’s Review Preparation Report). If any of the participants are not prepared, the facilitator must decide whether to continue with the review, or reschedule it for further preparation. 	
Reader	(4)	Paraphrases or reads the material.	
Reviewers	(5)	State potential defects found, discuss the defect stated and reach a consensus on whether or not the defect exists. 	
Recorder	(6)	Records defects found by origin, type, class, and severity on the Review Defect List.	
	(7)	Classifies each defect found with concurrence from all reviewers.	
	(8)	Prepares the Review Defect Summary.	
Author	(9)	Provides clarification of the work product, when requested. 	
Facilitator	(10)	Calls the review to an end if a high number of defects are found early in the review meeting indicating the work product was not ready for review. (The author is responsible for re‑initiating a review, through the facilitator, once the work product is ready)	
	(11)	Determines the disposition of the review and any follow‑up required.	
	(12)	Approves the Review Defect List and the Review Defect Summary and forwards a copy to the author and Project Software Quality Assurance.	
	(13)	Sign‑off on the Defect Certification Report if no defects were found.	

STEP 5: Rework/Follow‑up
To complete rework required, obtain a sign‑off or initiate a re‑review and capture review results.
Participant		Task	
Author	(1)	Completes all rework to correct defects found as a result of the review.	
	(2)	Re‑initiates the review process if the inspection ended with a disposition of major rework required.	
	(3)	Contacts the facilitator to approve the rework, if the review ended with a disposition of minor rework required.	
Facilitator	(4)	Reviews all rework completed and signs‑off on Review Certification Report when all defects have been corrected. 	
Recorder	(5)	Summarizes defect data and ensures entry into an Inspection Defect Database.	

Review Package Cover Sheet
Used by the Facilitator to determine if the work product is ready for review.
Unique for each type of artifact.

_____	Review Meeting Notice.

_____	Copy of work product.

_____	Exit Criteria Checklist Form.

_____	If re‑review, defect List from prior review.

_____	Applicable Work Product Standards.

_____	Standards.

_____	Departmental Planning Documents.

_____	Others.

Facilitator Checklist
Used by the Facilitator throughout the review cycle.

_____	Check that entry criteria (Review Package Cover Sheet) have been met.

_____	Meet with author and team leader to select qualified review participants and assign roles.

_____	Determine need for an overview session.

_____	Schedule Review Meeting.

_____	Complete Review Meeting Notice.

_____	Gather materials from author and distribute to review participants.

_____	Talk with reviewers to assure preparation time.

_____	Complete self‑preparation of material for review.

_____	Conduct Review Meeting.

_____	Ensure completion and distribution of Review Defect List and Review Defect Summary.

_____	Verify conditional completion (Facilitator review or re‑inspection).

_____	Complete Review Certification Report.

Exit Criteria
Used by review Participants to determine if the work product passes the review.
Defect Type	Criteria	
DC		DOCUMENTATION	
	______	Documentation complete	
EN		ENGLISH READABILITY 	
	______	Format and content correct	
MN		MAINTAINABILITY	
	______	Artifact maintainable and extendible	
ST		STANDARDS	
	______	Adherence to Quality/Project Standards and Practices	

Meeting Notice
Distributed to Reviewers by the Facilitator.
Project Name:		Current Date:		
Name of item being reviewed:		
Item version identification:		
Material Size (Lines/Pages):		Expected preparation time:		
Facilitator:		
Review Type:		Review		
		Re‑review		
		
THIS REVIEW HAS BEEN SCHEDULED FOR:	
	DATE:		
	TIME:		
	LOCATION:		
	DURATION:		
	
THE FOLLOWING INDIVIDUALS ARE SCHEDULED TO PARTICIPATE	
NAME		PHONE		ROLE	

					

COMMENTS:		

	

Preparation Report
Used by Reviewers during preparation to record defects.
Project Name:		Current Date:		
Name of item being reviewed:		
Item version identification:		
Material Size (Lines/Pages)			
Expect Preparation Time:			
Preparation Log:		
	DATE		TIME SPENT		

					
DEFECT LIST:	
LOCATION		DEFECT DESCRIPTION		EXIT CRITERIA VIOLATED	

					

					

					

Note: For additional defects use reverse side of form.

Defect List
Used by Recorder during the review to record defects.
Project Name:		Current Date:		
Name of item being reviewed:		
Item version identification:		
Facilitator:		
Phone:		
Review Type:		Review		Release #		
		Re‑review		Product Type:		
	
DEFECT TYPES: These are examples of defect types. Standard defect types do not currently exist. Until standard defect types are defined, senior project personnel should establish defect types for each type of work product to be inspected. 	
						
CM	Comments	LO	Logic	PF	Performance	
DA	Data	LR	Linkage Req.	RQ	Requirements	
DC	Documentation	MN	Maintainability	SC	Specs. Clarification	
EN	English Readability	MS	Messages/Return	ST	Standards	
IF	Interface	OT	Other	TP	Test Plan	
LD	Logical Design	PD	Physical Design			

DEFECT CLASS:	E Extra
M Missing
W Wrong
A Ambiguous
U Unknown

LOCATION		DEFECT DESCRIPTION		ORIGIN		TYPE		CLASS		SEVERITY	

											

											

											

											

											

											

											

											

											

Defect Summary
Prepared by Recorder at the end of the review meeting.
Project Name:		Current Date:		
Name of item being reviewed:		
Item version identification:		
Facilitator:		
Phone:		
Review Type:	Review		Re‑review:		
	
	MINOR DEFECT CLASS		MAJOR DEFECT CLASS	
DEFECT TYPES	M	W	E	A	U	Total		M	W	E	A	U	Total	
														
CM Comments														
DA Data														
DC Documentation														
EN English Readability														
IF Interfaces														
LD Logical Design														
LO Logic														
LR Linkage Reqs. 														
MN Maintainability														
MS Message/Return														
OT Other														
PD Physical Design														
PF Performance														
RQ Requirements														
SC Spec Clarification														
St Standards														
TP Test Plan														
														
TOTALS														
M Missing	W Wrong	E Extra	
A Ambiguous	U Unclear		

Review Certification Report
Prepared by Facilitator when work product has passed review. Copy forwarded to project Software Quality Assurance (SQA) group.
Project Name:		Review Date:		
Name of item being reviewed:		
Item version identification:		
	
The following people have reviewed the named Item and agreed that all technical, contractual, quality, etc. requirements and review criteria have been satisfied.	
					
Facilitator:		
Recorder:		
Reader:		
Author:		
Software Quality Representative: 		
Inspectors		

		

		
Facilitator signature/date