Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Pre-Implementation Audit (PIN) - October 10, 1984

Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.


No. 1984-03 
Date: 10-10-84 

Subject: Pre-implementation Audit

Question concerning this notice should be directed to:

Policy and Special Projects,
Centre of Excellence for Internal Audit
Comptrollership Branch, TBS
(613) 957-2270

Purpose and Scope

Modern internal auditing, as defined in the Standards1, has two main thrusts, namely: (l) helping managers achieve improved productivity ( auditing economy, efficiency and effectiveness) of their ongoing operations; and (2) advising management on the development of economic, efficient and effective major, new, or changed infrastructure (pre-implementation auditing).

The purpose of this Policy Interpretation Notice (PIN) is to provide the internal audit community with guidance related to pre-implementation audit. It supplements standard No. 2, dealing with "scope" and the associated discussions in Chapter Two, Scope and Frequency, and Appendix A, Auditing Computer-Based Systems.[1]

Issues

1. What is a pre-implementation audit?

A pre-implementation audit is an audit carried out on departmental/agency systems during the design/development and installation process rather than after the system has been turned over to the client for operation.

2. What is the scope of a pre-implementation audit?

To be consistent with the Standards, the scope of pre-implementation audits should include all major new systems under development, including: new legislation, policies and procedures, information systems (both EDP and non-EDP) and production processes.

Pre-implementation audit consists of two possible components:

  1. Audit of the project management/system development process ( for EDP systems, this includes the complete systems development life cycle[2]), and
  2. Audit of the control framework being designed for the system.

3. Why do a pre-implementation audit?

The rationale for initiating pre-implementation auditing is that it is more cost-effective to correct weaknesses in the control framework during the design/ development and installation process than after implementation, when large quantities of resources have been expended and strong commitment to the entity under design has been generated.

This does not eliminate the need for post-implementation audits as there is no assurance that what was designed and installed was maintained or operated as intended, and that the original requirements continue to hold true.

4. How far can the auditor go without impairing independence?

Auditors always have the problem of possibly compromising their objectivity through their efforts to gain an understanding of, and empathy with, the auditee's situation. This is particularly true for pre-implementation auditing. For example, there is a real danger that the auditor will get co-opted into participation in actual design of controls rather than playing the "advice" role. Even in playing this more restricted role the auditor may become too committed to the resultant systems configuration to be totally unbiased in later, post-implementation audits of that same system.

It is considered however, that the benefits of such auditing far outweigh the risk incurred and that, in any case, it is possible to minimize the danger of loss of objectivity by judicious control of the nature and scope of the auditor's involvement, and by adopting an appropriate assignment strategy.

Interpretation Notice Position

The position of this PIN is that: Pre-implementation audits should be undertaken for all major systems under development in departments and agencies; they should be reflected in the departmental/agency internal audit policies and plans; and, the potential loss of auditors' objectivity can be minimized through appropriate terms of reference and assignment strategy.

Disposition

It is our intention to adopt one or more of the following actions, based on the nature of the feedback received from the internal audit community:

  1. Prepare a chapter on pre-implementation auditing for Volume II of the IACIA Internal Audit Handbook (IAH).
  2. Prepare a guide on pre-implementation auditing for Volume III of the IAH.
  3. Integrate pre-implementation audit guidance into existing IAH chapters.
  4. Prepare changes to the relevant portions of the Standards for their next revision.

The internal audit community is invited to provide comments to IA&SSD on the attached Discussion Paper with the view to influencing the content and location of future guidance on this important subject.

Introduction

The purpose of this Paper is to explore various aspects of participation by auditors in systems under development. This type of participation will be termed "pre-implementation audit".

The subject of pre-implementation audit will be discussed from the point of view of its background, definition, scope, purpose, effect on other aspects of the audit function and possible means of implementation.

Background

There are several factors which have the effect of increasing pressure on internal auditors to participate in systems development projects: systems development projects are notorious for cost/time overruns; implemented systems are equally notorious for not meeting all user requirements; systems, particularly EDP systems, often have under-designed control frameworks; and, recent cost-cutting programs, prompted by a recessionary economy and large deficits, have focussed increased attention on improving the productivity/efficiency of all processes. This puts the spotlight particularly on the systems development process because of the costly down-stream effects of inadequate design and implementation.

Audit literature has recognized the potentially useful role of pre-implementation audit for some time, however, it has been almost exclusively centred on EDP information systems. This view of systems has two limitations: not all systems are "information" systems and EDP is only one possible vehicle for implementing systems. In what follows, we will employ a broader view of systems1.

Our recognition of the desirability of pre-implementation audit is reflected in the Standards2 in several ways:

  1. Standard 2., Scope, states that "The scope of internal audit shall encompass all aspects of a department's operations. The internal auditor assesses and expresses an opinion upon:
    1. The design, development, implementation ... of all systems, procedures, processes and controls, including computer-based systems; ...";
  2. Standard 3., Frequency, adds that "All major systems, ... performing significant responsibilities should be examined within a period not exceeding three to five years..."; and,
  3. In the elaboration of standards 2. and 3., in Chapter Two, it is made clear that pre-implementation audit was intended to include new legislation, agreements and contracts.
    1. A system is defined as a set of elements related to one another according to some coherent pattern. While the elements are important, it is the linkages or relationships among the elements, defined in terms of common purpose, which make it possible to speak of a system - abstracted from Managing Public Systems: Analytic Techniques for Public Administration, by Michael J. White, Clayton Ross, Robert Myrtle, Gilbert Siegel and Aaron Rose.
    2. Standards for Internal Audit in the Government of Canada.

Definition

As indicated above, we will term the auditor's participation in systems under development "pre-implementation audit" and will adopt the broadest definition of the term "system", a definition which encompasses all major infrastructural mechanisms in an organization, including: legislation, policies and procedures; information systems (EDP or other); production systems/processes; agreements and contracts; and organization structures. See Figure 1 for an illustration of typical infrastructural hierarchy and the relationships.

Scope and Purpose of Pre-Implementation Auditing

The scope of a pre-implementation audit has two components. The first has to do with the entities subject to pre-implementation audit - these have already been defined above as all major new systems (i.e. infrastructural hierarchy) and should include major re-designs of existing systems.

The second component of scope has to do with the type of auditor participation that could be contemplated. These are:

  1. Audit of the project management/systems development process.
  2. Audit of the control framework that is being designed in conjunction with (surrounding), or as an integral part of the system under design.

The first type of audit is aimed at assuring management that the development process adheres to prescribed (e.g. central agency) or generally accepted policies and practices, which ensure that sound systems are conceived, developed and implemented and that the development process is economic, efficient and effective. The second type of audit assures management that adequate operating and management controls are being conceived, developed and implemented, which in turn will ensure that the operators, managers and users of a system will have a means of determining whether the system is performing as intended.

For both types of audit the rationale employed is twofold. The first is that it is easier and less costly to ensure adequate design at its conception than to rely on remedial changes after implementation. The second is that systems that will be used repeatedly must be more carefully designed and managed at the front end because of the potentially negative effects on operating efficiency and effectiveness that could result from a poor design/development process.

A derivative benefit from pre-implementation audits is the resultant heightened awareness of the role of control accruing to all participants (managers, users, systems designers, etc.).

Infrastructural Hierarchy

The Effect of Pre-implementation Audit on Other Aspects of the Audit Function

There are two aspects to consider. One is the effect on other types of audit and the other is the effect on the audit function's independence.

A pre-implementation audit should reduce the number of systems-oriented findings that the audit group is likely to identify in future periods, however, it is unlikely to eliminate them. There are three reasons for this: It is unlikely that any design will be perfect to start with; even if a design is perfect, it is unlikely that it will be implemented, maintained and operated exactly as designed; and finally, it is unlikely that the original environmental conditions or requirements will continue to hold true over the entire life of the system.

Independence is a difficult aspect to deal with as it is very subjective. Various auditors and managers will have differing views as to what independence is and when it is jeopardized. Guidelines which may prove useful in this context follow:

  1. Avoid participation in actual design work (this parallels the traditional admonishment that auditors not participate in operations that they will be expected to audit in the future).
  2. Specify the need for "key controls" but not what form they should take (i.e. specify the "what" and not the "how").
  3. Negotiate the ground rules for participation ahead of time such that all parties understand and agree on their respective roles - independence and objectivity as an intent can be spelled out at that time (for example, specify that user managers should provide control objectives, systems designers should design the controls, and auditors should advise on the adequacy of controls).
  4. Ensure that the auditor who participates in the pre-implementation audit is not the one who performs future post-implementation audits on the same system.

Although the potential for loss of objectivity in performing audits cannot be eliminated completely, it can be limited. Judicious control of the degree of auditor involvement in pre-implementation audits can minimize both the existence and appearance of such loss of objectivity.

Possible Means of Implementation

To start with, it is assumed that all departmental/agency internal audit policies and plans appropriately reflect provisions for pre-implementation audit in accordance with the Standards.

Other prerequisites to successful implementation of such audits would include: a requirement (in departmental/agency policies or directives) that managers notify the head of internal audit of all major systems development projects (this does not preclude the internal audit group performing its own surveillance through review of plans and other relevant documentation and through personal contact with managers); a requirement that managers/project leaders invite the internal audit group to participate in all major systems development activity (including participation in systems development teams, systems development Steering Committees, etc.); active support of senior management for such participation, and, the acquisition of a sufficient number of appropriately qualified senior auditors who would lend credibility to the systems and to the internal audit function by their participation (contracting is an alternative, where resource budgets permit).

Although a comprehensive guide to pre-implementation auditing is beyond the scope of this Paper, some suggestions are presented for your consideration:

  1. Although it is desirable that the auditor have a background in the subject-matter area of the proposed system (e.g. finance, personnel, specific program/activity) and in the technology expected to be used in the design of the delivery vehicle (e.g. EDP, micrographics) it should be kept in mind that the internal auditor's role is that of control not subject- matter expert.
  2. It should be kept in mind that the main difference between pre- -implementation and post-implementation auditing is one of timing. Therefore the methods and techniques used do not change appreciably.

    For example, for the audit of the systems development process the relevant criteria would include those provided for project management, contracting and EDP systems in the Administrative Policy Manual and in the Guide on Financial Administration for Departments and Agencies. Also, relevant IACIA Guides (e.g. EDP Audit) are equally usable for pre-implementation auditing.

    In the case of pre-implementation audit of the control framework being designed into or around the system under development, the starting point would be a pre-determined, normative control model, as it is for any audit, except that in this case there would be only a paper representation of an actual control framework to compare it with, rather than the physical one that would exist in the case of a post-implementation audit of the system.

  3. In performing pre-implementation audits two key principles need to be kept in mind:
    1. Every system has to have the means whereby the operator and manager of that system can determine whether it is performing as intended, i.e. it must have controls.
    2. Where the auditor is involved in the pre-implementation audit of only one element of the infrastructural framework (see Figure 1) the auditor has to be very aware of its relative position and role in the hierarchy. It is unreasonable to expect the same degree of specificity/precision in the specification of objectives, criteria, etc., and associated controls at each level. For example, at the highest level (legislation), the designers should not be expected to incorporate detailed implementation procedures and performance criteria as would be expected at the systems and procedures level. What one would expect is progressive elaboration as one moves down the hierarchy, and consistency and continuity between levels.

      In general, when an auditor is involved in a pre-implementation audit at an intermediate level in the hierarchy, it is important for that auditor to become familiar with the role of adjacent (higher, lower and peer level) elements in the structure in order to be able to judge consistency and continuity.

  4. The auditors assigned to pre-implementation audits should be those senior auditors in the group with the highest level of relevant expertise. This is due to the crucial nature (downstream impact) of the entity under audit.
  5. The timing of the start of the auditor's participation should coincide, ideally, with the start of the formation of the development team, and certainly no later than coincident with the statement of the user's requirements.
  6. The auditors participation in a pre-implementation audit is unlikely to be full-time but rather at key points in the development process (e.g. at, or just prior to: turnover of user specifications to the systems designer, turnover of the systems design to the programmer/procedure writer, system testing and cut-over/user acceptance). This type of on-off participation may aggrevate the audits manager's resource allocation process, however, close collaboration with the systems development project leader can minimize the problem.
  7. The timing of the first post-implementation audit of a new system, or major change, should be no sooner than six months after its turnover to the user/operator, i.e. after it has progressed from transient to reasonably steady state.

Conclusions

Pre-implementation audit is a valuable managerial aid. This has been recognized in the literature on auditing and reflected in the Standards.

This type of auditing applies to all major infrastructural mechanisms (systems) under development. It encompasses two distinct activities, i.e. audit of the development process and audit of the control framework being designed.

Pre-implementation auditing methods and techniques do not differ radically from normal, post-implementation auditing. However, the skill of the auditor applying those methods and techniques must be of the highest order because of the downstream impact of such an activity.

Bibliography

  1. Lambrix, Robert J. and Singhui, Surendra S., "Preapproval audits of capital projects", Harvard Business Review (March-April 1984).
  2. Moser, J.W., "Assessing Controls in an Application During System Design", I.A. Communique (Feb. 1983), Treasury Board of Canada, Comptroller General, Internal Audit and Special Studies Division.
  3. Rittenberg, Larry E., Auditor Independence and Systems Design, the Institute of Internal Auditors, Inc., Altamonte Springs, Florida, U.S.A., 1977.
  4. Stidwell, Joanne, (edited by Donald A. Brown) "Improving the System Development Life Cycle", C.A. Magazine.
  5. The Institute of Internal Auditors, Standards for the Professional Practice of Internal Auditing, The Institute of Internal Auditors, Inc., Altamonte Springs, Florida, U.S.A., 1978 (latest re-issue Feb. 1984).
  6. Treasury Board of Canada (Comptroller General), Standards for Internal Audit in the Government of Canada, Treasury Board of Canada, 1982.
  7. Treasury Board of Canada, Administrative Policy Manual, Treasury Board of Canada.
  8. Wysong, Jr., Dr. Earl M., "Using the Internal Auditor for Systems Design Projects", Journal of Systems Management (July 1983), Association for Systems Management, Cleveland, Ohio, U.S.A.

End Notes

1. See Standards for Internal Audit in the Government of Canada.

2. See the Administrative Policy Manual, Chapter 440.