Treasury Board of Canada Secretariat
Symbol of the Government of Canada

ARCHIVED - Systems Under Development (Audit Guide) - March 1, 1991


Warning This page has been archived.

Archived Content

Information identified as archived on the Web is for reference, research or recordkeeping purposes. It has not been altered or updated after the date of archiving. Web pages that are archived on the Web are not subject to the Government of Canada Web Standards. As per the Communications Policy of the Government of Canada, you can request alternate formats on the "Contact Us" page.


Performing the Audit: Applied to the Systems Development Process

Introduction

This chapter deals with the audit process to be applied at each stage of a selected development project. The word "Stage" was selected for the sequential components of the SDLC in this Guide to avoid confusion with the word "Phase" that is used in audit methodology.

Scope and Purpose

The audit of Systems Under Development may have three main thrusts: first, to provide an opinion on the efficiency, effectiveness, and economy of project management; second, to assess the extent to which the system being developed provides for adequate audit trails and controls to ensure the integrity of data processed and stored; and third to assess the controls being provided for the management of the system's operation. These thrusts are clearly grouped for the auditor, in Chapter 3, by the presence of an A (Project Controls), B (Data Controls), or C (Systems Management Controls) letter as the second indicator in the Objective, Criteria and Detailed Criteria numbers.

The first thrust is pursued by having the auditor attend project and steering committee meetings, examining project control documentation and conducting interviews. The emphasis is on establishing, with the auditee, what project control standards are to be complied with, (such as a formal systems development process) and determining the extent to which compliance is being achieved. In carrying out this activity the auditor should keep in mind the requirements outlined in former Chapter 440 of the Treasury Board Administrative Policy Manual, the content of all circulars, policies and standards listed in Appendix J, and the material covered in the OCG's "Guide to the Audit of The Management Process".

As for the second thrust, the auditor is limited to examining system documentation, such as functional specifications, to arrive at an opinion on controls. The auditor's opinion will be based on the degree to which the system satisfies the general control objectives that any Information Technology system should meet. A list of such objectives should be provided to the auditee. The same is true for the third thrust, the system's operational controls. The auditor should provide the auditee with a list of the standard controls, over such operational concerns as response time, CPU usage, and random access space availability, that the auditor has used as assessment criteria.

Audit Phases

The audit of a system under development involves the conduct of certain audit procedures in connection with each stage of the SDLC. While this may appear to segment the process into separate and distinct audits, that is not the case. The audit of any stage or group of stages should consider all previous audits (or the lack of audit presence) in the continuous process of developing a system.

In conducting audits of systems under development the following activities, common to all audits conducted to Treasury Board standards, should be included in each audit stage:

  1. assignment planning
  2. review
  3. evaluation
  4. verification
  5. reporting and follow-up

For the most part, the above activities will be executed during a Systems Under Development audit, but there is some difference in how they are applied.

Special Planning Phase Considerations

All phases of an SDLC audit should be planned and be included in the initial planning for a system under development audit. As each subsequent stage of the SDLC is audited, the audit plan should detail the particular stage being audited and update plans for the remaining stages.

Review and Evaluation Phases

Compliance with an SDLC process will be reviewed and evaluated at each audit stage performed. However, as the documentation and/or programming of controls begins during the Feasibility Stage, review and evaluation of the data integrity and system controls can take place only at the Feasibility, General Design, Detailed Design, Implementation, Installation, and Post-Installation stages.

Verification Phase

Throughout the development process the auditor will verify compliance with the SDLC (see detailed approach in 4.A.10.1). The auditor will not, however, test the controls in the system being developed, but will review compliance with testing standards as per the SDLC. Where testing has been inadequate, the auditor should advise project management immediately. Testing is a project team and user function. Direct participation in the testing activity compromises the auditor's independence. However, the auditor may decide to re-perform selective tests to support control conclusions.

Special Report Timing Considerations

An important feature of Systems Under Development Audits is the avoidance of retrofitting costly controls. Such cost avoidance can be achieved only where communication between the auditor and auditee is at a level where action can be taken quickly in response to audit concerns. In order to fully support senior management, the audit reporting process must be guided by the systems development process. However, another important guiding principle is to ensure that audit findings are communicated, as soon as the auditor can support them, to the Project Manager level. To ensure audit independence and to keep senior management aware of SUD audit activity, summary audit reports should be made to coincide with project stages and check points. For example, if the process being followed provides for the stages shown below:

  • Project Initiation
  • Feasibility Study
  • General Design
  • Detailed Design
  • Implementation
  • Installation
  • Post-Installation

the related audit outputs would involve an Audit Plan Memorandum to be released to all levels of management during the Project Initiation stage audit with summary audit reports, according to the regular audit report process, after all subsequent audit stages.

Another special timing consideration is that of scheduling SUD audit activity to enable providing assistance to senior management at the time when the departmental approval of Treasury Board submissions is required. This would normally be in the form of delivering an opinion of the reasonableness of cost/benefit information contained in the submission, but could extend into the audit of other submission information.

General Note

The SDLC stages discussed above apply mainly to major new systems being developed or to major changes to existing systems. In developing smaller systems or where minor maintenance changes are made to existing systems, the SDLC stages may be grouped together or certain stages may be dropped completely. In the latter case, the project team must be careful that the system is not impaired. For example, inadequate consideration of alternatives in the Feasibility Study stage may lead to choosing an inappropriate alternative. Costs as a factor in a decision to drop a stage should be weighed against the degree of risk involved in doing so.

Prototyping, as a technique, has been described in Chapter 2. The concerns and expected controls in prototyping are included in the Initiation and Feasibility Stages.

Maintenance projects, that is the enhancements made to systems already in production, may be considered significant enough to be viewed as complete SUD projects in themselves. In that case, the audit concerns would be identical to those described below.

To ensure consistent development of all departmental projects, development standards should be in place and adhered to for each SDLC stage. However, it may be appropriate for the department to define a separate set of SDLC standards, based on the type of project being undertaken (i.e. major or minor system development). This will help ensure that minimum standards are applied in the minor systems development activities.

Lastly, in deciding whether a system is minor enough to justify grouping or eliminating some of the SDLC stages for an audit, the auditor should keep in mind that some relatively small system changes can be significant from a control point of view. The significance of a system amendment should be carefully assessed if it deviates from standards.

Control Objectives by Stage

Following are project management (A), data integrity (B) and Systems Management (C) control objectives, criteria, and detail criteria related to each stage of project development.

Project control is reviewed at every stage so that one can form an opinion about the efficiency, effectiveness and economy of project execution. Fortunately, the input, data integrity and system management controls being built into the system need be reviewed only at the Feasibility Study, General Design, Detail Design, and Implementation stages (see figure 4). The Feasibility Study stage is included in case control requirements are germane to selecting a design approach.

Note that while a few project control criteria are repeated from one audit stage to the next, the auditor should treat each audit stage of the guide as an independent stage. That means that the auditor must ensure that all objectives in all audit stages, prior to the one in which the auditor is working, are considered for application in the current audit stage.

Finally, the Chapter is organized by a combination of numbers and letters. This allows readers to situate themselves in a particular Stage and Audit Thrust at any point in the text. A diagram follows to ensure the numbering discipline is clear.

Figure 4: Activity by Stage Development

Stage Project Data System
Initiation YES NO NO
Feasibility Study YES YES YES
General Design YES YES YES
Detailed Design YES YES YES
Implementation YES YES** YES*
Installation YES NO NO
Post-Installation YES YES** YES**

YES means audit activity at this point.

NO means no audit activity possible or required.

* System Control concerns are covered as part of the review of Testing done at this stage. (See criterion 5.A.3.2 in Appendix F).

** Covers selective re-performing of testing of controls.

The audit information in the remainder of the chapter and associated appendices is organized and identified by means of a four field Dewey Decimal number (reference figure 5). This number will assist readers in identifying where they are at any point in the text.

Figure 5: Example of Dewey Decimal Numbering System

1.A.1.1

SDLC Stages:

  • Initiation
  • Feasibility
  • General Design
  • Detail Design
  • Implementation
  • Installation
  • Post-installation

1.A.1.1

Objectives:

A = Project Controls

B = Data Controls

C = System Development

1.A.1.1

Criteria

1.A.1.1

Audit Steps

1. Project Initiation Stage

It is essential that the auditor's participation be communicated formally to the Departmental Systems Steering Committee, by issuing a formal memorandum stipulating the need to include the auditor in Project Team and Steering Committee meetings.

After the auditor has completed the Planning Phase of the audit of the Initiation stage, a formal audit plan memorandum should be issued, outlining audit participation in the remainder of the project.

When compliance with the requirements of the SDLC at this stage is reviewed, any major non-compliance issues should be sent to the Steering Committee.

Project Control (A) Concerns

The "User or USER Management" term used in any Objective, Criteria, or Detailed Criteria refers to the "community of users", usually represented by one or more persons on the project team. The "community of users" can be one or many seperate organizations in a department. The auditor must ensure that the "community" is adequately represented on the project team.

Developing a new system must be clearly justified, usually by an economic analysis. In some cases, however, it may take the form of an evaluation of the need for improved or additional services, or other non-quantifiable concerns. In any case, some form of project justification should exist.

To be sure that management has made an informed decision to go ahead with the project, the constraints the project faces should be documented in the project documentation, along with a preliminary identification of the prerequisites by which the effectiveness of the new system will be assessed.

Project control, organization, responsibilities and authorities should be very clearly established.

Key Risks

The audit plan should include activities to address the following key risks which could contribute to not meeting user or project requirements:

  • vague identification of problem and project scope
  • acceptance of a project plan that does not contribute to corporate objectives or strategic planning
  • poor project management, including both human resource requirements and financial resource needs (i.e. budgeting)
  • inadequate assessment of project risk
  • inadequate documentation of security and privacy concerns, including the required and actual level of clearance and reliability of team members

Objective

1.A To establish that the project is formally initiated and that appropriate project control measures exist.

Criteria

1.A.1 The need for the project has been clearly stated in a Project Initiation Report or similar document.

1.A.2 The need for the project is acknowledged and financially supported by the appropriate level of user and Data Processing Management.

1.A.3 A project organization is outlined in the Project Initiation document.

1.A.4 A project development process is outlined in the Project Initiation document including tasks and responsibilities.

1.A.5 A work plan, including target dates, is provided in the Project Initiation document.

1.A.6 The project organization, the development process, and the work plan have been formally accepted by an appropriate level of management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix B.

Note 2: As shown in figure 4, there are no objectives for Data controls (B) and System controls (C) for the Initiation Stage (1) as there is no documentation of controls required in this stage (see Figure 3).

2. Feasibility Study Stage

During the Project Initiation stage the problem was defined. The Feasibility Study Stage conducts a study of general User Requirements to determine the appropriate conceptual solution in terms of organizational compatibility, economic justification, and technical suitability. Detail specifications will be prepared in the next stage based on these requirements, usually known as the USER REQUIREMENTS DOCUMENT.

Project Control (A) Concerns

The auditor should ensure that detailed user requirements have been properly identified and documented. The information contained in the User Requirements document should be gathered with great care directly from the users, by the project team, to be sure that they do not limit their definition of requirements because they are unaware of the new System's potential capabilities.

All practical alternatives should have been identified and analyzed. Facts and cost/benefit estimates used in the analysis must be reasonable and derived from valid sources. Conclusions made should follow logically from the analysis.

Resource estimates and time budgets must be complete and reasonable. Since these estimates are to be used for budget and project control, it is imperative that adequate detail be provided.

Key Risks

Audit activity, addressing the following key risks, should be incorporated into the audit plan.

  • Information provided to management for evaluation, approval and planning purposes may be incomplete or inaccurate.
  • The optimal alternative was not selected.
  • Inadequate planning, control, and administration.
  • Many users have difficulty in verifying their previously expressed needs when presented with a voluminous document. The auditor should determine that an optimum way of ensuring the user's active involvement in verifying User's Requirements (such as Prototyping described earlier in Chapter 2 of this Guide) was used by the project team.
  • Senior Management may treat this approval stage lightly because it consumes relatively few resources. As a result, work could proceed on later stages before this significant stage is complete.

Objective

2.A To establish that a feasibility study, including an Overall Project Plan, has been undertaken to determine the most appropriate solution to a stated problem in terms of organizational capability, economic justification, and technical suitability.

Criteria

2.A.1 User requirements are addressed in a User Requirements Report or similar document.

2.A.2 The accuracy and completeness of user requirements has been acknowledged by the appropriate level of user, and by Data Processing management.

2.A.3 The analysis of alternative processing configurations has been described in a Feasibility Study or similar document.

2.A.4 The appropriate level user and Data Processing management have confirmed that the analysis of processing alternatives including constraints or risks is accurate and complete, and they agree with the recommendations.

2.A.5 Resource estimates and other financial data have been addressed in a Cost/Benefit Analysis Report or similar document.

2.A.6 The accuracy and completeness of the cost/benefit analysis and acceptance of the recommended alternative has been acknowledged by the appropriate level of user and Data Processing management.

2.A.7 Based on the alternative recommended in the cost/benefit analysis, a Personnel Skills Summary has been prepared by the Project Manager summarizing the following information:

  • required skill categories (administrative and technical)
  • required skill levels
  • required number of skilled personnel
  • required authority level

2.A.8 Minutes of Steering Committee Meetings or similar document.

2.A.9 The status of the project compared to the work plan contained in the Project Initiation document has been addressed in a Feasibility Stage Project Status Report or similar document.

2.A.10 The accuracy and completeness of the Feasibility Stage Status document has been acknowledged by the appropriate level of user, and by Data Processing management, or concerns have been satisfactorily dealt with.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix B.

Data (B) and System Management (C) Controls Being Built into the System

At this stage of development the user, who is ultimately responsible for data integrity, must make the data control requirements (B) known. These requirements must be considered when carrying out the feasibility study and cost/benefit analysis to ensure their inclusion in the system. Both processing and security/privacy control requirements must be included in the User Requirements document, or prototyping equivalent.

System Management (C) Controls are the controls required to ensure that the system will continue to operate efficiently and effectively after installation. These controls are different from data and information controls and have different control objectives. These management controls over the operation of the system must be in keeping with the requirements definition for the system itself and with the cost benefit analysis.

Key Risks

Audit activities addressing the following key risks should be incorporated into the audit plan:

  • failure by user senior management to assess adequately the documentation of data control requirements and the chosen alternative's resolution of the requirements when they sign off this stage; and
  • failure of the user or operator to specify system management control requirements.

Objective

2.B To ascertain that data processed and stored by the system will be complete, accurate, and authorized, and that security, privacy, and accessibility levels for the system's data are specified.

Criteria

2.B.1 The need for processing control requirements is identified in a System Processing Controls Specifications or similar document.

2.B.2 The level of security, privacy, and accessibility of system data has been documented by the user representative including the sensitivity of system and data to loss, destruction, unauthorized access and changes.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix C.

Objective

2.C To ensure that the system operates efficiently, effectively, and economically.

Criteria

2.C.1 System management control requirements have been outlined in a Minimum System Management Controls Specifications or similar document.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix C.

3. General Design Stage

In this stage a detailed specification of the user's requirements is prepared from the system design alternative that has been approved in the previous phase. The specification that results is expressed in terms of those executing the business functions concerned and is free of technical design considerations. The functional specifications will then be translated into a system design in the next phase.

Security concerns, as noted at the end of Chapter 2, should continue as an audit concern in this Stage.

Project Control (A) Concerns

Project performance during the general design stage in comparison to the plans and budgets established at the Feasibility stage should have been monitored and variances justified to the project authorities.

The system, as designed, should meet the user's requirements. Control considerations (both financial and operational) should have been addressed. In some cases, the audit mandate may require evaluation of these application controls.

Both the manual and automated elements of the new system should have been addressed.

Documentation should address all elements of the system in enough detail to permit the detailed design of the system.

Key Risks

The following key risks should be considered in the audit plan:

  • incomplete/inaccurate identification and definition of key factors
  • lack of correspondence between defined and actual needs
  • inadequate assessment of the cost/benefit of the system
  • approval/authorization not obtained at designated checkpoints

Objective

3.A To ensure that the general design of the system expands on the findings of the feasibility study, produces a functional description of the manual and EDP processes, and devises an overall system design that can be used to obtain a commitment for further development.

Criteria

3.A.1 System specifications are addressed in a System Specifications Report or similar document.

3.A.2 The accuracy and completeness of system specifications has been acknowledged by the appropriate level of user and by Data Processing management.

3.A.3 The data dictionary/directory has been updated to reflect the contents of the System Specifications document.

3.A.4 All required skills are still available to the project.

3.A.5 Dates for committee meetings and the items to be discussed at each meeting continue to be addressed in a Steering Committee Meeting Schedule or similar document.

3.A.6 The status of the project compared to the budget and schedule contained in the Feasibility Stage Status document has been addressed in a General Design Stage Project Status Report or similar document.

3.A.7 The accuracy and completeness of the General Design Stage Status document and agreement with it has been acknowledged by the appropriate level of user and Data Processing management.

3.A.8 A human resources impact analysis is planned.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix D.

Data (B) and System Management Controls (C) Being Built into the System

The project team, made up of user and Data Processing representatives, must select techniques to satisfy the control requirements developed at the Feasibility stage. This selection is limited only by the team members' imaginations. The auditor's job at this point is to assess control, not the user's technical capability.

Key Risks

Audit activity, addressing the following key risks, should be incorporated into the audit plan:

  • inaccuracy of project data must be incorporated into the audit planning, to determine the extent, nature and timing of audit procedures for this stage and all subsequent SDLC stages;
  • failure of the user to specify techniques to satisfy System Management Control requirements, because "it's too early to know".

Objective

3.B To establish that data processed and stored by the system will be complete, accurate, and authorized.

Criteria

3.B.1 Processing control techniques to satisfy the requirements outlined in the List of Minimum System Processing Controls document have been outlined in a Processing Controls Specifications Report or similar document.

3.B.2 The accuracy and completeness of the processing control technique specifications have been acknowledged by the appropriate level of user and by Data Processing management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix D.

Objective

3.C To ensure that the system will operate efficiently and effectively.

Criteria

3.C.1 Control techniques to satisfy the requirements outlined in the List of Minimum System Management Controls are outlined in a System Management Controls Specifications Report or similar document.

3.C.2 The accuracy and completeness of the system management control technique specifications have been acknowledged by the appropriate level of user and by Data Processing management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix D.

4. Detailed Design Stage

During the Detailed Design stage, the functional specifications prepared in the previous stage are translated into a description of the system that will meet the specified functional requirements. The system design will then be implemented in computer-based and manual systems in the Implementation stage.

The main objective of this stage is to translate the user design specifications into systems, processes and data bases that will operate within hardware and systems software constraints.

Project Control (A) Concerns

Project performance during the detailed design stage compared to the plans and budgets established at the Feasibility stage (or as revised subsequently) should have been monitored and variances justified to the project authorities.

All elements of the system must have been designed in detail. Both the manual and computerized elements and the control features of the system must have been addressed.

If the audit mandate includes the evaluation of application controls an examination, as described in the "Guide to the Audit of Application Controls", may be required.

The detailed design is the complete and final design of the operational system. That is, there must be no apparent technical flaws or inconsistencies. Depending on the audit mandate, the auditor may either examine evidence that this has been established or satisfy himself by re-evaluating.

Key Risks

Audit activity, covering the following key risks, should be incorporated as part of the audit plan.

  • The underlying principles on which the testing approach is designed may be inappropriate and therefore lead to faulty conclusions.
  • Insufficient information on hardware and software characteristics and contract terms may prohibit optimal selections.
  • Inaccurate/incomplete assessments of the cost and benefits of the system.
  • The Implementation stage is begun before required planning for testing is complete.

Objective

4.A To ascertain that a detailed system design is developed from the functional specifications created in the general design.

Criteria

4.A.1 Programming specifications are addressed in a Detailed System Design Report or similar document.

4.A.2 The accuracy and completeness of Detailed System Design specifications has been acknowledged by the appropriate level of user and by Data Processing management.

4.A.3 The data dictionary/directory has been updated to reflect the contents of the Detailed System Design document.

4.A.4 Testing has been addressed in a Test Plan or similar document.

4.A.5 The accuracy and completeness of the Test Plan has been acknowledged by the appropriate level of user and by Data Processing management.

4.A.6 The testing covers all user requirements.

4.A.7 All required skills continue to be available to the project.

4.A.8 Dates for Committee meetings and the items to be discussed at each meeting continue to be addressed in a Steering Committee Meeting Schedule or similar document.

4.A.9 The status of the project compared to the budget and schedule contained in the General Design Stage Status document has been addressed in a Detailed Design Stage Project Status Report or similar document.

4.A.10 The accuracy and completeness of the Detailed Design Stage Status document and agreement with it has been acknowledged by the appropriate level of user and by Data Processing management.

4.A.11 A human resources impact analysis has been performed.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix E.

Data (B) and Systems Management Controls (C) being designed into the System

In this Detail Design Stage, the application control techniques, identified in the previous stage, have been developed into input, processing, and output controls.

In many respects, auditing data and system management controls now begins to resemble the auditing of an ongoing system. The main difference is that the auditor has to audit the test plan/design of the project team and should plan to re-perform control tests only selectively.

Input controls will involve the transmission, acceptance, conversion, and validation of data and the correction of errors. Processing controls will involve access restrictions and verification of data integrity between processing steps and within data bases. They will also minimize the impact of system failures in on-line systems. Output controls consist of overall reconciliation and balancing of output files and reports and the safeguards over them.

To a certain extent, the nature of the application controls developed will be a function of the particular system application. It is therefore not practical to try to anticipate the detailed systems design specifications associated with each control technique to be included. Again, Appendix I contains references dealing with the audit of Ongoing Systems. As well as reviewing the testing to be undertaken by the project team to verify that it covers control requirements, the auditor should begin to identify those controls that should be re-tested (note RE-TESTED, not TESTED) once the first programs are available, probably in the late Implementation or early Installation Stages.

Key Risks

Audit activity, addressing the following key risk, should be incorporated into the audit plan:

  • failure to include all control requirements in the test plan.

Objective

4.B To ensure that the data processed and stored by the system are complete, accurate and authorized.

Criteria

4.B.1 Processing control techniques outlined in the Processing Controls Specifications Report have been included for testing in the Test Plan or similar document.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix E.

Objective

4.C To ensure that the system will operate efficiently and effectively.

Criteria

4.C.1 Control techniques to satisfy the requirements outlined in the System Management Controls Specifications document have been included for testing in the Test Plan or similar document.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix E.

5. Implementation Stage

Based on the system design specifications documented in the previous stage, the system under development is implemented in computer-based and manual systems and put into operational status in the next stage.

Computer programs and manual procedures are written and tested. Training material and the Installation Schedule are prepared.

Project Control (A) Concerns

Project performance during the Implementation Stage compared to the plans and budgets established at the Feasibility stage (or revised subsequently) should have been monitored and variances justified to the project authorities.

System and program testing must be comprehensive and thoroughly documented. Problems encountered should have been addressed and rectified. The auditor may choose to re-perform selectively some tests but should never be perceived as being responsible for testing.

User manuals, input and output formats, screen layouts and any other form of user interface should be designed to optimize user efficiency.

Key Risks

Audit activity, covering the following key risks, should be incorporated as part of the audit plan:

  • Adequate site preparation may not have been performed. The auditor should take care to assess that telecommunication line installation, hardware delivery and physical site preparation activity have not been compromised because the installation date is close to the expiration of budgeted resources
  • Adequate training plans may not have been developed and adequately shared with the user
  • Systems personnel may not completely understand the users' needs
  • There may be inadequate documentation of systems design, programs/dialogue, and the user's manuals
  • Testing may be inadequate owing to time or other resource constraints

Objective

5.A To establish that all appropriate forms, manuals, programs and training materials are created from the detailed systems specifications.

Criteria

5.A.1 All manuals and other outputs required have been completed before installation begins.

5.A.2 The accuracy and completeness of the required manuals and outputs has been acknowledged by the appropriate level of user and by Data Processing management.

5.A.3 Testing results have been addressed in a Test Report or similar document.

5.A.4 The accuracy and completeness of the Test Report have been acknowledged by the appropriate level of user and by Data Processing management.

5.A.5 All required skills continue to be available to the project.

5.A.6 Dates for Committee meetings and the items to be discussed at each meeting continue to be addressed in a Steering Committee Meeting Schedule or similar document.

5.A.7 The status of the project compared to the budget and schedule contained in the Detailed Design Stage Status document has been addressed in an Implementation Stage Project Status Report or similar document.

5.A.8 The accuracy and completeness of the Implementation Stage Status document and agreement with it has been acknowledged by the appropriate level of user and by Data Processing management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix F.

Data (B) and Systems Management Controls (C) Being Built into the System

At this stage of development, to ensure that data integrity and systems management will exist, controls are being built into the new system. While the auditor's main efforts are focused on auditing testing activity, which has been covered in the section above on Project Control Concerns, there is the opportunity now to re-perform selected tests of key controls.

Appendix I contains reference material for determining appropriate techniques to re-test the selected controls, depending on the nature of the system and its environment. Real-time on-line systems, using data base management systems under the control of separate data administrative management will demand more sophisticated re-tests than typical batch input, tape master file systems. The judicious use of the reference material will enable an auditor with substantial EDP experience to perform efficient and effective re-testing.

Key Risks

Audit activity, addressing the following key risks, should be incorporated as part of the audit plan:

  • The total reliance by the auditor on the project team's documentation of its own team testing could result in erroneous judgements on the overall completeness and accuracy of project team testing
  • The auditor could expend effort in re-testing controls that are in the process of being re-programmed. It is important to determine how stable the test system is before re-performing selected control tests

Objective

5.B To ensure that key data controls are effective.

Criteria

5.B.1 Re-perform selected data integrity control tests.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix F.

Objective

5.C To ensure that key system controls are effective.

Criteria

5.C.1 Re-perform selected system integrity control tests.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix F.

6. Installation Stage

During this stage the system is put into operational status. Training is undertaken and files are converted.

The system is installed in accordance with the plan developed in the previous stage. This may require a phased installation, by geographical location, by organizational component, or location determined by need for instance. The last case requires needs criteria. "Sign-off" of acceptance is required from the user.

Project Control (A) Concerns

Project performance during the Installation Stage compared to the plans and budgets established at the Feasibility stage (or as revised subsequently) should have been monitored and variances justified to the project authorities.

Data conversion should be accurate. Testing of the data conversion process should be well documented. Depending on the quality and coverage of the testing performed by the auditee, auditors may wish to re-perform their own tests.

Key Risks

Audit activity, addressing the following key risks, should be considered during this audit stage:

  • inaccurate or incomplete conversion of files
  • inadequate training of personnel
  • poor management of cut-over
  • improperly functioning system, owing to incomplete testing or ineffective user involvement during acceptance testing
  • as fiscal year end, or any other type of calendar/time/money constraint approaches, training, testing, and project control can be short-changed, compromising the project team's reaction to the audit recommendations from previous stages
  • a tested contingency plan may not be available to the operators of the system

Objective

6.A To ensure that the system and any file conversions properly move from development status to operational and maintenance status.

Criteria

6.A.1 The accuracy, completeness, and authenticity of the files created by conversion are ensured through the use of appropriate control techniques.

6.A.2 Training has been carried out in accordance with the schedule prepared.

6.A.3 Installation was carried out in accordance with the schedule prepared.

6.A.4 The status of the project relative to the budget and schedule contained in the Implementation Stage Status document has been addressed in an Installation Stage Project Status Report or similar document.

6.A.5 The accuracy and completeness of the Installation Stage Status document and agreement with it has been acknowledged by the appropriate level of user and Data Processing management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix G.

Note 2: As shown in figure 4, there are no objectives for Data controls (B) and System controls (C) for the Installation Stage (6).

7. Post-Installation Stage

During this phase, the system will be in operation and will be adjusted using controlled system changes, to run correctly and according to current needs. A project report should be prepared, three to six months after installation, to indicate the degree of adherence to user functional requirements and the degree to which predicted costs/benefits were achieved.

Project Control (A) Concerns

Audit activity in this phase involves a review of the Post-Installation report and working papers against all earlier documentation, to ensure compliance with the SDLC and to attest to the accuracy and completeness of the findings.

Key Risks

Audit activity, addressing the following key risks, should be included in the audit plan:

  • An adequate number of trained operators, quality system and user documentation, and an availability of appropriate support may not exist
  • Support, in the form of "hot line" technical advice, Information Centre technicians, or any other appropriate form, may not exist
  • Documented operational inadequacies, so that management can prevent future inadequacies of the same type, may not exist

Objective

7.A To establish that the system operates in accordance with the design objectives and other measurement criteria, and project costs/benefits have been achieved.

Criteria

7.A.1 A formal post-installation review has been undertaken and the results reported to management.

Note 1: This objective and its criteria correspond to identically named Audit Programs in Appendix G.

Note 2: As shown in figure 4, there are no objectives for Data controls (B) and System controls (C) for the Post Installation (7).