Performing Cost-Effective Pre-Implementation Audits
Part 1 of 2
By: Mitchell H. Levine, CISA
Audit Serve, Inc.
Back in the 80s, prior to the automation of all aspects of the business processes, audit departments were able to amass their audit resources and perform pre-implementation audits for all high risk systems within their respective organizations. The 80’s also represented the breakout of the IT audit profession with the primary areas of growth being pre-implementation audits. Since larger organizations developed their applications in-house, there was justification for these pre-implementations audits to be performed due to the size of the project budget.
The common approach for these pre-implementation audits consisted of IT Auditors being included as part of the project team. Auditors would participate in all critical project meetings and review key project deliverables. However, with expansion of the number of systems being deployed by organizations in the 90’s, the audit department found itself with insufficient resources to perform pre-implementation audits for all of their high risk systems. In addition, the realization of the amount of time required to participate in all aspects of the project life cycle brought about the reduction of the number of pre-implementation audits. There was always a question of the value-added services audit provided to the overall project. This was more attributed to auditors who just followed the projects and “rubber stamped” the project deliverables they reviewed. This lack of “value” in many cases was attributed to the auditor not having system development or business experience. However, these criticisms were misguided because there are other project team members who are responsible for ensuring that the (1) new system meets the requirements of the business and that (2) technical solutions are properly integrated into the new system being deployed.
Instead, the true value the auditor brings to the new system deployment is the enforcement of a project management structure to ensure the project follows an appropriate systems development life cycle. The area where the auditor has unique expertise is the design and validation of controls which are included in the (1) business requirements which are transposed into (2) designed specifications and eventually (3) tested and (4) implemented into the newly deployed system.
Choosing the projects to perform pre-implementation audits has become a more complex task since the 80s. In the 80s, meetings were held with development managers during the budget process to identify new systems being deployed the next fiscal year. Currently, systems in many cases are no longer deployed centrally and have been extended across all business areas that have their own “mini development staffs” to deploy systems for their respective areas. Therefore, the audit department will need to have their “tentacles” throughput the organization to identify all new system deployments which may need to be included in the overall evaluation of systems requiring a pre-implementatation audit.
In the 90’s, the use of SDLC (Systems Development Life Cycle) audits were used as mechanism to reduce and potentially eliminate the need to perform pre-implementation audits. SDLC audits assume that all areas comply with a formal SDLC methodology for all system deployments (i.e., new systems, enhancements, maintenance projects). Assuming that a formal SDLC methodology was being followed, the auditor would select a sample number of system deployments (i.e., which included large system deployments which previously would have been subjected to a separate pre-implementation audit) to be included as part of their compliance test to determine whether these system deployments followed the SDLC. The SDLC audit would focus on the review of the critical SDLC deliverables (i.e., user requirements, system design specifications and test plans) along with assessment of project management controls, in order to conclude whether a system deployment successfully met the requirements. However, there were issues with using SDLC audits to replace standalone pre-implementation audits of system deployments. The first time the SDLC audits were performed, the SDLC methodology was found to be inadequate or did not exist at all. Therefore, there was no basis to assess the compliance to the methodology. In most cases the SDLC methodology was too generic and was not scaled to the SDLC deliverable requirements of the different types of system deployments of an organization. For instance, for a system enhancement which did not involve any complex changes (e.g., no new calculations) only a subset of SDLC deliverables (e.g., user requirements, design specification, user acceptance test plan) would be expected to be established for the project. Therefore, the audit compliance test, using the SDLC methodology as the criteria for the deliverables expected to be in place, would have yielded test failures for these missing deliverables. Obviously, if a separate standalone pre-implementation audit was performed for these system deployments (i.e., instead of being the audit samples selected for the SDLC audit), the auditor assigned to the project would have scaled their expectations based on their in-depth knowledge of the project scope. However, it would still be expected that an SDLC methodology be used for the project.
Overall, regardless if an SDLC audit or a standalone pre-implementation audit is performed, it is important for organizations to establish a scalable SDLC methodology. It is common for a checklist to be established which ties the SDLC deliverables which need to be established for each type of development activity used within an organization.
The second part of this article will focus on the various types of system deployments and the “angles” that an auditor should take in establishing the scope of their involvement along with key project strategy decisions which need to be enforced.