Performing an Audit of an Automated
Mainframe Software Change Management System

By: Mitchell H. Levine, CISA
Audit Serve, Inc.

Introduction

Prior to the 1990’s, before automated software change management systems were used throughout the industry, manual processes were used to perform software change management. The available controls used in these manual processes were so limited that inherent weaknesses in change management had to be accepted. For those controls which were available, in many cases they were manual controls where enforcement required additional staff to support separation of duties. It also required a tremendous amount of review processes to be utilized.

This article is not intended to be a discussion about the various manual change management processes.

Currently, these controls are available in most automated mainframe change management systems such as Computer Associate’s Endevor or Serena Software’s Change Man products. The problem is that many organizations have not designed their change management systems in a manner which properly utilizes these control capabilities. In addition, most organizations have approached these products as a tool implementation and have not properly understood that they are installing processes which are an extension beyond the tool.

After reading this article, in order to apply these control objectives to your environment, a review would have to be performed of the change management system to determine how the control objective is handled and in order to identify the installation option which controls its use.

If the control is not available in the change management system, then it would be carried forward as an inherent weakness. It also should be noted that the terminology used to describe change management processes are not used universally across all automated change management systems.

Components installed into production are required to be migrated to UAT

Most change management systems have promotion levels which represent either centralized storage locations or actual test environments where testing is performed. Depending on the change management system, these promotion levels may or may not be required to be performed prior to allowing a software release from being installed into production. If these promotion levels, which represent UAT (User Acceptance Test) environments, are not required to be populated by installed releases, then there is no enforcement to ensure that proper testing is occurring. This enforcement can occur if there is person involved in the change management process who performs a review of all software released prior to their installation to ensure that they were promoted to the test environment during its change management life cycle.

 Components migrated to UAT environment cannot be changed

Most change management systems provide an installation controlled option to determine whether releases (i.e., collection of software components which comprise the software project) are required to be frozen (i.e., prevent changes from occurring). The freeze of a release (i.e., referred to as "package" in many automated software management systems) is required prior to the migration to the promotion level used for UAT. Otherwise, the integrity of the test performed cannot be assured.

Preventing overlays in the UAT environment

Depending on the automated change control system being used, each release (i.e., project) may be isolated from one another during most of the change management life cycle. This is achieved by the change management system by defining a separate set of libraries for each project. Each software component (e.g., program, parameters, PROCS, JCL) would also be defined within its own library as it pertains to each project. This approach prevents a shared module between two projects from be accidentally overlaid. However, the UAT environments are a shared set of libraries where inadvertent overlays can occur if they are not properly controlled. The change management system in most cases cannot be configured to prevent this type of an overlay but a warning message is sent to the person attempting the promotion. For this reason, it important to consider establishing an independent person to perform the promotions to the UAT environment. The policy should be either wait till the project residing in UAT to complete its install process or require that package to migrate out of the UAT environment to allow the other project with the duplicate module to be promoted to UAT. Otherwise, overlays will occur and the wrong version of a module will be executed during the UAT test. If the change management system migrates to the production environment from UAT, then a much greater exposure exists that is the wrong version of a module will be installed in production.

Control concurrent development

Concurrent development (i.e., utilizing the same module within two projects) can be set-up to either prevent it entirely or allow it. In most development environments it is not realistic to expect concurrent development from being prevented since there are certain modules which are changed in many releases. Therefore, when allowing concurrent development, it is important to ensure that the options in the change management system are set to send messages to all parties using the same module. The warning message should prompt all impacted parties to coordinate their changes to ensure they account for each others changes.

In order for this coordination effort to be effective, it is important that the change management system is used at the start of development which will monitor all activity. However, depending on the design of the change management system, developers may have the ability to perform most of their development outside the control of the change management system. By allowing this to occur, the developer will checkout (i.e., indicate to change management system that the production module is being used) the module they plan to use for a project at a later stage. However, the warning messages indicating concurrent development may be too late to allow for proper action to be taken. To prevent this from occurring, an enforced policy must be in place to require the developer to checkout components at the early stages of the project.

Another requirement to control concurrent development is to ensure that all modules which exist in production are required to be checked out from production. The change management system allows new modules to be migrated from libraries outside the control of the change management system. However, if a module exists in production it must be the version which is used for all changes. Otherwise, the developer may not be using the most current version and will reverse out the changes made to the last version which was last installed into production.

The only exception to the process is new releases received from third party vendors since the production version would be considered the "-1" version. However, when setting up a change management system to allow the bypass of the control which requires existing production modules to be checked out, a security process must exist to ensure that only vendor releases can use it.

The last component required to control concurrent development is to ensure that dormant projects are not allowed to migrate forward in the change management lifecycle if another package using the same module leapfrogged the project and migrated the module to production. If this was permitted, then the dormant project when installed into production would reverse out the changes of the previously installed release. The enforcement of this control is an installation option within the change management system.

There are other critical components which must be included in the design of the change management system, which are specified as follows but are not explained in the context of this article:

- controlling of shared components such as copybooks (i.e, data definitions) to ensure that all referenced programs are forced to be recompiled

- security over system libraries and called parm libraries (i.e., parm file containing compile and link options) used in compile and link process

- process for performing a back out of a release when the entire release is not desired to be backed out

- set-up of package groupings to ensure that development groups cannot access each other’s modules

- inventory of all change management component types to ensure that all modules are controlled through the change management system

- proper set-up within change management system to ensure that all compilable modules are forced to be compiled

- proper set-up in change management system to prevent load modules from being directly changed

- set-up of test environments to ensure that only modules currently being changed are actually contained within the environment

Conclusion

There are additional upgrades required within the change management controls to support the Y2k project which are not discussed in the article but must be defined.

Products exist for control software change management on Client/server systems but primarily focus on controlling change migration. However, they are not integrated sufficiently to support configuration management (i.e., shared components and standardized compile procedures).

For a free proposal to perform an audit of your organization or provide SOX support & testing services, contact Mitchell Levine of Audit Serve at (203) 972-3567 or via e-mail at Levinemh@auditserve.com.

Copyright  2006, Audit Serve, Inc. All rights reserved. Reproduction, which includes links from other Web sites, is prohibited except by permission in writing.