Mainframe Backup & Recovery: Pre-Batch Checks

Before a batch job runs, you must verify that the mainframe’s backup and recovery environment is ready: the last full and incremental backups completed successfully, the recovery catalog is synchronized, and the target volumes or datasets that the job will write to have sufficient space and are not protected by a pending restore. Skipping these checks can turn a routine batch window into a multi-hour outage if a dataset is accidentally overwritten or a tape is missing. Below is a practical checklist and the reasoning behind each item, written for operators and batch schedulers who need to protect production data without slowing down the job stream.

Why Pre-Batch Verification Matters

Batch jobs are the backbone of mainframe processing—payroll, billing, end-of-day updates, and reporting all run in scheduled windows. A batch job typically reads input datasets, updates master files, and writes new output. If a backup is incomplete or a recovery procedure is misconfigured, a job can corrupt a production dataset, and the only way back is a restore. Restores take time, and during that time the batch window is missed, downstream jobs fail, and business users see errors. The cost of a failed restore is far higher than the cost of a few minutes of pre-flight checks.

Mainframe environments also have unique characteristics that make verification non-negotiable: datasets are often shared between jobs, catalogs are hierarchical, and tape volumes can be offsite. A backup that looks successful in the log may still be unusable if the tape was not labeled correctly or the catalog entry points to a stale version. Verifying before the job runs means you catch these issues while there is still time to fix them.

Core Checks: What to Verify Before Any Batch Job

The following list covers the minimum verification steps for a typical z/OS environment using DFSMSdss, DFSMShsm, or a third-party tool like BMC Control-M or CA-1. Adjust based on your shop’s tools and policies.

• Last backup completion status: Check the job log or backup report for the most recent full and incremental backups of every dataset the batch job reads or writes. Look for condition code 0 or a clear “successful” message. Any abend (ABEND) or warning means the backup is suspect.
• Recovery catalog synchronization: Ensure the backup control dataset (BCDS) and the recovery catalog (e.g., for DFSMShsm) are current. If a dataset was added or deleted since the last backup, the catalog must reflect that, or a restore may target the wrong generation.
• Dataset existence and integrity: Run a quick LISTCAT or DSLIST to confirm that all input datasets exist, are not held by another job, and have the correct record format and block size. A missing or corrupted input is the most common cause of batch abends.
• Free space on target volumes: For output datasets, check that the target volume has enough free tracks or cylinders. Use LISTVTOC or the storage management tool to see free space. Do not assume—a full volume is a classic cause of a job failing at the last minute.
• Tape and virtual tape availability: If the job writes to tape, verify that the tape drives are online, the correct volumes are mounted (or will be mounted by the scheduler), and the tape labels match the expected dataset names. Offsite tapes may need to be recalled, which takes time.
• Pending restore or migration flags: Check if any dataset is in a migrated state (e.g., to ML2 or ML1) or has a pending restore. A batch job that reads a migrated dataset will trigger an automatic recall, which can cause a delay or fail if the recall is not allowed. Use HMIGRATE or the HSM query commands.
• Security and RACF/ACF2 permissions: Confirm the batch job’s user ID has read access to input datasets and write access to output datasets. A permission failure will stop the job immediately, but it is better to catch it before the job starts.
• Job scheduling dependencies: Verify that all predecessor jobs completed successfully and that any required restart or checkpoint datasets are present. If a predecessor failed, the backup may not include its output, and the current job could run against stale data.

How to Verify: Commands and Reports

Verification is not a single action; it is a combination of automated checks and manual review. Most shops use a scheduling tool that can run pre-job validation steps, but you can also do it manually via TSO or ISPF.

• Use the scheduler’s pre-flight feature: Tools like Control-M, CA Workload Automation, or IBM Workload Scheduler allow you to define a pre-run job that checks conditions. For example, a job can run a REXX script that queries the backup report and abends if the last backup is older than 24 hours.
• Review the backup job log: For each backup job, look at the SYSPRINT and the job log. A condition code of 0 is good; anything else requires investigation. Pay attention to messages like IEF404I for job completion, but also scan for IEC161I (tape error) or IGD100I (catalog error).
• Check the catalog with LISTCAT: Run LISTCAT ENT('DATASET.NAME') ALL to see the dataset’s volume, device type, and last reference date. If the dataset is not in the catalog, the job will fail with a S013 or S114 abend.
• Use HSM QUERY: For managed datasets, QUERY BACKUP shows the last backup date and whether a backup exists. QUERY MIGRATION shows if a dataset is migrated and to which level.
• Automate with a REXX or CLIST: Write a small REXX exec that checks the backup report and returns a non-zero return code if any condition is not met. The scheduler can then hold the job.

Common Pitfalls and Misconceptions

Even experienced operators make mistakes when verifying backups. Here are the most frequent ones:

• Assuming a backup job’s condition code 0 means the data is restorable. A backup can complete with a 0 but still miss a dataset because of a JCL error (e.g., a wrong DD name). Always spot-check the backup report’s dataset list against the expected list.
• Forgetting about the recovery catalog. The recovery catalog is separate from the system catalog. If it is not updated, a restore request may fail even though the backup tape exists. Verify that the catalog update job ran successfully.
• Ignoring tape expiration dates. A tape with an expired retention period may be overwritten by another job. Check the tape label and the retention policy before relying on it for recovery.
• Not checking for pending restores. If a previous restore is still in progress (e.g., a large dataset being restored from tape), the batch job may conflict with it. Use the restore job’s status to avoid contention.
• Assuming free space is enough. A volume may have free tracks but not enough contiguous space for a large sequential dataset. Use LISTVTOC with the FORMAT option to see the largest free extent.

Automating Pre-Batch Verification

Manual checks are error-prone, especially in a 24/7 operation. Automating verification reduces risk and frees up operators. Here is a practical approach:

1. Define a standard verification job that runs before the batch window. This job can be a single REXX exec that checks all critical datasets and backup statuses.
2. Integrate with the scheduler. Make the verification job a predecessor of all batch jobs. If it fails, no downstream job starts.
3. Use exit codes to control flow. The verification job should return a non-zero code if any check fails. The scheduler then holds the batch stream and notifies the operations team.
4. Log verification results to a dataset that can be reviewed later. This provides an audit trail for compliance.
5. Test the verification job itself. Introduce a deliberate error (e.g., a missing dataset) and confirm that the job catches it. Do this in a test environment, not production.

Automation is not a substitute for a periodic manual audit. At least once a quarter, perform a test restore of a critical dataset to ensure the backups are actually usable. This is the only way to be sure the recovery process works.

Comparing Backup and Recovery Tools

The verification steps may vary slightly depending on the backup tool. Here is a comparison of the most common mainframe backup methods:

• DFSMShsm (HSM): Provides automatic backup and migration. Verification focuses on the HSM control datasets and the backup schedule. Use QUERY BACKUP and QUERY MIGRATION to check status.
• DFSMSdss (DSS): Used for full-volume or dataset dumps. Verification involves checking the dump job’s return code and the dump dataset’s catalog entry. DSS is often used for disaster recovery copies.
• IBM GDPS: For continuous availability and disaster recovery. Verification includes checking the copy consistency and the health of the coupling facility. GDPS is complex and requires specialized monitoring.
• Third-party tools (e.g., CA-1, BMC AMI): These often provide a unified view of backup and recovery. Verification may be done through a GUI, but the underlying checks are similar: backup status, catalog, and tape availability.

Regardless of the tool, the principle is the same: verify that a recent, restorable backup exists for every dataset the batch job touches, and that the environment (catalog, tapes, volumes) is ready to support a restore if needed.

Frequently Asked Questions

Q: How far back should the last backup be before I let a batch job run?
A: It depends on your recovery point objective (RPO). If you can afford to lose a few hours of data, a daily backup may be enough. If not, you need more frequent backups, such as incremental every hour. The key is that the backup must be recent enough that a restore would not lose more data than your business allows.

Q: What is the difference between a full backup and an incremental backup for mainframe?
A: A full backup copies all data in a dataset or volume. An incremental backup copies only the changes since the last backup (full or incremental). To restore, you need the last full backup plus all subsequent incrementals in order. Verification should confirm that the entire chain is intact.

Q: Can I rely on the backup job’s condition code to know it worked?
A: No. A condition code of 0 means the job step ended normally, but it does not guarantee that every dataset was captured. Always check the backup report for the list of datasets and compare it to what you expect. Also verify that the backup data is readable by doing a test restore periodically.

Q: What should I do if a dataset is migrated and a batch job needs it?
A: The job will trigger an automatic recall, which may take time. To avoid delays, you can explicitly recall the dataset before the job using HRECALL or HMIGRATE with the RECALL option. Verify that the recall completed successfully before the job starts.

Q: How do I check if a tape volume is available for a batch job?
A: Use the tape management system (e.g., CA-1, RMM) to query the volume’s status. Check that it is not in use by another job, that it is not expired, and that the drive is online. If the tape is offsite, you may need to request a recall, which can take hours.

Q: What is a recovery catalog and why is it important?
A: The recovery catalog (or backup control dataset) is a database that tracks which backups exist, where they are stored, and their retention dates. Without it, a restore request cannot locate the correct backup tape. Verify that the catalog is current and backed up itself, because if you lose the catalog, you lose the ability to restore.

Related

• What to Do After Device Is Rooted? Next Steps
• Turn Steam Frame Files Into Custom Accessories
• Apple Developing Alternative to Google Search?