IBM i CIS Compliance: What Benchmark Automation Can and Can’t Do

IBM i (AS/400, iSeries) systems face a specific compliance challenge: the Center for Internet Security (CIS) publishes a benchmark for IBM i, and automating the audit of that benchmark can save hundreds of hours. Benchmark automation can quickly collect system values, user profiles, and object authorities to show where you stand against the CIS rules, but it cannot interpret business context, remediate every finding, or replace a human auditor’s judgment. This article explains exactly what automated CIS benchmark scanning on IBM i does, where it falls short, and how to use it effectively.

Background: What the CIS IBM i Benchmark Actually Contains

The CIS IBM i Benchmark is a set of configuration recommendations published by the Center for Internet Security. It covers areas like user profile security (password rules, *ALLOBJ authority), system values (QSECURITY, QMAXSIGN, QPWDEXPITV), object ownership, auditing settings (QAUDCTL, QAUDLVL), and network attributes. Each recommendation is labeled with a severity level (Level 1 or Level 2) and a mapping to NIST 800-53 or other frameworks. The benchmark is not a law; it is a consensus-based checklist that many organizations adopt voluntarily to demonstrate due diligence in audits.

The benchmark is lengthy—hundreds of individual checks. Doing this manually on a single partition means running CL commands, dumping output to spool files, and comparing values against the benchmark table. On a fleet of LPARs, this becomes a full-time job. That is why automation tools exist, but they vary widely in what they actually automate.

What Benchmark Automation Can Do

Good automation on IBM i can reliably perform the mechanical parts of a CIS audit:

• Collect system values: Retrieve all relevant system values (e.g., QSECURITY, QMAXSIGN, QPWDEXPITV) in one pass, without manual DSPSYSVAL commands.
• Enumerate user profiles: Pull every profile, its status, password expiration, special authorities, and group memberships.
• Check object authorities: For key objects (e.g., QSYS libraries, commands like CHGUSRPRF), list who has *ALLOBJ or *SECADM authority.
• Compare against the benchmark: Automatically flag each setting that does not match the CIS recommendation, with a pass/fail status.
• Generate reports: Produce a CSV or PDF that lists each check, the current value, the required value, and the severity.
• Track drift: Run the scan periodically and compare results to see what changed since the last audit.

These tasks are deterministic—the tool reads a value and compares it to a threshold. That is exactly what a computer does well. A well-built scanner can cover 100% of the benchmark’s technical checks in minutes, with zero missed entries due to fatigue.

What Benchmark Automation Cannot Do

Automation hits a hard wall when the benchmark requires judgment or context. Here are the realistic limits:

• Interpret business necessity: A user profile might have *ALLOBJ because it is a backup operator account, not because of poor hygiene. The scanner flags it; a human must decide if this is acceptable risk.
• Remediate automatically: Most tools will not change system values or authorities without explicit approval. Even if they could, changing QSECURITY or revoking *ALLOBJ can break applications. Automation should never silently alter production settings.
• Handle exceptions: The benchmark allows exceptions (e.g., a service account that must have *IOSYSCFG). A scanner can only flag; it cannot know why an exception exists.
• Validate evidence: Auditors require evidence that a control is effective. A scan result is evidence, but it does not prove that a user actually uses a password or that a system value is enforced at the OS level.
• Assess non-technical controls: The CIS benchmark includes items like “document the change management process” or “have a security policy.” No scanner can verify a policy document exists and is followed.

The most common misconception is that a passing scan means you are compliant. It does not. It means your technical configuration matches the benchmark’s recommendations at that moment. Compliance is a broader state that includes processes, people, and ongoing monitoring.

How Automation Tools Work on IBM i

Typical tools run as a native IBM i program or as a service that connects via ODBC/JDBC. They execute CL commands or SQL queries against system catalogs (e.g., QSYS2.SYSTEM_VALUE_INFO, QSYS2.USER_INFO). The tool then applies a rules engine that maps each benchmark item to a query. For example, the check for QMAXSIGN (maximum sign-on attempts) would query the system value and compare it to the benchmark’s recommended value (often 3). The output is stored in a database table or a spreadsheet.

Some tools also offer a “remediation script” that generates the CL commands needed to fix a finding. This is helpful, but it is not the same as applying them. You still need change control and testing. The best practice is to run the scan in a test partition first, review the generated commands, and then apply them manually after approval.

Choosing the Right Automation Approach

You have three main options, each with trade-offs:

• Commercial compliance tools: These are purpose-built for IBM i security auditing (e.g., from vendors like HelpSystems, now Fortra, or Raz-Lee). They offer pre-built CIS checks, dashboards, and scheduling. Cost is higher, but they save the most time and typically handle the benchmark’s nuances.
• Scripting with CL/SQL: You can write your own scripts using RUNSQL or CL programs to query system catalogs and compare values. This is free but requires deep IBM i knowledge and ongoing maintenance as the benchmark updates.
• Manual audit with a spreadsheet: This is the baseline. It is slow and error-prone, but it forces a human to look at every item. For a single partition with few users, it might be acceptable.

For most organizations, a commercial tool is worth it if you have more than one partition or need to audit quarterly. If you have a single system and a skilled admin, scripting can work. The key is to ensure the tool you choose can export raw data, because auditors will want to see the evidence, not just a summary.

Common Problems and Pitfalls in Automation

Even with good tools, several issues trip up teams:

• False positives due to IBM i versions: The benchmark is written for a specific OS release (e.g., V7R3, V7R4). If you run an older or newer version, some checks may not apply or may report incorrectly.
• Authority checks on objects: The benchmark checks *PUBLIC authority on critical objects. Automation can list these, but it cannot tell you if a specific user actually uses that authority. An object might have *PUBLIC *USE, which is often acceptable, but the scanner flags it as a finding.
• Performance impact: Running a full scan on a busy production partition can consume CPU and lock tables. Schedule scans during maintenance windows.
• Ignoring exit programs: The benchmark recommends exit programs for FTP, ODBC, etc. Automation can check if an exit program is registered, but it cannot verify that the exit program actually enforces the intended policy.

The biggest pitfall is treating the scan as a one-time event. CIS updates the benchmark periodically, and your system changes constantly. Automation only helps if you run it on a schedule and review the diffs.

Comparing Automation with Manual Audits

Manual audits have one advantage: a human can ask “why” and catch context that a script cannot. For example, a user profile with *ALLOBJ might be flagged by automation, but a manual audit might discover that this user is the only one who can perform nightly backups, and the risk is accepted. Automation would still flag it, and you would have to document the exception.

However, manual audits are impractical for large systems. A typical IBM i with 200 user profiles and 50 system values would take a skilled admin 8-10 hours to check manually. Automation does it in 15 minutes. The trade-off is that automation gives you a list of findings, not a risk assessment. You still need a human to triage the findings, assign severity, and decide what to fix.

The best approach is hybrid: use automation to collect and flag, then use a human to review the exceptions and document them. Most compliance frameworks actually require this—they want evidence of a review process, not just a scan report.

Costs and Effort of Implementing Automation

Costs vary widely. Commercial tools can cost thousands of dollars per year per partition, plus implementation services. Open-source or in-house scripts cost only your time, but that time can be significant—writing a robust script that handles all benchmark items might take weeks. There is also the cost of maintaining the tool as the benchmark updates (CIS releases new versions roughly every 1-2 years).

Beyond the tool itself, you need to allocate time for:

• Initial setup and baseline scan
• Reviewing findings and creating exception documentation
• Remediating high-severity items (which may require testing)
• Training staff to interpret the reports

Do not underestimate the review time. A scan might produce 200 findings, but only 10 require action. Someone has to go through each one and decide. That is where the real cost lies, and automation does not eliminate it.

Safety and Accuracy Considerations

When running automation, never let it change system values or authorities without explicit human approval. A misapplied change could lock out users or break an application. Always run scans in read-only mode first. Also, be aware that some CIS checks require interpreting the output of commands like DSPAUTUSR, which can be complex. A tool that simply greps for a string might miss a subtlety, leading to a false pass. Verify a sample of the tool’s results manually before relying on it.

Finally, remember that the CIS benchmark is a starting point, not an end. Your organization’s security policy may be stricter or more lenient. Automation can help you meet the benchmark, but true compliance is about managing risk, not just checking boxes.

Frequently Asked Questions

Q: Can I use the CIS benchmark as proof of compliance for PCI-DSS or SOX?
A: No. The CIS benchmark is a security configuration guide, not a compliance standard. Auditors may accept it as evidence of due diligence, but you still need to map each CIS control to the specific requirements of PCI-DSS, SOX, or other frameworks.

Q: Does automation work on all IBM i versions?
A: Most tools support recent versions (V7R2 and later), but some checks are version-specific. Always verify that the tool’s benchmark version matches your OS release. Older versions may have different system value names or missing catalogs.

Q: How often should I run the CIS scan?
A: At least quarterly, or after any significant change (new users, new applications, OS upgrades). For high-security environments, monthly is common. The scan itself is cheap; the review is what takes time.

Q: What if a finding is a false positive?
A: Document it. Most compliance frameworks allow exceptions if you can justify them. Create a formal exception register that lists the finding, why it is acceptable, and who approved it. Automation tools often have a feature to mark findings as “accepted risk.”

Q: Can automation fix the issues it finds?
A: Some tools generate remediation scripts, but they do not apply them automatically. You must review, test, and run the commands manually. Even then, some fixes (like changing QSECURITY) require a restart and can cause downtime.

Q: Is there a free way to automate CIS checks on IBM i?
A: Yes, you can write SQL queries against system catalogs and compare values. For example, you can query QSYS2.SYSTEM_VALUE_INFO to get all system values and then filter for those that do not match the benchmark. This is free but requires programming knowledge and time to maintain.

Related

• Diablo 3 Unexpected Error on Startup: Fixes
• Spring Boot Automation APIs: Key Design Decisions for Reliability
• How to Fix 502 Bad Gateway on Nginx with WordPress