Emergency OT Changes: Keep Security Visibility

When an emergency forces a last-minute change to your operational technology (OT) environment—a failed sensor, a new IP address, a temporary bypass—security monitoring often takes a back seat. This guide shows you how to make that change without losing sight of what matters: knowing exactly what is on your network, who is talking to whom, and whether anything unusual is happening. You’ll learn a repeatable process that keeps your security team informed, your logs intact, and your incident response ready—even when the clock is ticking.

What You’ll Need

• Access to your OT asset inventory (even a spreadsheet works)
• A change management or ticketing system (or a shared document)
• Network visibility tools: firewall logs, switch logs, or a passive monitoring solution
• A communication channel to reach the security team (chat, email, phone)
• Approval from the operations lead (or a pre-agreed emergency protocol)
• A rollback plan for the change

Step-by-Step

1. Declare the emergency and pause normal change review. The first step is to formally recognize that a change must happen immediately. This means you skip the normal Change Advisory Board (CAB) meeting, but you do not skip documentation. Create a ticket or record that says “EMERGENCY” and includes the reason, the system affected, and the person who authorized it. This record is your anchor for later review.
2. Identify the exact scope of the change. Write down what is changing: a new device IP, a firmware update, a firewall rule, a physical cable swap. Be specific. If you are adding a new device, know its MAC address, make, model, and purpose. If you are changing a rule, know the source and destination IPs and ports. This scope becomes the basis for your security monitoring.
3. Update your asset inventory immediately. Even before you make the change, add a placeholder entry in your inventory. Include the new device or the modified configuration, the date/time, and the ticket number. This is the single most important step for maintaining visibility—if you don’t record it now, you will forget later.
4. Adjust security monitoring rules to cover the change. If you have a SIEM or a network monitoring tool, add temporary rules or exceptions to ensure the new traffic is not flagged as anomalous. For example, if you are adding a new PLC at 10.10.5.50, create a rule that recognizes that IP as a known asset. If you are opening a port for remote access, log that exception and set an expiry date. This prevents alert fatigue while still capturing the traffic.
5. Make the change, but keep logs on. Perform the physical or logical change. Ensure that logging is not disabled on the affected devices—this is a common emergency shortcut that destroys visibility. If you must reboot a device, verify that its logs are set to persist. If possible, enable extra logging on the switch port or firewall during the change window.
6. Verify connectivity and security posture. After the change, confirm that the system is working as intended. Then, check that your security tools see the change: can the SIEM see the new device? Are logs flowing? Do a quick packet capture or check the ARP table to confirm the device is present and communicating. If you have a passive monitor, verify it shows the new traffic.
7. Communicate the change to the security team. Send a message to the security team with the ticket number, the scope, and the time of the change. Ask them to acknowledge that they are aware. This is not about approval—it is about ensuring the people watching the alerts know that this change is expected.
8. Set a follow-up review date. Emergency changes should become permanent or be rolled back within a defined period. Schedule a review within 48 hours to either formalize the change through the normal process or reverse it. Until then, keep the temporary monitoring rules active.
9. Document the outcome. Once the emergency is over, close the loop. Update the asset inventory with final details, note any security events that occurred during the window, and record what worked and what didn’t. This becomes your playbook for the next emergency.

Common Mistakes

• Skipping the inventory update. The most common failure is assuming you’ll remember the new device later. You won’t. Update it in real time.
• Disabling logging to “speed things up.” Turning off logs or alerting to avoid noise means you lose the only evidence you have. Keep them on, even if you have to filter later.
• Not telling the security team. If the SOC doesn’t know about the change, they will treat it as an anomaly and may block or investigate it, causing more delays.
• Forgetting to set an expiry on temporary rules. A temporary firewall rule that stays forever becomes a permanent vulnerability. Always set an end date.
• Making the change without a rollback plan. If the change breaks something, you need to know exactly how to undo it. Write it down before you start.

Tips

• Use a pre-approved emergency change template. Have a form ready that includes all the fields you need—scope, reason, authorization, rollback plan. This saves time and ensures consistency.
• Leverage read-only access. If you have a network tap or a read-only account on the switch, use it to monitor the change without affecting operations.
• Take screenshots. Before and after the change, capture the device’s configuration or the network diagram. This helps in the post-incident review.
• Involve security early. Even if they can’t approve, they can advise on what to watch for. A quick call can save hours of cleanup later.
• Automate where possible. If your inventory is in a CMDB, use an API to add the device automatically. If you have a SIEM, create a script to add the exception rule.

Troubleshooting

If the security team is not responding: Leave a voicemail and send a message with the subject “EMERGENCY CHANGE IN PROGRESS.” If they still don’t respond, proceed with the change but document that you attempted contact. After the change, send a detailed summary so they can catch up.

If the new device isn’t showing up in monitoring: Check the switch port for link status, verify the device’s IP and subnet, and confirm that your monitoring tool’s scope includes that VLAN or network segment. If you use DHCP, check the DHCP lease table. If the device is on a different segment, you may need to add a new monitoring rule.

If alerts are flooding in despite your temporary rules: Your exception rule may not match the exact traffic pattern. Check the source and destination ports and protocols. Adjust the rule to match the actual traffic. If alerts are still too noisy, you can temporarily mute the specific alert ID, but only for the duration of the emergency.

If you cannot update the inventory because the system is down: Use a local spreadsheet or even a piece of paper. The key is to record the information somewhere that will be transferred to the official system later. Do not skip this step.

If the change fails and you need to roll back: Follow your rollback plan. If you don’t have one, reverse the exact steps you took. After rollback, verify that the system is back to its previous state and that monitoring is still seeing the original configuration. Then document why the change failed.

Frequently Asked Questions

What counts as an “emergency” OT change?

An emergency change is one that must be implemented immediately to prevent or mitigate a safety incident, production loss, or security breach. Examples include a failing pump that needs a new controller, a critical vulnerability that requires an immediate patch, or a network outage that demands a temporary bypass. The key is that waiting for normal approval would cause harm.

Can I skip the security team if the change is urgent?

No. You should always inform the security team, even if you cannot wait for their approval. They need to know about the change to avoid misinterpreting the resulting traffic as an attack. A quick message takes seconds and can prevent a full-blown incident response.

How do I handle a change that requires disabling a security control, like a firewall?

If you must disable a security control, do it for the shortest possible time and only for the specific traffic that needs to pass. Document the exact time it was disabled and re-enabled. After the change, review the logs for that window to ensure no unauthorized activity occurred. This is a high-risk action—get the security team’s explicit acknowledgment before doing it.

What if I don’t have a formal asset inventory?

Start one now, even if it’s a simple spreadsheet. For the emergency, record the device’s IP, MAC, serial number, location, and purpose. That information is critical for both security and troubleshooting. After the emergency, make building a proper inventory a priority.

How long should temporary security exceptions last?

Set an expiry date that aligns with the review date—typically 24 to 72 hours. If the change becomes permanent, you should go through the normal change process to make it official. If it is truly temporary, the exception should be removed when the emergency is resolved.

What should I do after the emergency is over?

Conduct a post-incident review. Compare the logs from the emergency window with the expected traffic. Update your asset inventory and monitoring rules. Identify what went well and what didn’t, and update your emergency change template accordingly. This turns a crisis into a learning opportunity.

Related

• Location Not Available Access Denied: Fixes
• Content Deleted Part 3: What It Means & How to Recover
• Do You Need a 120Hz TV for 60 FPS Gaming?