On-Call Runbook for AI Agents: Build One

This guide walks you through creating an on-call runbook specifically for production AI agents—systems that make decisions, generate content, or interact with users in real time. You’ll define what to monitor, how to respond to failures, and how to escalate. By the end, you’ll have a practical, testable document your team can use during incidents.

What You’ll Need

• Access to your AI agent’s deployment environment (e.g., Kubernetes, cloud VM, or serverless platform).
• Monitoring and logging tools (e.g., Prometheus, Grafana, Datadog, or cloud-native logs).
• An incident management process (e.g., PagerDuty, Opsgenie, or a simple rotation schedule).
• Documentation of your agent’s architecture, including dependencies (LLM APIs, vector DBs, etc.).
• Contact information for the team and any external vendors.

Step-by-Step

1. Define the agent’s critical functions. List what your AI agent does in production: e.g., answering user queries, generating summaries, making recommendations. Prioritize these functions by business impact. This helps you focus monitoring and response on what matters most.
2. Identify failure modes. For each function, list possible failures: model API timeouts, invalid outputs, excessive latency, high error rates, data drift, or security anomalies. Be specific—e.g., “LLM returns empty response” vs. “LLM returns offensive content.”
3. Set up monitoring and alerts. For each failure mode, define metrics that indicate a problem. Common ones: request latency, error rate, token usage, cost per request, and output validation scores. Configure alerts with thresholds. For example, alert if error rate exceeds 5% for 5 minutes. Ensure alerts are actionable—include a link to the runbook.
4. Create a response procedure for each failure. For each failure mode, write a step-by-step response. Start with immediate mitigation (e.g., rollback to previous model version, switch to fallback model, or disable a feature). Then list diagnostic steps (e.g., check logs, query metrics, inspect recent inputs). Finally, define a resolution plan (e.g., retrain model, update prompt, fix code).
5. Define escalation paths. Specify who to contact if the issue isn’t resolved within X minutes. Include the on-call engineer, the team lead, and any external support (e.g., LLM provider). Provide contact methods (phone, Slack, email).
6. Document rollback and recovery steps. Clearly state how to revert to a previous version of the agent, how to restore from a backup, and how to resume service. Include commands or UI paths. Test these steps regularly.
7. Include communication guidelines. Specify how to communicate during an incident: what to post in the incident channel, who updates stakeholders, and how to handle user-facing status pages. For AI agents, also note if you need to inform users about degraded service or incorrect outputs.
8. Add a post-incident review template. After an incident, the runbook should guide the team through a blameless review: what happened, what was done, what worked, what didn’t, and action items. Include a section to update the runbook based on findings.
9. Test and maintain the runbook. Run tabletop exercises or chaos experiments to validate procedures. Update the runbook whenever the agent’s architecture, dependencies, or failure modes change. Set a review cadence (e.g., quarterly).

Common Mistakes

• Not distinguishing between model failure and infrastructure failure. A timeout might be a network issue, not an LLM problem. Make sure your runbook covers both.
• Over-alerting. Too many alerts cause alert fatigue. Only alert on symptoms that require human action.
• Ignoring data drift. AI agents can degrade silently as input data changes. Include monitoring for distribution shifts in your runbook.
• Assuming the on-call engineer knows the codebase. Write steps for someone who may be new. Include exact commands, file paths, and links.
• Forgetting about cost and quota limits. AI APIs have costs and rate limits. Include steps to check usage and handle quota exhaustion.

Tips

• Keep the runbook concise—use bullet points and numbered steps. Avoid long paragraphs.
• Use a wiki or version-controlled document so changes are tracked.
• Include a “quick reference” section with key commands, URLs, and credentials (stored securely).
• Make the runbook accessible offline or in a mobile-friendly format, as incidents can happen anywhere.
• Automate what you can: e.g., a script that captures logs and metrics when an alert fires.
• Involve the whole team in writing and reviewing the runbook—different perspectives catch blind spots.

Troubleshooting

If your runbook isn’t working as intended, consider these issues:

• Alerts not firing: Check alert thresholds, metric names, and query syntax. Ensure the monitoring agent has access to the right data sources.
• Steps are outdated: If a procedure references a service that no longer exists, update it immediately. Schedule regular reviews.
• On-call not responding: Verify escalation contacts are current. Test the alert-to-escalation flow with a dummy incident.
• Runbook too long: If it takes more than 10 minutes to find the relevant section, restructure with a clear index and searchable headings.
• Incident still unresolved after following steps: Escalate to a higher tier or the vendor. Document what was tried so you don’t repeat steps.

Frequently Asked Questions

How often should I update the runbook?

At least quarterly, or whenever you change the agent’s model, infrastructure, or dependencies. Also update after every incident to capture lessons learned.

What metrics are most important for AI agents?

Latency, error rate, token usage, cost per request, and output validation scores. Also monitor input distribution to detect drift.

Should I include a rollback plan for the model itself?

Yes. If you deploy a new model version, you should know how to revert to the previous one. This often involves swapping a model ID or container image.

How do I handle a situation where the AI produces harmful content?

Immediately disable the feature or route traffic to a fallback. Then investigate the cause—prompt injection, training data issue, or missing filters—and update your safeguards.

Do I need a separate runbook for each AI agent?

If agents have different architectures or failure modes, yes. But you can share common sections like escalation paths and communication guidelines.

What if my team is small and doesn’t have a dedicated on-call rotation?

Start with a simple rotation—one person per week. The runbook makes it feasible by providing clear steps, so even a less-experienced team member can respond.

Related

• Motherboard Battery: Can It Ruin Your PC’s Performance?
• Starlink India: Equipment & Installation Needs
• Synology Glacier Backup: How It Works