keepcalculator

06 Numbers and Powers

Agent Failure Your Approval Gate Can’t Catch

Card No. 06·79331 · Updated · Standard formula, shown below

This article explains why a traditional approval gate in an AI agent workflow can fail to catch faulty outputs, and what you can do about it. You’ll learn the specific failure modes that slip through, how to build a more robust review layer, and when to use which safeguard.

Why Approval Gates Fail

Approval gates are the most common control mechanism for AI agents: the agent produces a result, a rule checks it (e.g., “is the output valid JSON?” or “does it contain a required field?”), and if the check passes, the action proceeds. This works for deterministic errors—missing fields, wrong types, out-of-range values. But it fails for semantic errors, where the output is structurally perfect but factually or logically wrong.

The core issue is that gates check form, not meaning. A gate can verify that a generated email includes a subject line, but it cannot verify that the subject line matches the email body. It can check that a SQL query is syntactically valid, but not that it queries the correct table. These are the failures that slip through.

Common Failure Modes

  • Hallucinated facts: The agent invents a statistic, a date, or a name that looks plausible. The gate sees a well-formed sentence and passes it.
  • Context drift: The agent starts following the user’s instruction, then gradually shifts to a different interpretation. The final output is internally consistent but not what was asked.
  • False confidence: The agent’s own confidence score is unreliable. It may be 95% confident on a task it has no real knowledge of.
  • Multi-step compounding: A small error in step 2 becomes a large error in step 7. The gate only sees the final output, which looks fine.

How to Build a Better Gate

Instead of a single rule-based check, use a layered approach:

  1. Deterministic checks: Validate types, ranges, required fields, and format. This catches the easy stuff.
  2. Round-trip verification: If the agent produces a summary, feed the summary back and ask it to regenerate the original. If the two don’t match, flag it.
  3. Cross-check with a second model: Use a different model (or a different prompt) to independently verify the output. If they disagree, send to a human.
  4. Grounding against source data: If the agent cites a number, have it also output the source snippet. The gate can check that the snippet actually contains the number.
  5. Human review for high-risk actions: For anything that costs money, deletes data, or sends an external message, require a human click—not just a confidence score.

The key is to design the gate around what could go wrong, not just what the output looks like.

Real-World Example: The Email Incident

Consider an agent that drafts and sends customer emails. A simple gate checks that the email has a recipient, a subject, and a body. It passes. But the agent was told to apologize for a billing error, and instead it wrote “We hope you enjoy the extra charge.” The gate cannot catch this because it doesn’t understand the meaning.

A better gate would include a sentiment check (is the tone apologetic?), a keyword check (does it mention “billing error”?), and a round-trip test (ask the agent: “What is the purpose of this email?” and compare). Even then, a human should review any email that offers a refund or a discount.

When to Use a Human-in-the-Loop

Not every task needs a human. Use these guidelines:

  • Low risk, low complexity: Let the gate pass automatically (e.g., formatting a date, generating a placeholder).
  • Medium risk, medium complexity: Use the layered checks above, and if any fail, send to human.
  • High risk or high complexity: Always require human approval, regardless of confidence scores.

Humans are slow and expensive, but so is a marketing email that says the opposite of what you meant. The cost of a human review is usually far less than the cost of a bad automated action.

Limits of Any Gate

No gate is perfect. Even a human reviewer will miss things. The goal is not to catch every error—it’s to catch the ones that matter. That means you should prioritize:

  • Impact: A 1% error rate on a million-dollar transaction is worse than a 10% error rate on a joke generator.
  • Reversibility: If an action can be undone easily (like a draft email), you need less scrutiny than if it’s irreversible (like a wire transfer).
  • Speed: A gate that takes 10 seconds per task is fine for a few tasks, but not for a batch of 10,000.

Use it to decide how much review effort to invest, not to decide whether an action is safe.

Frequently Asked Questions

Can I use a confidence score from the agent as a gate?

Only as a weak signal. Many agents are poorly calibrated—they say 90% confident when they’re actually 50% right. Use confidence as one input among many, not as the sole gate.

What is the best way to verify an agent’s factual claims?

Require the agent to output a source for every factual claim, then programmatically check that the source exists and contains the claim. If the source is a web page, fetch it and search for the key phrase. If it’s a database, run a query.

How do I handle multi-step tasks where errors compound?

Add intermediate gates after each step, not just at the end. Check that each intermediate output is reasonable before feeding it to the next step. This is more work, but it prevents small errors from becoming large ones.

Should I use a different model for verification?

Yes, if you can afford it. A second model with a different training set or architecture is more likely to catch errors that the first model systematically makes. Even a smaller, cheaper model can work as a verifier if you give it a clear checklist.

What if the agent refuses to follow the gate rules?

That’s a sign of a deeper problem. The agent’s system prompt should state that gate checks are mandatory and cannot be bypassed. If the agent is still ignoring them, you may need to enforce the gate at the infrastructure level, not the prompt level.

Is there a way to make the gate itself smarter over time?

Yes. Log every gate failure and what the human reviewer changed. Use that data to update the gate rules—for example, if you notice that certain keywords always lead to bad outputs, add a check for those keywords. This turns your gate into a learning system.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →