Why AI Agents’ 77% Pass Rate Drops to 53% on Repeated Tasks

AI agents that score 77% on a benchmark can fall to 53% when asked to repeat the same task because of a combination of non-deterministic model sampling, context-window degradation, and subtle changes in the evaluation environment. This drop is not a sign the agent is broken; it reflects how current large language models (LLMs) and their surrounding tooling behave under repeated execution. Below we break down the mechanics, the common causes, and what you can do to stabilize performance.

The Non-Deterministic Nature of LLMs

Most AI agents are built on large language models that generate text by sampling from a probability distribution over possible next tokens. Even with a fixed temperature setting (often 0.2 or 0.7), the sampling process is stochastic. On the same prompt, the model can produce slightly different completions each time. When the task is a multi-step agentic workflow—such as browsing a website, extracting data, and filling a form—a single token difference early on can cascade into a completely different path. A 77% pass rate means that on 77% of runs the agent succeeded; the other 23% failed for various reasons. When you run the same task repeatedly, you are sampling from that same distribution, so you will see a mix of successes and failures. The 53% figure simply reflects a particular run or a different test set where more of those stochastic failures occurred.

Context Window and Memory Contamination

Agents often maintain a conversation history or a memory buffer that grows as they perform steps. On repeated tasks, the agent may receive its own previous outputs as part of the context (for example, in a multi-turn evaluation). This can cause two problems:

• Context overflow: When the history exceeds the model’s context limit, the system truncates or summarizes older parts. Losing key details can break the agent’s understanding of the task.
• Self-contamination: If the agent sees a previous wrong answer or a partial failure, it might incorporate that into its reasoning, leading to a different (and often worse) outcome.

In a fresh session, the agent starts clean; in a repeated session, it might not. This is especially common in benchmarks that run multiple episodes in one conversation.

Evaluation Environment Variability

Benchmarks like WebArena, SWE-bench, or custom agent evaluations often involve live websites, APIs, or code repositories. These external factors can change between runs:

• Web pages may load differently, have dynamic content, or show different ads.
• APIs may rate-limit or return different data.
• Timeouts or network latency can cause the agent to miss a step.
• If the evaluation uses a simulated environment, the random seed might differ, altering the task layout.

Even a 0.5-second delay can cause an agent to click the wrong element. When you repeat a task, you are not repeating an identical environment; you are repeating it under slightly different conditions.

Agent Architecture and Tool-Use Errors

Modern agents use tools (web search, code execution, file manipulation) via function calling. The model must generate a structured call (e.g., JSON with the correct tool name and arguments). A small error in the JSON—like a missing quote—can cause the tool to fail. On repeated runs, the model might occasionally generate malformed calls, especially if the task is long and the model’s attention wanes. Also, agents often have a loop: they call a tool, observe the result, and decide the next step. If the observation is truncated or the model misinterprets it, the loop can go off track. These errors are probabilistic, so the pass rate varies.

Common Misconception: It’s Not Just ‘Randomness’

People often assume that setting temperature to 0 makes the model deterministic. In practice, many LLM providers still use non-deterministic sampling even at temperature 0 due to batching, floating-point arithmetic, or hardware differences. Also, the agent’s own code (e.g., using a random number generator for exploration) can introduce variability. So you cannot rely on temperature=0 to fix the drop.

How to Measure and Stabilize Performance

If you are evaluating an agent, run the same task multiple times (e.g., 10 or 20) and report the mean and variance, not a single pass/fail. To improve stability:

• Set a lower temperature (0.1 or 0) and use a fixed seed if your provider supports it.
• Clear the conversation history between runs to avoid contamination.
• Add explicit instructions to re-read the task description before acting.
• Implement retry logic for tool calls: if a call fails, let the agent try once more.
• Use a deterministic environment (e.g., a local sandbox) rather than live websites.

Remember that a single run’s score is a sample; the true performance is a distribution.

Frequently Asked Questions

Does a lower temperature always improve repeatability?

Not always. Temperature 0 reduces randomness but can lead to repetitive or stuck behavior. Some tasks benefit from a little exploration. Test with your specific agent and task.

Why did my agent pass the first time but fail the second?

Because the model’s output is sampled from a probability distribution. The first run got a lucky sample; the second got an unlucky one. Also, the environment may have changed slightly.

Can I make my agent 100% deterministic?

In practice, no. Even with temperature 0, there can be non-determinism from the serving infrastructure. You can reduce variability but not eliminate it.

Is a 77% pass rate good?

It depends on the task. For high-stakes tasks like medical diagnosis or financial transactions, 77% is not acceptable. For research or data extraction, it might be fine if you have a human review step.

How many runs should I do to get a reliable estimate?

At least 10, but 20 or more is better. Use the mean and standard deviation to report performance.

What is the best way to compare two agents?

Run both on the same set of tasks, with the same number of repetitions, and use a paired statistical test (like a t-test) on the pass rates.

Related

• AI Agent Memory: Stop Adding More Context?
• AI Meeting Assistants: Who Still Does the Work?
• AI Research Agents: Investigate Topics and Verify Sources