Benchmark AI Coding Agents on Large Python Codebases

This guide explains how to set up a repeatable benchmark for AI coding agents working on large Python codebases. You will learn to define tasks, measure success, and interpret results so you can compare agents fairly and track improvements.

What You’ll Need

A large Python codebase (at least 50k lines, ideally with multiple modules and tests).
A clean environment (Docker container or VM) to isolate each run.
Git for version control and to create task branches.
Python 3.9+ and pip for installing test dependencies.
Access to the AI coding agents you want to benchmark (CLI, API, or IDE plugin).
A timer or logging script to record time and resource usage.
A test runner (pytest is recommended) to verify correctness.
A scoring spreadsheet or script to aggregate results.

Step-by-Step

Select a representative codebase. Choose a project that is large enough to require navigation, has a clear structure, and includes tests. Avoid codebases with external services or heavy setup that could slow down the benchmark. If possible, use a well-known open-source project like Django or a smaller but still substantial library.
Define a set of tasks. Create 10–20 tasks that reflect real development work: fixing a bug, adding a small feature, refactoring a function, writing a unit test, updating documentation, or optimizing a hot path. Each task should have a clear acceptance criterion (e.g., “all tests pass” or “the function returns the correct output for edge cases”).
Create a clean baseline. Clone the repository at a specific commit. Run the full test suite to confirm it passes. Record the baseline test count and any known failures. This ensures that any changes in test results are due to the agent’s work, not pre-existing issues.
Set up an isolated environment. Use Docker or a virtual machine to create a fresh environment for each run. Install the project’s dependencies, the test runner, and any tools the agent might need (e.g., linters, type checkers). Do not include the agent’s configuration in the baseline image—keep it separate so you can swap agents easily.
Prepare the agent’s workspace. For each task, create a new branch from the baseline commit. Give the agent access to the repo, the task description, and any necessary credentials (if the task requires pulling from a private registry). Ensure the agent’s output is restricted to the workspace—no network access to external AI services unless that is part of the test.
Run the benchmark. For each task, start the agent with a fresh context (clear any conversation history). Record the start time, then let the agent work until it signals completion or a timeout (e.g., 30 minutes per task). Save the agent’s output (code changes, messages, and any files created).
Evaluate the results. After the agent finishes, run the test suite on the modified branch. Count how many tests pass, fail, or error. Compare the diff to the baseline to see if the agent changed files outside the task scope. Also check for code style issues (e.g., PEP 8 violations) if that is part of your criteria.
Score each task. Assign a score based on: (1) correctness (tests pass), (2) completeness (the task is fully addressed), (3) efficiency (time taken), and (4) code quality (readability, no dead code). Use a weighted formula, e.g., 50% correctness, 25% completeness, 15% efficiency, 10% quality. Record the score for each task.
Repeat across multiple runs. Run the entire benchmark at least three times per agent to account for variability. Use different random seeds or shuffle task order to avoid learning effects. Take the average score and standard deviation.
Analyze and report. Aggregate the scores by task type (bug fix vs. feature) and by complexity (simple vs. hard). Create a report that shows the mean score, median time, and pass/fail rates. Identify patterns—e.g., an agent might excel at refactoring but struggle with adding new features.

Common Mistakes

Not isolating the environment: If the agent can access the internet or pre-existing caches, results become inconsistent. Always use a clean container.
Vague task descriptions: If the task is ambiguous, the agent may produce a plausible but wrong solution. Write acceptance criteria that are testable.
Ignoring baseline flakiness: Some tests are flaky by nature. Run the baseline multiple times to identify flaky tests and exclude them from evaluation.
Only measuring correctness: A solution that passes tests but takes 20 minutes and adds 500 lines of dead code is not a good solution. Include time and code quality in your scoring.
Not resetting the agent’s context: If you reuse the same agent session across tasks, it may “remember” previous tasks, which is unrealistic. Start fresh for each task.
Overfitting to one codebase: Benchmarking on a single project may not generalize. If you have time, use two different codebases (e.g., a web framework and a data-processing library).

Tips

Use a harness script: Automate the entire process with a Python script that creates the environment, launches the agent, and collects results. This reduces human error and makes the benchmark reproducible.
Include a “warm-up” task: Before the actual timed tasks, give the agent a simple task to get familiar with the codebase. Do not include this in the scoring.
Record the agent’s reasoning: If the agent outputs a log or chain-of-thought, save it. This helps you understand why a task failed.
Use a timeout: Set a reasonable timeout per task (e.g., 30 minutes). If the agent exceeds it, mark the task as failed for time.
Check for unintended changes: Use git diff to see if the agent modified files unrelated to the task. This is a sign of poor scoping.
Version everything: Record the exact version of the agent, the codebase commit, and the dependencies. This allows you to rerun the benchmark later.

Troubleshooting

The agent fails to start: Check that the agent’s API key or CLI path is correctly configured in the environment. Ensure the workspace directory is writable and has enough disk space.
Tests fail due to missing dependencies: Some tasks may require additional packages that are not in the baseline. Add them to the Docker image before running, but document them so you can reproduce.
The agent times out on every task: The codebase might be too large for the agent to navigate. Try reducing the scope by providing a file tree or a list of relevant files in the task description.
Results vary wildly between runs: This is normal for AI agents. Increase the number of runs (e.g., 5 instead of 3) and use the median as the primary metric.
The agent produces syntax errors: This often happens with large files. Check if the agent is truncating output. If so, increase the output token limit if the agent allows it.
The baseline tests fail after the agent runs: The agent may have modified a shared file (e.g., conftest.py). Use git diff to see what changed and revert if necessary.

Frequently Asked Questions

How many tasks should I include?

At least 10 tasks to get statistically meaningful results. If you have time, 20 is better. Make sure the tasks vary in difficulty and type.

Can I benchmark agents on a private codebase?

Yes, but ensure the environment is isolated and that you have the right to use the code. For reproducibility, consider using a public codebase so others can replicate your results.

Should I use a single agent or multiple?

Benchmark each agent separately. If you want to compare agents, run the same tasks on each and compare scores.

How do I handle tasks that require external APIs?

If the task needs an API key, provide a mock or a sandbox. Avoid using real external services that could be rate-limited or change behavior.

What if the agent asks for clarification?

Allow a limited number of clarifications (e.g., 3 per task). If the agent asks more, mark the task as incomplete. This mimics real-world constraints.

How long should the benchmark take?

For 10 tasks with a 30-minute timeout, expect 5–6 hours per agent run (including setup). Multiply by the number of runs and agents.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →