keepcalculator

06 Numbers and Powers

How to Evaluate AI Agents on Cooperative Games Like Overcooked

Card No. 06·82077 · Updated · Standard formula, shown below

This guide explains how to set up and run evaluations of AI agents in cooperative games such as Overcooked. You will learn to define evaluation metrics, create reproducible scenarios, run agent trials, and analyze results. You need a working game environment (e.g., the Overcooked-AI gym), an agent implementation, and basic Python knowledge.

What You’ll Need

  • A cooperative game environment with an API (e.g., Overcooked-AI, Gymnasium-compatible).
  • Python 3.8+ and the environment’s dependencies installed.
  • Your AI agent code (or a baseline like random or scripted agent).
  • A way to record metrics (e.g., CSV, JSON, or a logging library).
  • Sufficient compute (CPU is often enough for small layouts).

Step-by-Step

  1. Define the evaluation question. Decide what you want to measure: task success (e.g., number of dishes served), efficiency (steps per dish), or coordination (e.g., collisions or idle time). Write down the primary metric and any secondary metrics.
  2. Select a set of layouts. Choose 3-5 maps that vary in difficulty (e.g., from the Overcooked-AI layouts like “cramped_room” to “asymmetric_advantages”). Include at least one layout with asymmetric roles to test coordination.
  3. Fix the environment configuration. Set a fixed random seed for each layout. Use the same time limit (e.g., 400 steps) and the same reward function for all agents. Record these settings.
  4. Implement your agent and at least one baseline. A random agent that takes random valid actions is essential. A scripted agent (e.g., one that always moves toward the nearest onion) is useful. Your agent should expose a act(obs) method.
  5. Write the evaluation loop. For each layout and agent, run multiple episodes (e.g., 10-20) with different seeds. For each episode, reset the environment, then loop until done: get observations, let both agents act (your agent and a fixed partner), step the environment, and accumulate rewards and metrics.
  6. Record per-episode metrics. Store the total reward, number of dishes served, steps taken, and any coordination metrics (e.g., number of times agents collide). Save to a structured file with columns: layout, agent, seed, metric, value.
  7. Run the evaluation. Execute the loop for all combinations. This may take hours; use parallelization if possible (e.g., multiprocessing). Monitor progress and log any errors.
  8. Analyze the results. Compute the mean and standard deviation of each metric per agent per layout. Use a table or plot to compare. Check for statistical significance using a paired t-test or bootstrap if you have enough episodes.
  9. Report the results. Present the primary metric (e.g., average dishes served) as the headline, with secondary metrics as supporting. Include the exact environment settings and seeds so others can reproduce.

Common Mistakes

  • Using different reward functions for different agents—this invalidates comparisons.
  • Not fixing seeds, leading to high variance and unreproducible results.
  • Running too few episodes (e.g., 1-2) which makes results noisy.
  • Ignoring the partner agent: your agent’s performance depends on the partner. Always specify the partner type (e.g., scripted, human-like, self-play).
  • Measuring only total reward, which can mask coordination failures (e.g., high reward but many collisions).
  • Overfitting to one layout—evaluation should use multiple layouts.

Tips

  • Start with a small layout and a random agent to sanity-check your loop.
  • Use a fixed time limit and count steps; otherwise, agents that stall can inflate reward.
  • Record the full episode trajectory (actions, observations) for debugging, but store only summaries for analysis.
  • Consider using multiple seeds per layout (e.g., 5) and report the mean and standard error.
  • If your agent is trained with self-play, evaluate against a fixed partner to isolate its skill.
  • Visualize a few episodes (record videos) to qualitatively assess coordination.

Troubleshooting

  • Environment crashes or hangs: Check that the environment version matches the API. Wrap the step call in a try-except to catch errors. If it hangs, add a timeout per episode.
  • Agent returns invalid actions: Ensure your agent only returns actions from the action space. Clip or mask invalid actions.
  • Results vary wildly across runs: Increase the number of episodes or seeds. Check that you are resetting the environment correctly between episodes.
  • Reward is always zero: Verify that the reward function is correctly wired. Print rewards during a random-agent episode to see if any are positive.
  • Memory issues: If storing trajectories, reduce the number of stored episodes or use a streaming logger.
  • Agent is too slow: Profile your agent’s inference. Use a simpler model or reduce the observation space (e.g., downscale the grid).

Frequently Asked Questions

What is the best metric for Overcooked?

Number of dishes served is the most direct measure of task success. Total reward is also common but can be noisy; report both.

How many episodes should I run?

At least 10 per layout per agent, with 5 different seeds. More if the variance is high. For statistical significance, aim for 20+.

Should I evaluate against a fixed partner or self-play?

It depends on your goal. Fixed partner (e.g., scripted) tests your agent’s ability to adapt to a known style. Self-play tests coordination but can be expensive. Use both if possible.

Can I use human players in evaluation?

Yes, but it’s time-consuming and less reproducible. For controlled experiments, stick to scripted partners.

How do I handle different layouts?

Report results per layout, not just aggregated. A layout like “asymmetric_advantages” requires different coordination than “cramped_room”.

What if my agent uses a different observation space?

Standardize the observation space for all agents. If your agent needs custom features, wrap the environment to provide them, but keep the underlying game state identical.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →