AI Evaluation Checklist: Reproducible Benchmarks

This checklist explains how to report AI benchmark results so that another researcher can reproduce them exactly. You need to document the model version, dataset version, sampling parameters, hardware, and evaluation code, and then publish the raw outputs alongside the summary scores.

Why Reproducibility Matters in AI Evaluation

AI models are stochastic—the same prompt can produce different outputs on different runs, even with identical settings. Without a strict reporting protocol, two teams can run the “same” benchmark and get very different numbers. Reproducibility is what separates a trustworthy benchmark from an anecdote. It also lets you compare your results against published baselines, detect regressions when you update a model, and audit whether a performance gain is real or just noise.

The core principle is simple: another person, with the same code and the same documented environment, should be able to run your evaluation and obtain the same or statistically equivalent results. This requires you to control and report every variable that can affect the output—from the random seed to the GPU driver version.

Core Components of a Reproducible Benchmark Report

A complete report should include the following elements. Each one is a potential source of irreproducibility if omitted.

• Model and tokenizer versions. Report the exact model identifier (e.g., the Hugging Face model card name) and the commit hash or version number. A model can change between releases without the name changing.
• Dataset version and preprocessing. Name the dataset and its version (e.g., “MMLU, test split, revision 2024-01-15”). Describe any filtering, deduplication, or prompt formatting you applied.
• Sampling parameters. List temperature, top-p, top-k, max tokens, frequency penalty, presence penalty, and any other generation settings. Even a small change in temperature can change the output.
• Random seed. If you use any sampling (temperature > 0), you must set and report the seed. For full determinism, set temperature to 0 if the API supports it, but note that some APIs still have nondeterminism.
• Hardware and software environment. GPU model, CPU, RAM, CUDA version, PyTorch or TensorFlow version, and the exact library versions (transformers, vLLM, etc.).
• Evaluation code. Share the exact script or notebook used to compute the metrics. Version-control it and link to the commit.
• Raw outputs. Publish the model’s complete outputs for every prompt, not just the aggregate score. This allows others to verify the scoring and to spot anomalies.
• Metric definition. State the exact formula (e.g., accuracy = correct / total) and any special handling of ties, empty outputs, or formatting errors.

How to Report Results: A Step-by-Step Protocol

Follow this protocol when you run a benchmark and want to publish the results. The goal is to leave nothing to guesswork.

1. Freeze the model and dataset. Download the model weights and the dataset, and record their checksums (e.g., SHA-256). Use the same files for all runs.
2. Set a fixed random seed. Choose a seed (e.g., 42) and use it for all sampling, data shuffling, and any other random operation. If you run multiple seeds, report each seed’s results separately.
3. Run the evaluation once with a small subset. Verify that the pipeline works, that the outputs look sensible, and that the metric calculation is correct.
4. Run the full evaluation. Execute the same script on the same hardware. If you run multiple times with different seeds, report the mean and standard deviation.
5. Save everything. Save the raw outputs, the configuration file, the environment list (e.g., pip freeze), and the evaluation script.
6. Write the report. In the methods section, include every item from the checklist above. In the results, show the aggregate score and, if possible, a breakdown by category or difficulty.
7. Publish the artifacts. Upload the code, the raw outputs, and the environment files to a public repository (e.g., GitHub, Zenodo) with a DOI.

Common Pitfalls and How to Avoid Them

Even experienced teams make these mistakes. Watch for them in your own work and in papers you read.

• Not reporting the exact prompt template. A small change like adding a newline or a “Question:” prefix can change accuracy by several points. Always include the full prompt template in an appendix.
• Using a different dataset split. Some datasets have multiple versions or splits. If you use the validation set instead of the test set, say so explicitly.
• Ignoring nondeterminism in GPU operations. Some operations (e.g., atomic adds) are not deterministic even with a fixed seed. If you need exact reproducibility, use deterministic algorithms (e.g., torch.use_deterministic_algorithms(True)) and report that you did.
• Rounding intermediate values. If you round the per-example scores before averaging, you introduce error. Keep full precision until the final metric.
• Not reporting the number of runs. A single run with temperature 0.5 is not enough. Report the mean and standard deviation over at least 3 runs with different seeds.
• Changing the evaluation code between runs. If you fix a bug in the scoring script after running some models, you must rerun all models with the fixed script.

How to Choose the Right Evaluation Settings

Your choices depend on what you are testing. For a standard benchmark like MMLU or HumanEval, use the official evaluation harness and its default settings. For a custom task, choose settings that match the intended use case.

• For classification tasks (e.g., MMLU): Use temperature 0 and greedy decoding (top-p=1). This is deterministic and gives the model’s most likely answer.
• For generative tasks (e.g., summarization): Use a small temperature (0.2–0.7) and report the seed. Use multiple seeds and report the average metric (e.g., ROUGE).
• For code generation (e.g., HumanEval): Use temperature 0.8 and pass@k metric, which requires sampling multiple times. Report the number of samples per problem.
• For safety or bias evaluations: Use a diverse set of prompts and report the full distribution of outputs, not just a single score.

Tools and Frameworks for Reproducible Evaluation

Several open-source tools help you standardize your evaluation. Using them is not mandatory, but they reduce the chance of error.

• Hugging Face Evaluate – a library for computing metrics consistently.
• lm-evaluation-harness – the standard harness for many public benchmarks (MMLU, HellaSwag, etc.). It handles dataset loading, prompt formatting, and metric computation.
• vLLM or TGI – serving frameworks that give you control over sampling parameters and often provide deterministic modes.
• Weights & Biases or MLflow – experiment tracking tools that log parameters, code, and results automatically.
• Docker – containerize your environment so that the exact software versions are preserved.

Comparing Results Across Papers and Models

When you compare your results with a published paper, check the following before trusting the numbers:

• Did they use the same dataset version and split?
• Did they use the same prompt template? Many papers do not include the full template, so you may need to contact the authors.
• Did they use the same metric? For example, “accuracy” can be computed with or without normalization of whitespace.
• Did they use the same number of samples for pass@k? The value of k matters.
• Did they report the standard deviation? If not, the difference between two models may be within noise.

If the paper does not provide enough detail, treat the comparison as approximate. The best practice is to run the benchmark yourself with your own setup and compare using the same code.

Frequently Asked Questions

What is the minimum information I must publish to call my results reproducible?

At minimum, publish the model version, dataset version, sampling parameters, random seed, and the evaluation code. Without these, another person cannot even attempt to reproduce your run. Ideally, also publish the raw outputs.

Should I always set temperature to 0 for reproducibility?

Temperature 0 makes the model greedy and often deterministic, but not always—some APIs have nondeterministic batching. For tasks that require creativity (e.g., story generation), temperature 0 may not be appropriate. In that case, use a fixed seed and report the variance across seeds.

How many runs do I need to report a reliable average?

For generative tasks, at least 3 runs with different seeds is a good start. If the variance is high, run more. For deterministic tasks (temperature 0), one run is enough if the environment is deterministic, but you should still verify by running twice.

What if I can’t share the model weights (e.g., proprietary model)?

You can still share the exact API endpoint, the date of access, and the model version. But note that the provider may update the model silently, so your results may not be reproducible later. In that case, state this limitation clearly.

How do I report results when the evaluation metric is not standard?

Define the metric precisely in the report, including edge cases. For example, if you use a custom F1 score, explain how you handle partial matches. Publish the scoring code so others can apply it to their own outputs.

What is the difference between a benchmark and an evaluation?

A benchmark is a standardized test (e.g., MMLU with a fixed set of questions and answers). An evaluation is the process of running a model on that test and measuring performance. Reproducibility applies to both: you must report the benchmark version and the evaluation procedure.

Related

• Get Workbook Path Only: 5 Quick Ways
• Beginner Dual Booting Mistakes: 7 to Avoid
• Dependent Dropdown Lists: How They Work

in t | Total generation time (s) | s | 120 | 0.1 | 86400
in n | Number of prompts | count | 100 | 1 | 100000
out rate | Throughput (prompts/s) | prompts/s | n / t | 3
out tpp | Time per prompt (s) | s | t / n | 3