Span 01 vs Mercury: Why Scores Differ

Span 01 and Mercury are two different AI language models, and their reported benchmark scores are not directly comparable because they are measured under different conditions, on different test versions, and with different evaluation protocols. This article explains the specific reasons for the discrepancy and what you should look at instead of a single number.

Formula

The calculator above simply subtracts one score from the other: Difference = Mercury score − Span 01 score. Both scores are entered as points on a 0–100 scale. A positive result means Mercury scored higher; a negative result means Span 01 scored higher. The formula assumes both scores are on the same scale, which is often not the case in real benchmarks—this is exactly why the raw difference can be misleading.

Worked Example

Using the default values: Span 01 = 85, Mercury = 90. The calculation is 90 − 85 = 5 points. So on the surface, Mercury appears 5 points better. But if the two scores come from different test sets or different prompting methods, that 5-point gap may reflect the test setup rather than actual model capability.

Why the Scores Are Not Comparable

Benchmark scores are only meaningful when the evaluation is identical. Here are the main factors that break comparability between Span 01 and Mercury:

• Different benchmark versions – A model evaluated on MMLU 2023 vs MMLU 2024 is not being compared on the same questions.
• Different prompting templates – Some models are tested with few-shot examples, others with zero-shot; even the wording of the instruction changes results.
• Sampling temperature – A model run with temperature 0 (greedy decoding) gives different answers than one run with temperature 0.7.
• Pass@k vs top-1 accuracy – Some benchmarks report the best of k samples, which inflates scores compared to a single attempt.
• Different test splits – If one model was evaluated on a held-out subset and the other on the full set, scores are not directly comparable.
• Contamination – If a model’s training data included the benchmark questions, its score is artificially high.

How Benchmark Scores Are Actually Produced

Most public benchmarks (like MMLU, HumanEval, or GSM8K) work like this: a fixed set of questions is fed to the model, the model’s answers are compared against a golden answer, and the percentage of correct answers becomes the score. But the details matter enormously:

• Question format – Multiple choice vs free-form generation changes difficulty.
• Scoring rubric – Some benchmarks accept any answer containing the correct number; others require exact string match.
• Number of runs – A single run can be noisy; averaging over multiple runs gives a more stable estimate.
• Compute budget – A model given more tokens to “thinkâ€� (like chain-of-thought) scores higher than one forced to answer immediately.

Because these settings are rarely identical between two different model families, the reported numbers are not apples-to-apples.

What the Scores Actually Measure

Even if the evaluation were identical, a benchmark score measures only a narrow slice of capability. For example:

• MMLU measures world knowledge and reasoning in 57 subjects, but it is multiple-choice, so it doesn’t test free-form generation.
• HumanEval measures code generation, but only on short Python functions—not real-world software engineering.
• GSM8K measures grade-school math, but the questions are formulaic and often appear in training data.

So a 5-point difference on one benchmark tells you little about which model writes better emails, summarizes documents, or follows complex instructions.

Common Mistakes When Comparing Scores

People often make these errors when looking at benchmark tables:

• Comparing scores from different papers – Each paper may use different evaluation code, even for the same benchmark.
• Ignoring the variance – A model can score 85 on one run and 87 on another; a 2-point gap is often noise.
• Using the wrong column – Some tables report “averageâ€� across subtasks, others report “weighted averageâ€� or “macro average.â€�
• Not checking the date – Models improve over time; a score from six months ago is not comparable to today’s score.

Always look for the original evaluation report, not a summary table, and check the exact settings.

How to Choose Between Span 01 and Mercury

Instead of relying on a single score, do your own side-by-side test on the tasks you care about. Here is a practical approach:

1. Write 5–10 prompts that represent your real use case (e.g., drafting emails, summarizing reports, writing code).
2. Run the same prompts on both models with the same settings (temperature, max tokens).
3. Blindly compare the outputs—have someone else rate them or use a rubric.
4. Repeat the test a few times to account for randomness.
5. Consider cost, speed, and API reliability—not just quality.

This gives you a decision that matches your needs, not a number from a benchmark that may not reflect your task.

Frequently Asked Questions

Can I directly compare the MMLU scores of Span 01 and Mercury?

Only if both scores come from the same evaluation harness, the same version of MMLU, and the same prompting method. Otherwise, the comparison is invalid. Check the original papers or official reports for the exact setup.

Why does one model score higher on one benchmark but lower on another?

Different benchmarks test different abilities. A model may be strong at math (GSM8K) but weak at general knowledge (MMLU), or vice versa. Also, some benchmarks are more susceptible to overfitting or contamination.

What is a “pass@k� score and why is it higher than a normal score?

Pass@k means the model gets k chances to generate a correct answer, and the score is the probability that at least one of them is correct. This is always higher than a single-attempt score because it allows multiple tries. It is common in code generation benchmarks.

Does the order of the questions affect the score?

Yes, especially for models with long context windows. Some models perform better when easy questions come first, or when questions are grouped by topic. Good evaluations randomize the order, but not all do.

Is a 5-point difference in a benchmark significant?

Often not. Many benchmarks have a standard error of 1–2 points, so a 5-point gap might be real, but it might also be due to randomness. Look for confidence intervals or multiple runs.

Should I trust the higher-scoring model?

Not automatically. A higher score on a public benchmark does not guarantee better performance on your specific task. Always test on your own data before making a decision.

Related

• Microsoft Copilot Agents: What They Actually Do
• Convert Live Photo Popular Formats Explained
• Steal an Egg Eternal Oni Tiger Bloom Mutation?

in score_span | Span 01 score | points | 85 | 0 | 100
in score_mercury | Mercury score | points | 90 | 0 | 100
out diff | Difference (Mercury - Span) | points | score_mercury - score_span | 1