Speculative Decoding: How to Tell If It Changed the Output

Speculative decoding speeds up LLM inference by having a small draft model guess tokens and a large target model verify them in parallel. Because the target model accepts or rejects each draft, the final output is supposed to be identical to what the target model would have produced alone. In practice, implementation bugs, sampling quirks, or non-deterministic kernels can cause subtle differences. This guide shows you how to check whether speculative decoding actually changed the response, using a controlled comparison and a few simple statistical tests.

What You’ll Need

• Access to the target model with speculative decoding enabled (e.g., via vLLM, Hugging Face TGI, or TensorRT-LLM).
• Access to the same target model with speculative decoding disabled (or a way to force rejection of all drafts).
• A set of prompts that cover different lengths and topics (at least 20–50 for meaningful comparison).
• A script to run inference and capture outputs (Python with requests or the model’s client library works).
• Basic tools to compare text: a diff tool, or Python with difflib and statistics module.

Step-by-Step

1. Set up a controlled environment. Use the same model checkpoint, same GPU, same batch size, and same sampling parameters (temperature, top-p, max tokens, seed) for both runs. If your inference server allows, fix the random seed to reduce nondeterminism.
2. Create a test prompt set. Include short prompts (one sentence), medium prompts (a paragraph), and long prompts (several paragraphs). Mix factual, creative, and code-generation prompts. Save them in a plain text file, one per line.
3. Run baseline (no speculative decoding). Disable speculative decoding in your server config (e.g., set speculative_config to None in vLLM, or set num_speculative_tokens to 0 in TGI). Run inference on all prompts and save the outputs to a file, e.g., baseline.jsonl, with prompt ID and generated text.
4. Run speculative decoding. Enable speculative decoding with your usual draft model and settings. Run the same prompts with the same sampling parameters and seed. Save outputs to speculative.jsonl.
5. Compare outputs line by line. Write a script that reads both files, matches prompts by ID, and checks if the generated text is identical. Use difflib.SequenceMatcher to get a similarity ratio (1.0 = identical). Print any prompts where the ratio is less than 1.0.
6. Quantify differences. For each differing output, compute the number of differing tokens (you can tokenize with the model’s tokenizer) and the edit distance. Also note the position of the first difference (e.g., token 10, token 200).
7. Check for systematic patterns. Are differences always at the beginning? That might indicate a bug in the draft model’s first token. Do they happen only with long prompts? That could be a context-window issue. Group differences by prompt length and type.
8. Run a statistical test. If you have enough samples, use a paired test (e.g., Wilcoxon signed-rank) on the similarity ratios to see if the differences are significant. But even a simple count: if more than 1–2% of prompts differ, something is off.
9. Repeat with a fixed seed. If you see differences, rerun both modes with the same seed and check again. If differences disappear, they were due to nondeterminism, not speculative decoding per se.
10. Inspect the draft model’s acceptance rate. If your server logs the number of accepted tokens, compare that to the expected rate. A very low acceptance rate (<50%) might indicate the draft model is too weak, which can lead to more verification steps and potential for divergence.

Common Mistakes

• Not fixing the seed. Even without speculative decoding, GPUs can produce slightly different results across runs due to nondeterministic operations (e.g., atomic adds). Always fix the seed and, if possible, set torch.use_deterministic_algorithms(True).
• Comparing across different batch sizes. Some kernels behave differently with batch size, and speculative decoding may change the effective batch size. Use the same batch size for both runs.
• Ignoring the sampling parameters. If you use temperature > 0, the output is random. Even with the same seed, speculative decoding might consume random numbers differently, causing divergence. Use temperature = 0 for a deterministic baseline.
• Only checking exact equality. Exact equality is the gold standard, but if you use sampling, you need to compare distributions. However, for a quick check, exact equality with temperature 0 is the most reliable.
• Not accounting for the draft model’s tokenizer. If the draft model uses a different tokenizer, the mapping between tokens can introduce subtle differences. Ensure both models share the same tokenizer or handle the mapping correctly.

Tips

• Use temperature = 0 for the main comparison. This removes sampling randomness and makes any difference attributable to speculative decoding.
• Log the acceptance rate. Most inference engines expose this metric. If the acceptance rate is above 80%, differences are less likely; if it’s low, investigate.
• Test with a trivial prompt first. A prompt like “Hello” should produce the same output in both modes. If it doesn’t, your setup is broken.
• Check the target model’s logits. Some frameworks allow you to dump the logits for each token. Compare the logits of the target model with and without speculative decoding. They should be identical for the same input.
• Run the comparison multiple times. If you see occasional differences, run the same prompt 10 times with temperature 0 and compare the outputs. If they vary, there’s nondeterminism in the system.

Troubleshooting

• Differences appear only with a specific prompt. Check if the prompt triggers a long generation or a particular token pattern. It might be a bug in the draft model’s handling of certain tokens (e.g., whitespace, special tokens). Try simplifying the prompt.
• All outputs differ from the start. This suggests the draft model is not being used at all, or the verification is broken. Check your server logs for errors. Ensure the draft model is correctly loaded and the speculative config is active.
• Differences appear after a certain token count. This could be due to a context-length limit or a bug in the speculation buffer. Try reducing the number of speculative tokens (e.g., from 5 to 3) and see if the difference moves.
• Differences vanish when you set temperature = 0. That means the differences were due to sampling randomness, not speculative decoding. If you need temperature > 0, you’ll need to compare distributions, not exact outputs.
• Your server doesn’t allow disabling speculative decoding. You can simulate it by setting the draft model to be the same as the target model. Then the acceptance rate should be 100% and outputs should be identical. If not, there’s a bug.
• You see differences but the acceptance rate is 100%. If the draft model is perfect (same as target), the output should be identical. If not, the verification step is not working correctly. Check for off-by-one errors in the token indices.

Frequently Asked Questions

Is speculative decoding supposed to be lossless?

Yes, in theory. The target model verifies each draft token and accepts it only if it matches the target’s distribution. If implemented correctly, the final output should be identical to the target model alone.

Why did my output differ even with temperature = 0?

Possible reasons: nondeterministic GPU kernels, a bug in the speculative decoding implementation, or a mismatch between the draft and target tokenizers. Try updating your inference engine or using a different seed.

Can I use a simple diff to check?

Yes, for short outputs. But for long outputs, a diff might show many small changes. Use a similarity ratio or token-level comparison for a clearer picture.

What if my framework doesn’t expose the acceptance rate?

You can estimate it by comparing the number of tokens the draft model proposes vs. the number the target model accepts. Some frameworks log this; otherwise, you can infer it from the speedup—a higher speedup usually means higher acceptance.

Do differences always mean a bug?

Not necessarily. If you use temperature > 0, differences can occur due to random sampling, even without speculative decoding. But with temperature = 0, any difference is a red flag.

How many prompts should I test?

At least 20–50 to cover diverse cases. If you’re testing a fix, 10 might be enough. For statistical significance, aim for 100 or more.

Related

• Power Tool Gifts: What Not to Buy
• Brushless Power Tools: When Worth It?
• Niche Milwaukee Power Tools: What They’re Used For