Speculative Decoding Indexing Wrong Format

Speculative decoding for coding agents was indexing the wrong format when it treated the token stream as a flat sequence, ignoring the tree-structured nature of code edits and the agent’s action history. This article explains what that means, why it happens, and how to fix the indexing so the speculative model proposes tokens that the target model actually accepts.

Background: Why Speculative Decoding Needs Good Indexing

Speculative decoding speeds up autoregressive generation by having a small draft model propose several tokens, which a larger target model then verifies in a single forward pass. The speedup depends on the draft model’s proposals matching what the target model would have generated. For coding agents, the draft model is often a smaller code model, and the target is a larger one. The draft model must be conditioned on the same context that the target model sees, otherwise its proposals are random and the acceptance rate collapses.

In a coding agent, the context is not just the current file text. It includes the conversation history, the agent’s previous actions (like function calls or file edits), and the current state of the codebase. Indexing that context into a format the draft model can use is non-trivial. The most common mistake is to flatten everything into a single token sequence, losing the structure that tells the model where a new edit begins or which part of the file it is looking at.

How Speculative Decoding Works for Coding Agents

A typical coding agent loop looks like this:

1. The agent receives a user request and the current repository state.
2. It decides on an action (e.g., edit a file, run a command, search for a symbol).
3. It generates a response or a code diff.
4. The result is applied to the repository, and the loop repeats.

When you apply speculative decoding to step 3, the draft model must generate the same tokens the target model would. That means the draft model’s input must include the full context: the system prompt, the user request, the conversation so far, and the current file contents. The standard approach is to concatenate all of that into one token sequence, with special tokens separating sections. The draft model then predicts the next token, and the target model verifies the whole sequence.

The problem arises when the agent’s actions are not just text. For example, an edit action might be represented as a structured object like {"file": "src/main.py", "old": "...", "new": "..."}. If you serialize that object into tokens, the draft model sees the JSON keys and values, but it does not know that the new field is the actual code that will be written. The target model, however, might be trained on a different format, such as a unified diff or a code-specific edit format. When the draft model’s input format does not match the target model’s expected format, the draft model is effectively indexing the wrong format.

The Wrong Format: Flat Token Sequences vs. Structured Context

Most speculative decoding implementations for general text use a flat token sequence: prompt + generated tokens. That works because the target model is also trained on flat text. But coding agents often use structured formats for actions and observations. For example, an agent might represent a file edit as:

EDIT: src/main.py
OLD:
    def foo():
        return 1
NEW:
    def foo():
        return 2

If the draft model is trained on this format, it will propose tokens that follow the format. But if the target model is trained on a different format, like a JSON diff or a code block with line numbers, the draft model’s proposals will be rejected. The indexing is wrong because the draft model is indexing the context in a format that does not match the target model’s training distribution.

Another common issue is that the draft model is indexed on the entire conversation history, including tool outputs that are very long. The target model might have a shorter context window or a different attention pattern. If the draft model sees a truncated or differently ordered context, its proposals will be off.

How to Choose the Right Indexing Format

The correct format depends on the target model. You need to know what format the target model was trained on for coding agent tasks. Some models are trained on plain text with special tokens for tool calls. Others are trained on a specific JSON schema. The safest approach is to use the same format that the target model uses for its own input, because the target model will always see that format during inference.

In practice, that means:

• If the target model uses a chat template with system/user/assistant roles, the draft model should see the same roles.
• If the target model uses a special token for file edits, the draft model should use that token too.
• If the target model expects a unified diff, the draft model should be given a unified diff, not a JSON object.

You can also train a small draft model specifically for the target model’s format. That is the most reliable way to get high acceptance rates, but it requires additional training data and compute.

Common Problems and Misconceptions

One common misconception is that the draft model only needs to see the last few tokens. In reality, the draft model needs the full context, because the target model’s predictions depend on the entire history. Truncating the context will cause the draft model to propose tokens that are plausible locally but wrong globally.

Another problem is that the draft model might be a general code model, not fine-tuned for the agent’s specific action format. A general code model will predict ordinary code, but the target model might be predicting a special token that indicates an edit. The draft model will never propose that token, so it will be rejected.

Some implementations try to solve this by using the target model’s logits to guide the draft model, but that defeats the purpose of speculative decoding because you need the target model’s logits for every token, which is expensive.

Costs and Trade-offs

Fixing the indexing format usually requires either changing the draft model or changing the serialization. Changing the serialization is cheap: you just write a function that converts the agent’s state into the target model’s expected format. The cost is that you need to know the target model’s format precisely, and you need to test that the draft model’s acceptance rate actually improves.

Training a custom draft model is more expensive, but it can yield much higher acceptance rates. The trade-off is training time and data collection. For most applications, using an off-the-shelf small model with the correct format is sufficient.

Alternatives to Speculative Decoding for Coding Agents

If speculative decoding is not working because of format mismatch, you have alternatives:

• Use a single model without speculation, and rely on faster hardware.
• Use a smaller target model that is faster, but still accurate enough.
• Use lookahead decoding, which generates multiple candidates in parallel without a draft model.
• Use a retrieval-augmented approach to reduce the context length, making the generation faster.

Each has its own trade-offs. Lookahead decoding can be simpler because it does not require a separate draft model, but it may not achieve the same speedup.

Frequently Asked Questions

Q: Why does the draft model need to see the same format as the target model?
A: Because the draft model is predicting tokens that the target model will verify. If the draft model does not know the exact token sequence the target model expects, it will propose tokens that are out of distribution, and the target model will reject them, so the acceptance rate drops and you get no speedup.

Q: Can I use a general code model as the draft model?
A: Yes, but only if the target model also uses plain code text. If the target model uses special tokens for edits or tool calls, the general code model will not propose those tokens, so they will always be rejected. You need a draft model that is aware of the agent’s action format.

Q: What is the most common indexing mistake?
A: Flattening the entire context into a single string without preserving the structure of the agent’s actions. For example, putting the file content and the edit instruction in the same text block without clear delimiters. The draft model then cannot distinguish between the current file and the proposed edit.

Q: How do I know what format the target model expects?
A: Check the model’s documentation or the training data. Many models are released with a chat template or a specific serialization for tool calls. You can also run the target model on a few examples and see what tokens it generates for the same context.

Q: Is it worth training a custom draft model?
A: If your agent has a very specific action format and you need high throughput, yes. But for most cases, using an off-the-shelf small model with the correct serialization is enough. Training a custom model adds complexity and maintenance overhead.

Q: Can I fix the format mismatch without changing the draft model?
A: Yes, you can change the serialization to match what the draft model expects. For example, if the draft model is a general code model, you can convert the agent’s actions into plain code comments or a diff format that the draft model has seen during training. But you must ensure the target model also accepts that format.

Related

• Bose QuietComfort SE: What to Know Before Buying
• Specified User Does Not Have a Valid Profile? Fix
• GTA 6 Music Broken? How to Tell