Prompt Compression: Measure Coding Agent Accuracy & Cost

This guide walks you through a controlled experiment to measure whether compressing prompts improves a coding agent’s accuracy and reduces cost. You’ll set up a baseline, apply compression, run the same tasks, and compare results using clear metrics.

What You’ll Need

• A coding agent (e.g., GitHub Copilot, Cursor, or an API-based agent like Claude or GPT) with a way to log prompts and responses.
• A set of 10–20 representative coding tasks (e.g., bug fixes, feature additions, refactors) with known correct outputs.
• A prompt compression tool or method (e.g., LLM-based summarization, removing redundant context, or using a compression library).
• Access to usage logs or API cost tracking (if using an API).
• A spreadsheet or notebook to record results.

Step-by-Step

1. Define your tasks and ground truth. Choose a set of tasks that are representative of your typical workload. For each, write a clear description, the exact input (e.g., code snippet), and the expected output (e.g., correct diff or answer). Use 10–20 tasks to get statistically meaningful results.
2. Create a baseline prompt for each task. Write a prompt that includes all relevant context: task description, relevant code files, and any constraints. This is your “uncompressed” prompt. Save it exactly as you’ll use it.
3. Run the baseline. For each task, feed the uncompressed prompt to the coding agent. Record the agent’s output, whether it matches the ground truth (exact or functionally equivalent), and the time taken. If using an API, also record token usage (input and output tokens) and cost.
4. Compress the prompts. Apply your compression method to each baseline prompt. The goal is to reduce token count while preserving essential information. You can use an LLM to summarize, remove redundant comments, or trim verbose instructions. Ensure the compressed prompt still contains all necessary code and task requirements.
5. Run the compressed prompts. Feed the compressed prompts to the same agent under the same conditions. Record the same metrics: output correctness, time, token usage, and cost.
6. Measure accuracy. For each run, mark a binary pass/fail based on whether the output meets the ground truth. Calculate the accuracy percentage for both baseline and compressed runs: (number of passes / total tasks) * 100.
7. Measure cost. If using an API, sum the token usage across all tasks for each condition. Multiply by your provider’s per-token rate (or use the cost field from the API response) to get total cost. If using a subscription, estimate cost by token count or time.
8. Compare and analyze. Create a table with columns: Task ID, Baseline Accuracy, Compressed Accuracy, Baseline Tokens, Compressed Tokens, Baseline Cost, Compressed Cost. Calculate averages and percentage differences. For example, accuracy change = (compressed accuracy – baseline accuracy) / baseline accuracy * 100. Cost change = (compressed cost – baseline cost) / baseline cost * 100.
9. Run multiple trials (optional but recommended). To account for randomness in LLM outputs, run each condition 2–3 times and average the results. This gives a more reliable picture.
10. Draw conclusions. Look at the trade-off. If accuracy drops by more than a few percentage points but cost drops significantly, decide if that’s acceptable. If accuracy is maintained or improved while cost decreases, compression is beneficial. Document your findings.

Common Mistakes

• Not controlling variables: Changing the agent version, temperature, or other parameters between baseline and compressed runs invalidates the comparison. Keep everything identical except the prompt.
• Compressing too aggressively: Removing essential code or instructions will degrade accuracy. Ensure the compressed prompt still contains all functional requirements.
• Using a non-representative task set: If your tasks are too simple or too niche, the results won’t generalize to real usage.
• Ignoring output token cost: Compression might reduce input tokens but could increase output tokens if the agent asks for clarification or generates more verbose code. Track both.
• Not measuring latency: Cost isn’t just money; it’s also time. Compression might reduce token count but increase thinking time. Record time as a secondary metric.

Tips

• Start with a small pilot: Test with 3–5 tasks first to validate your compression method and measurement process before scaling up.
• Use a consistent compression method: If you’re using an LLM to compress, use the same model and settings for all prompts to avoid variability.
• Log everything: Save all prompts, outputs, and metadata in a structured format (e.g., JSON) for easy analysis.
• Consider semantic equivalence: When judging accuracy, allow for functionally equivalent outputs, not just exact string matches. Use unit tests or manual review.
• Automate where possible: Write a script to run the tasks and collect results, reducing manual errors.

Troubleshooting

• Accuracy drops significantly after compression: Your compression is likely removing critical context. Review a few compressed prompts and see what’s missing. Try a less aggressive compression or a different method (e.g., extractive vs. abstractive summarization).
• Cost doesn’t decrease as expected: Check if output tokens increased. If so, the compressed prompt might be causing the agent to generate more verbose code or ask for clarification. Also verify your token counting is accurate.
• Agent behaves differently between runs: Even with the same prompt, LLMs have randomness. Run multiple trials to average out noise. Also ensure the agent’s temperature is set to 0 for deterministic output (though some agents don’t expose this).
• Compression process itself is too time-consuming: If manual compression is slow, consider using an LLM-based compression pipeline that can process prompts in batch.
• Results are inconclusive: Increase the number of tasks or trials. Also check if your tasks are too similar—diversify them to get a broader picture.

Frequently Asked Questions

How much cost reduction should I expect?

It varies widely based on the original prompt verbosity and compression method. Typical reductions range from 20% to 60% in input tokens, but output tokens may offset some of that. Measure, don’t assume.

Can compression improve accuracy?

Yes, sometimes. Removing irrelevant context can reduce noise and help the agent focus on the core task. However, it can also hurt if essential details are lost. The experiment will tell you which effect dominates.

What is the best compression method?

There’s no universal best. LLM-based summarization often preserves semantics better than simple truncation, but it adds latency and cost. Experiment with a few methods on a small set to find what works for your tasks.

How many tasks do I need for reliable results?

At least 10, but 20+ gives more confidence. If tasks are complex and outputs are hard to judge, you may need even more. Use statistical significance tests (e.g., paired t-test) if you want rigor.

Should I measure time as well as cost?

Yes, because compression can change latency. If the agent takes longer to process a shorter prompt, the user experience might not improve even if API cost drops. Record time per task.

What if my agent doesn’t expose token usage?

If you’re using a GUI tool, you can estimate tokens by counting characters and dividing by ~4 (approximate tokens per character for English). Alternatively, use the agent’s built-in logging if available.

Related

• Why Signal? Top Reasons to Switch
• SSH Security Best Practices
• More Docker Containers I Can’t Work Without