Coding Agent Rule Following: What Evidence Shows

Coding agents—AI tools that write, edit, and debug code—often fail to follow instructions exactly, and the evidence from developer reports and benchmark testing shows that rule adherence is inconsistent and context-dependent. While they excel at common patterns, they frequently miss specific constraints, misinterpret edge cases, or silently drop requirements when the task grows complex. This article explains what the evidence reveals about how coding agents follow rules, why they fail, and what you can do to improve compliance.

Background: How Coding Agents Interpret Rules

Coding agents are large language models fine-tuned on code. They process your instructions as natural language, not as formal specifications. Evidence from user reports and open-source issue trackers shows that agents treat rules as suggestions unless they are repeated, placed prominently, or reinforced by the surrounding code. For example, a developer might say “use only the standard library” and the agent will happily import a popular third-party package because it has seen that pattern thousands of times in training data. The model is not being malicious—it is optimizing for the most statistically likely completion, which often ignores rare constraints.

How Rule Following Works (and Fails)

When you give an agent a task, it breaks the problem into steps and generates code token by token. Each token is chosen based on the entire context, including your rules. However, the attention mechanism has limited capacity: rules stated early in the prompt get less weight than code snippets or error messages later. This is called the lost in the middle effect. A 2023 study on long-context models found that information in the middle of a prompt is recalled less accurately than information at the beginning or end. So if you put your constraints in the middle of a long description, they are more likely to be ignored.

Common Problems and Real-World Examples

• Rule conflicts: When two rules contradict (e.g., “be concise” and “add detailed comments”), agents typically satisfy one and silently drop the other. Evidence from code review forums shows that agents rarely ask for clarification—they just pick the more common pattern.
• Overfitting to examples: If you provide a few example outputs, the agent will mimic them even when your rules say otherwise. For instance, if you show a function that uses a loop, the agent will continue using loops even if you wrote “use recursion” in the instructions.
• Context window overflow: When the codebase is large, the agent may truncate or compress earlier parts of the conversation, including your rules. This is why long sessions often see the agent start ignoring instructions it followed earlier.
• False compliance: The agent may output code that looks like it follows the rule but doesn’t. A classic case is “no floating-point comparisons”—the agent will use an epsilon comparison in one place but forget it in another, because it doesn’t have a global view of the code.

How to Choose and Configure an Agent for Better Compliance

Not all coding agents are equal. Some are trained with more emphasis on instruction following, and some allow you to set custom system prompts. When choosing an agent, look for:

• System prompt support: The ability to set persistent rules that are always in context.
• Rule enforcement tools: Some agents integrate with linters or test runners that automatically check constraints (e.g., “no unused variables”) and feed failures back to the model.
• Transparency: Agents that show their reasoning steps tend to follow rules better because they are forced to “think” about each constraint.

Once you have an agent, configure it by writing rules in a separate RULES.md file and referencing it in the prompt. This keeps the rules near the top of the context and reduces the lost-in-the-middle effect.

Comparisons: Rule Following Across Agent Types

There are two broad categories: autonomous agents (that plan and execute multi-step tasks) and assistive agents (that complete snippets or suggest edits). Evidence from developer surveys suggests that assistive agents follow rules more reliably because they work on smaller, well-scoped tasks. Autonomous agents, by contrast, often lose track of rules as they navigate files and run commands. For example, an autonomous agent told to “use tabs for indentation” might create a new file with spaces because the file creation step doesn’t have the rule in its immediate context. Assistive agents, working within an open file, are more likely to match the existing style.

Safety and Limitations

Rule following is not just about convenience—it has safety implications. If an agent ignores a rule like “never delete files” or “use parameterized SQL queries”, it can introduce bugs or security vulnerabilities. The evidence shows that agents are particularly bad at following security rules because those rules are often stated as negatives (“do not use eval”) and the model has a bias toward action. To mitigate risks, always run code review and automated tests. Do not rely on the agent’s self-reported compliance; verify with tools like grep for forbidden patterns or unit tests that enforce constraints.

Frequently Asked Questions

Why does my coding agent ignore my instructions?

Most likely because the instructions are buried in a long prompt or conflict with patterns the model learned during training. Try moving the rules to the top or bottom of the prompt, and make them more specific (e.g., “use tabs, not spaces” instead of “follow style guidelines”).

Can I train a coding agent to follow rules better?

You can’t retrain a commercial agent, but you can use few-shot examples: provide 2-3 input-output pairs that demonstrate the rule. This often works better than stating the rule in words.

Do larger models follow rules better?

Generally yes, but not always. Larger models have more capacity to remember instructions, but they also have more ingrained patterns that can override them. The evidence is mixed—some small models fine-tuned specifically for instruction following outperform larger general models.

What is the best way to give rules to a coding agent?

Use a dedicated rules file and reference it, or put the most critical rule first and last in the prompt. Break complex rules into simple, atomic statements. Avoid giving more than 5-6 rules at once, as compliance drops sharply beyond that.

How do I check if the agent followed the rules?

Run automated checks: linters, style checkers, and unit tests. For security rules, use static analysis tools. Manually review the diff, focusing on areas where the rule would apply. Do not trust the agent’s summary of what it did.

Related

• Ark Crashing Issues: Causes & Fixes
• Legendary Pranks Tech Companies Pulled Off
• TMS Selection: Match Freight Needs to Workflows & Costs