keepcalculator

06 Numbers and Powers

Why Changed Code Can Still Miss Tests: Trace Filter

Card No. 06·72170 · Updated · Standard formula, shown below

Changed code can still miss your tests because most test-trace filters compare the code that ran against the test’s coverage map, not against the diff itself. If a test exercises a function that calls another function, and you change only the called function, the test still “covers” the caller, so the filter may not flag it — even though the change can break the test.

What a Trace Filter Actually Does

A trace filter is a tool that decides which tests to run after a code change. It works by recording which lines, branches, or functions each test executes (a trace), then comparing that trace to the set of lines you changed. If a test’s trace overlaps the changed lines, the filter schedules that test. If not, it skips it. This is the core idea behind “test impact analysis” or “selective testing.”

The key word is overlap. The filter does not understand what your change means; it only checks whether the changed line appears in the test’s recorded trace. That sounds simple, but the gap between “the test ran this line” and “the test is affected by this line” is where misses happen.

Why the Diff Isn’t the Whole Story

The most common reason a changed line escapes the filter is that the test never directly executes that line. Consider a simple example:

def calculate_total(items):
    return sum(item.price for item in items)

def apply_discount(total):
    return total * 0.9

def checkout(items):
    return apply_discount(calculate_total(items))

Your test calls checkout(). The trace records that the test executed lines inside checkout, calculate_total, and apply_discount. Now you change the logic inside apply_discount (for example, change the discount to 0.8). The filter sees that the changed line is in apply_discount, and the test’s trace includes that line, so the test is selected — good.

But what if the change is in a function that is called indirectly? Suppose calculate_total calls a helper get_price:

def get_price(item):
    return item.price

def calculate_total(items):
    return sum(get_price(item) for item in items)

If you change get_price, the trace of the test that calls checkout will include get_price because it ran. So that test is still selected. The problem arises when the test’s trace is coarse — for example, if the trace only records method-level coverage, not line-level. Then a change to a single line inside get_price might not be matched if the trace records only the method name and the changed line belongs to a different method that the test didn’t call.

Coarse-Grained Traces Are the Usual Culprit

Trace filters differ in granularity. Some record every line executed; others record only function or class names. The coarser the trace, the more likely a change will be missed. For example, if your trace records only the top-level function that a test calls (say checkout), and you change a nested helper like get_price, the filter won’t see the overlap because the changed line is not in checkout’s trace — it’s in get_price’s trace, which might not be recorded separately.

Even with line-level traces, there are gaps. If a test uses mocking or stubbing, the real implementation of a function may not execute at all. Your test might mock apply_discount to return a fixed value, so the real apply_discount never runs. If you change the real apply_discount, the trace shows no overlap because the test never executed that code — even though the test’s behavior depends on what apply_discount does in production.

Dynamic Behavior: Conditionals and Polymorphism

Another gap comes from conditional execution. A test may execute a line only under certain conditions. If your change adds a new branch, the test might not hit that branch because the input data doesn’t trigger it. The trace filter sees that the test didn’t execute the new line, so it skips the test. But the test could still fail if the new branch changes behavior for a different input that the test does use — but the trace only records what actually ran, not what could run.

Polymorphism and dynamic dispatch cause similar issues. If a test calls a method on an interface, the trace records the concrete implementation that ran. If you change a different implementation of the same interface, the filter won’t match because that implementation wasn’t executed. Yet the test could fail if the change alters the contract that the test relies on.

How to Reduce Misses

You can’t make a trace filter perfect, but you can shrink the gap:

  • Use finer-grained traces — line-level or branch-level coverage instead of method-level. This catches more direct changes.
  • Include “caller” information — some tools record not just the executed line but also the call stack. That way, if a test calls checkout and checkout calls apply_discount, a change in apply_discount is matched even if the trace only recorded checkout.
  • Run a “safety net” suite — always run a small set of critical tests (e.g., smoke tests) regardless of the filter, to catch integration issues.
  • Combine with static analysis — some tools also look at the call graph to infer which tests might be affected, even if they didn’t execute the changed line.
  • Review the filter’s logic — know whether it uses line coverage, method coverage, or call-graph analysis, and adjust your testing strategy accordingly.

Common Misconception: “The Filter Runs All Affected Tests”

People often assume that a trace filter is a safety guarantee. It isn’t. It’s a heuristic that trades completeness for speed. The filter is designed to skip tests that are very likely unaffected, not to prove that a test is unaffected. That’s why most CI systems that use trace filtering still run the full suite on a schedule (e.g., nightly) or on merge to main.

Another misconception is that the filter is “smart” enough to understand the semantics of the change. It doesn’t. If you change a constant from 0.9 to 0.8, the filter sees the line changed, but it doesn’t know whether that affects a test that only checks the type of the result, not the value. The test might pass, but the filter will still run it because the line changed — that’s a false positive, which is safe. The dangerous case is the false negative: a test that should run but doesn’t.

What to Do When You Suspect a Miss

If a test fails in production but the trace filter didn’t run it, you can manually force the test to run. Most tools allow you to “run all tests” or bypass the filter. In CI, you can add a label or a comment to trigger the full suite. After that, examine the trace for that test: did it record the changed line? If not, you’ve found the gap — likely due to mocking, indirect calls, or coarse granularity.

To prevent future misses, consider adding a “coverage check” step that fails the build if a changed line is not covered by any test that the filter selected. That way, you force yourself to write a test that directly exercises the changed line, making the filter’s job easier.

Frequently Asked Questions

Why does my trace filter miss a test that I know is affected?

The most common reasons are: the test mocks the changed function, the trace is method-level and the change is in a helper that the test doesn’t directly call, or the change is in a branch that the test’s input doesn’t execute. Check the trace for the test — if the changed line isn’t in it, the filter can’t match it.

Should I stop using trace filters?

No. They save significant time in large codebases. But you should treat them as an optimization, not a guarantee. Run a small critical suite always, and run the full suite on a schedule or before release.

How do I know if my trace filter is safe enough?

Look at the false-negative rate: how often does a test fail that the filter skipped? You can measure this by running the full suite in parallel for a while and comparing results. If you see misses, adjust the granularity or add a safety net.

Can I make the filter smarter?

Some tools allow you to add “dependency” rules — for example, if a file changes, always run tests that import it, even if the trace doesn’t show it. This is a manual way to close gaps. You can also use static call-graph analysis to supplement dynamic traces.

Does this happen with all trace filters?

Yes, to varying degrees. The fundamental limitation is that dynamic traces only know what ran, not what could run. Any filter based on traces will have this blind spot. The severity depends on the granularity and the tool’s ability to infer indirect dependencies.

Related

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →