Beyond Resolution Rates: Why AI Coding Agents Need Structural Awareness to Avoid Regressions

The current generation of AI coding agents has achieved impressive performance on benchmarks like SWE-bench Verified, with state of the art systems resolving over 70% of real GitHub issues from major open source repositories. Yet beneath these headline numbers lies a troubling pattern that receives far less scrutiny. When Pepe Alonso evaluated vanilla agent behavior across 100 instances in his paper, TDAD: Test-Driven Agentic Development – Reducing Code Regressions in AI Coding Agents via Graph-Based Impact Analysis, he documented 562 pass-to-pass (P2P) test failures, an average of 6.5 previously working tests broken per patch. On one particularly severe instance (astropy-13977), a single patch broke all 322 P2P tests. These regressions represent a fundamental mismatch between how we evaluate agents and how software engineering actually works. A fix that breaks three existing features would never pass human code review, yet our benchmarks largely ignore this dimension.

The Architecture of Impact Awareness

Alonso proposes a solution that eschews heavy infrastructure in favor of surgical precision. TDAD (Test-Driven Agentic Development) constructs an abstract syntax tree (AST) derived code-to-test dependency graph, then applies weighted impact analysis to surface exactly which tests a proposed change might affect. This is not merely a matter of identifying tests in the same file as the modified code. The system traces transitive dependencies through shared interfaces, recognizing that a change to a utility module might break tests located several architectural layers away.

The technical implementation relies on GraphRAG principles adapted for software engineering contexts. By parsing the repository structure into a graph where edges represent call dependencies and data flow, TDAD can perform path analysis to determine test relevance. The weighting scheme likely incorporates factors such as call frequency, inheritance relationships, and import depth, though the paper emphasizes the practical integration over algorithmic complexity. What distinguishes this approach is its zero-dependency runtime footprint. The agent requires only grep and pytest, receiving the impact analysis as a static test map and a 20-line instruction file. This design choice reflects a pragmatic understanding of agent environments; requiring a graph database, MCP server, or API calls would create friction that outweighs the benefits for many deployments.

The contrast with baseline agent behavior highlights why structural awareness matters. Without dependency graphs, agents face an impossible choice. Running the entire test suite does not scale to large codebases where tests take minutes or hours. Running only tests near changed files misses indirect dependencies where code breaks due to interface changes across module boundaries. Diffs alone cannot reveal these transitive relationships, which explains why vanilla agents consistently produce patches that satisfy the immediate issue while destabilizing the broader system.

The TDD Paradox: Context Beats Procedure

Perhaps the most surprising finding in Alonso's work concerns the failure of naive test-driven development approaches. When agents received TDD prompting instructions (essentially being told to write tests first, then implement fixes), regression rates actually increased from 6.08% to 9.94%. On the django-13089 instance, TDD prompting converted 4 failures into 352, effectively regressing the entire test suite. This counterintuitive result suggests a fundamental limitation in how we instruct smaller local models.

The analysis reveals that procedural knowledge (instructions on how to perform TDD) proves less valuable than contextual information (specific guidance on which tests to verify). Models like Qwen3-Coder 30B and Qwen3.5-35B-A3B appear to lack the capacity to both understand the issue, generate a fix, and simultaneously maintain the mental context required for proper test-driven methodology. However, when provided with TDAD's graph-based test map, these same models reduced test-level regressions by 70%, dropping the failure rate from 6.08% to 1.82%. Resolution rates simultaneously improved from 24% to 32% when deployed as an agent skill.

This distinction between procedural and contextual support has profound implications for agent tool design. Rather than teaching agents new methodologies through prompt engineering,