ResearchPod Summary
Modern coding agents are primarily evaluated on their ability to produce functionally correct code, often using benchmarks like SWE-bench. However, real-world software engineering involves continuous maintenance tasks—such as refactoring, improving documentation, and modernizing library usage—that preserve functionality while enhancing code quality. These non-functional improvements (NFIs) are critical for long-term software evolution but remain largely ignored by existing benchmarks.
To address this gap, the authors introduce SWE-NFI, a benchmark consisting of 188 tasks derived from real-world merged pull requests in Python repositories. The researchers operationalized NFIs into 92 executable rules across five key categories: Documentation, Error Handling, Library Constraints, Coding Style, and Logic Patterns. These rules are categorized as either deterministic (e.g., enforcing naming conventions) or threshold-based (e.g., limiting cyclomatic complexity). The benchmark evaluates agents by comparing their output against the original code, measuring whether they successfully implement these improvements while maintaining functional correctness.
The study evaluated several state-of-the-art commercial and open-source coding agents. While the top-performing agent achieved a 70.0% functional correctness rate, all agents struggled to match human-level performance in NFI tasks. The performance gap was most pronounced in structural code improvements, where agents scored between 0.0 and 1.3, compared to 1.5 for human developers. The results indicate that while agents are becoming proficient at fixing bugs and adding features, they lack the nuanced capability to perform the high-quality, behavior-preserving maintenance that characterizes professional software development.
This research highlights a critical blind spot in current AI-assisted software engineering. By shifting the focus from mere functional correctness to maintainability and code quality, SWE-NFI provides a necessary foundation for developing more robust coding agents. It demonstrates that future progress in autonomous software engineering must account for the non-functional aspects that ensure software remains readable, secure, and maintainable over time.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.