How Do Practitioners Build SE Agents? Insights from a Mixed-Methods Study

As software engineering (SE) agents—LLM-based systems capable of performing engineering tasks with minimal human intervention—become routine, the traditional software development lifecycle is undergoing a fundamental transformation. This study, utilizing a mixed-methods approach with 20 in-depth interviews and a survey of 80 practitioners, reveals that the ease of generating code via LLMs does not eliminate development bottlenecks; instead, it relocates them.

While coding speed has increased, the complexity of managing agent behavior has shifted the focus toward non-coding activities. Requirements, coordination, and deployment have become more visible and critical. Most notably, the review and evaluation of agent-generated output have emerged as the central activity in the development process.

The study identifies a seven-stage workflow for building SE agents: requirements, evaluation, data, system construction, testing and deployment, human feedback, and adaptive maintenance. A key finding is the transition to evaluation-driven development. In this model, evaluation is no longer a final quality-assurance step but the primary steering mechanism for iteration. Furthermore, specifications are evolving from static human-readable documents into versioned, executable artifacts that are consumed by both human developers and the agents themselves.

Practitioners face significant hurdles in this new paradigm. Evaluation signals are often unreliable, leading to a version of the test-oracle problem where tests are trusted simply because they exist. Teams also struggle with 'comprehension debt,' where the speed of code generation outpaces the team's ability to understand the codebase. Additionally, the 'change nothing, change everything' effect—where provider-side model updates silently alter agent behavior—forces teams to adopt new maintenance strategies, such as preserving 'regenerative software' (the infrastructure to rebuild code) rather than the code itself.