ICAE-Bench: Evaluating Coding Agents as Interactive Project Builders

As coding agents evolve from simple code assistants into autonomous project builders, they are increasingly expected to operate in "vibe-coding" workflows—scenarios where they must clarify ambiguous user intent, plan architecture, and iteratively build software. Existing benchmarks primarily focus on static, fully specified tasks, failing to capture the dynamic, interactive nature of real-world software construction. This paper introduces ICAE-Bench to evaluate how well coding agents can bridge the gap between vague requirements and functional, testable repositories.

ICAE-Bench simulates the project-building lifecycle by providing agents with a fuzzy product requirement document (PRD) and an "ultimate image" (a containerized environment stripped of golden code). The benchmark introduces three core design innovations:

1. Grounded Ambiguity: Tasks are derived from real, executable open-source repositories. Ambiguity is introduced by systematically hiding specific constraints and requirements, which are then stored as "User Agent Data."
2. Controlled Interaction: Agents can query an automated User Agent to clarify requirements. To prevent leakage or hallucination, the User Agent is restricted to returning only benchmark-authored records.
3. Multi-dimensional Evaluation: Rather than requiring literal code reproduction, the benchmark uses standardized black-box tests to verify functional correctness, alongside diagnostics for structural fidelity, design quality, and API similarity.

The benchmark includes 480 tasks across 12 programming languages, allowing for a rigorous assessment of an agent's ability to handle varying levels of requirement clarity.

Experiments with six coding models and two agent frameworks (Claude Code and OpenHands) reveal that ambiguous project generation remains a significant challenge. While agents are generally capable of reproducing visible behaviors, they struggle with hidden constraints, edge cases, and long-horizon integration. The authors find that simply increasing the amount of information provided to the agent does not guarantee a higher pass rate; the primary bottleneck is the agent's ability to preserve clarified requirements and translate them into a coherent, functional repository structure.