Yukang Feng, Jianwen Sun, Zelai Yang, Jiaxin Ai, Chuanhao Li, Zizhen Li, Fanrui Zhang, Kang He, Rui Ma, Jifan Lin, Jie Sun, Yang Xiao, Sizhuo Zhou, Wenxiao Wu, Yiming Liu, Pengfei Liu, Yu Qiao, Shenglin Zhang, Kaipeng Zhang
8 min
Abstract
Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance.
Alex: A clear improvement with guidance, but the solo agent gap ties back to crumbling on real engineering chains.
Sam: To trust those low rates, they built strict checks. Each task starts with code that fails new-feature tests but passes old ones, plus a full human-written solution that flips it right. If tests don't confirm, experts review and fix docs, environments, or scripts—up to three tries per task, or it's discarded. Human experts do a final review for soundness.
Alex: No fuzzy tasks—everything verified against a working human fix. What about the agent setup?
Sam: Agents work in Docker containers—identical virtual rooms pre-loaded with starting code and tools, so every run starts the same, no outside interference. The agent gets the task, makes a plan, types commands to navigate files, edit code, run tests—until done or timed out. Tests run automatically after.
Alex: How much bigger are these codebases than typical tests?
Sam: On average, over 15,000 lines across 104 files—real projects like full games—versus under 230 lines in something like Terminal-Bench. Experts need over 1,000 minutes to solve them, versus 200 for shorter ones. That scale forces juggling more context over longer sequences.
Alex: With strict scoring—100% on every sub-step for a pass—what do breakdowns show?
Sam: Most stall under 30% on new features. Average progress on old functions hits over 98%, but full passes drop to 70-88% because any single break fails it—agents tweak for new stuff and snap something unrelated. Top models hit 16.7% overall, since both must be perfect. The paper suggests losing the big-picture view amid details, not raw coding skill.
Alex: Context drift explains early stalls—and why human plans boost to over 60%.
Sam: They tested self-correction: agent retries using feedback from failure, up to three times—like replaying a game level after seeing what went wrong. It lifts top models to 40-55%, mainly on new features, but old functions sometimes regress as changes widen. Gains shrink after the first retry.
Alex: Self-fixes help incrementally but risk more breaks. What do humans provide exactly?
Sam: High-level plan upfront—like a roadmap without code details—hits 41-58% passes, faster by avoiding wrong paths. Interactive guidance, with up to three human suggestions on next steps—no direct code—pushes 45-58%. Combining both gets 61% with fewer interventions. Humans fix strategy while agents handle edits.
Alex: Synergy there. For solo agents under 20%, what do failure logs show?
Sam: Manual review of 50 failed runs shows workflow breakdowns. Agents loop on shallow fixes: hit an error, tweak one line, rerun, repeat until timeout—weak adaptation to deeper issues. Or they lose track of unrelated parts, snapping old code, or misread codebase state. Step scores confirm stalls before 30% on new features.
Alex: Loops and drift nail the planning crisis—and why collaboration jumps performance.
Sam: Agents stick to surface tweaks without rethinking roots, or blame code for setup issues like a missing tool. Over long runs, they forget earlier notes or widen changes that snap unrelated parts. This points to lacking steady oversight of the whole workflow.
Alex: Where humans step in effectively. Pulling back, what does this mean for AI engineering tools next?
Sam: Focus on hybrid setups now, with agents on routine edits under human strategy—that lifts success meaningfully. Future work might blend self-correction with better memory to cut drifts. The benchmark has limits: each task took experts 40 hours on average, keeping the set at 20 problems, so broader tests are needed. Step scores catch stalls but miss code cleanliness or speed.
Alex: Small scale for quality, but sharp on planning weak spots and collab value.
Sam: This underscores advancing agentic programming means tackling long-term consistency alongside raw smarts—prioritizing practical teams for now.
Alex: Well said, Sam. That's our look at challenges in AI for long software workflows. Thanks for listening to ResearchPod.