Beyond Simpson’s Paradox: A Cascade of Confounders in AI Agent Pull-Request Co-Authorship

This paper investigates whether human involvement—specifically the use of a Co-Authored-By trailer in AI-generated pull requests (PRs)—improves the likelihood of those PRs being merged. While a simple aggregate analysis suggests that human co-authorship reduces merge rates, the authors hypothesize that this finding is misleading due to confounding variables.

The authors analyze 33,596 PRs from the AIDev dataset, covering five AI coding agents: Codex, Copilot, Devin, Cursor, and Claude Code. They employ a staged robustness analysis to disentangle the effects of co-authorship from other factors. First, they stratify the data by agent identity to address Simpson's Paradox. Second, they apply within-repository fixed effects to control for repository-level differences. Finally, they control for PR structure (specifically commit counts) to determine if the observed effects are driven by the nature of the PRs themselves rather than the act of co-authorship.

The study reveals that the initial negative association between co-authorship and merge rates is a classic Simpson's Paradox driven by agent composition; Codex, which dominates the dataset, has high merge rates but rarely uses co-authorship. When stratifying by agent, the relationship reverses for most agents. However, further controls reveal a 'cascade of confounders.' For instance, Copilot's apparent positive effect of co-authorship disappears when restricted to multi-commit PRs, as many purely autonomous PRs are simply single-commit drafts. Ultimately, once repository selection and PR structure are accounted for, no agent shows a statistically significant causal benefit from co-authorship.

These findings caution researchers and practitioners against relying on pooled statistics when evaluating AI agent performance. The study demonstrates that what appears to be a 'human-in-the-loop' benefit is often a reflection of PR maturity and repository-specific workflows. It highlights the necessity of rigorous stratification and structural controls in empirical software engineering research to avoid drawing spurious conclusions about the efficacy of human-AI collaboration.