How AI Coders Discuss, Disagree, and Reach Consensus: Challenges and Opportunities for LLM-Based Qualitative Coding

Qualitative coding is a labor-intensive, interpretive practice. While LLMs are increasingly used to automate this process, there is little empirical evidence regarding the conditions under which AI agents perform reliably. This study investigates how multi-agent LLM systems can be designed to simulate human-like collaborative coding, specifically focusing on how agents discuss, disagree, and reach consensus to improve coding accuracy.

The authors developed a baseline pipeline where two LLM agents independently code qualitative data, debate their disagreements, and attempt to reconcile them. They evaluated this pipeline across four diverse datasets (education, law, sociology, and medicine). The researchers analyzed the relationship between coding accuracy and various factors, including codebook length, data similarity, and the nature of agent discussions. They also compared the discursive moves of AI agents against established human coding behaviors from the literature.

The study found that coding accuracy is significantly influenced by structural and contextual factors. Specifically, concise codebooks and consistent, focused data excerpts lead to higher initial accuracy. A key, counterintuitive finding is that intense, unresolved debates between agents often result in higher accuracy than quick consensus. These debates act as a diagnostic tool, revealing nuanced details about code application that clarify inclusion and exclusion criteria. While LLMs successfully emulate many human discursive moves—such as stance maintenance and evidence reassessment—they lack the adaptive, inquisitive, and emotionally nuanced nature of human coders, often failing to ask clarifying questions or adjust to evolving conversational contexts.

This research provides a conceptual toolkit for designers of automated qualitative coding systems. It suggests that developers should move away from viewing disagreement as a system failure. Instead, designers should implement features that leverage uncertainty (e.g., 'Undecidable' labels) and controversy as indicators of interpretive richness. By tuning agent discussion styles to match the task—using 'debate' modes for complex, ambiguous data—designers can create systems that act as partners in reflexive inquiry rather than mere black-box classifiers.