Can Language Model Agents be Helpful Circuit Explainers in Mechanistic Interpretability?

Mechanistic interpretability (MI) aims to reverse-engineer the internal logic of language models by identifying specific sub-networks, or circuits, responsible for certain behaviors. While automated methods have improved the localization of these circuits, explaining their functional roles remains a manual, labor-intensive, and non-standardized process. This paper investigates whether language model (LM) agents can automate this explanation phase by analyzing localized circuits through iterative experimentation.

The authors introduce HyVE (Hypothesize, Validate, Explain), an agentic framework that treats circuit explanation as an iterative scientific process. For each component in a localized circuit, HyVE follows a three-stage loop:

1. Observation: The agent generates and executes Python code to inspect model tensors and activation patterns.
2. Hypothesis Generation: The agent proposes a natural-language explanation for the component's role, grounded in the observed evidence.
3. Validation: The agent designs and executes controlled interventions (such as ablation or activation patching) to test the hypothesis. If the hypothesis is refuted, the agent revises its reasoning and loops back to the hypothesis stage.

To evaluate this, the authors created AgenticInterpBench, a benchmark consisting of 84 semi-synthetic transformer circuits with 163 component-level annotations, covering tasks like counting and fraction computation.

HyVE demonstrates that LM agents can effectively recover component roles and overall task behaviors. Across four tested backbones (GPT-5.4, Claude-Sonnet-4.6, Gemini-3.1-Pro, and Qwen-3-Coder), the framework achieved up to 79% accuracy in component tagging and 83% in task-level description. The study reveals that while hypothesis generation is generally strong, the primary failures occur during the validation loop, often due to incomplete experimental plans or errors in code execution. A case study on a real-world arithmetic circuit in Llama-3-8B suggests that the framework is robust enough to extend beyond synthetic benchmarks to naturally trained models.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.