Trust the Critic More | Kaiyue Wen et al. | ResearchPod

Transcript: Trust the Critic More

Alex: A language model can be trained on partial reasoning trajectories, before the final answer is scored, provided a gate decides when the critic is trustworthy enough to use. That is AC2, from a Stanford group led by Kaiyue Wen.

Sam: Critics for language models have a poor track record, though. Skipping the terminal reward is exactly where I'd expect the policy to get worse, not just cheaper.

Alex: On the evidence reported, it doesn't. AC2 reaches a higher peak score than GRPO using about two-and-a-half times fewer decoding FLOPs. The saving has two sources: fewer training steps are needed, and each step is cheaper because it avoids full rollouts. That comparison is the one the paper rests on. Everything else is scaffolding around it.

Sam: So the cost being attacked is that GRPO has to roll every trajectory out to completion before it sees any reward.

Alex: Yes. Terminal rewards force full rollouts. AC2 uses a learned critic to score intermediate states, so the model can effectively check its work partway through. It also groups tokens into action chunks, which gives the critic a meaningful segment of reasoning to evaluate rather than a single token.

Sam: But if the critic is wrong, you bake that error straight into the policy update. What stops the model learning garbage?

Alex: That is the local readiness gate. Critic-based updates are switched on for a given problem only when the critic's error on that problem falls below a threshold. Until then, the problem is treated as unready and gets standard terminal-reward updates.

Sam: So it's a safety filter. Noisy critic, fall back to the GRPO-style update. Once the critic has earned it, the gate opens and the model learns from intermediate chunks.

Alex: Right. When the gate is open, the advantage estimate comes from the critic alone, the lambda-zero case, instead of from the terminal reward.

Sam: How does it bootstrap, though? A randomly initialized critic can't pass any threshold.

Alex: That is the cold-start hurdle. Every problem begins as unready, so the system behaves like a standard policy-gradient method while the critic accumulates experience. Readiness uses two thresholds. The global one tracks mean critic error over the last five steps, and the local one checks error on that specific problem. It is a deliberately conservative design.

Sam: And the auditing piece matters here, doesn't it? Even after a problem is marked ready, some fraction of cases still get full rollouts.

Alex: Yes, and that is the ground-truth check against drift. If the critic diverges, the audited error spikes and the gate closes, which sends those problems back to terminal-reward updates.

Trust the Critic More

Kaiyue Wen, Luke Bailey, Arvind Mahankali, Tengyu Ma

Oct 6, 2026·5:04·user_upload

Actor-Critic with Action Chunking (AC2)Credit AssignmentLocal ReadinessDecoding FLOPsTerminal RewardGRPO (Group Relative Policy Optimization)

Listen

5 min

Summary

Research Question and Approach

Standard reinforcement learning for LLMs, such as Group Relative Policy Optimization (GRPO), typically relies on terminal rewards to evaluate entire reasoning trajectories. This coarse credit assignment penalizes potentially correct reasoning if a minor error occurs at the end. The authors propose Actor-Critic with Action Chunking (AC2), which treats rollouts as sequences of action chunks. By using a learned critic to score the state at the end of each chunk, the policy can update without waiting for a terminal reward. To ensure the critic remains reliable, AC2 employs three design choices: local readiness (updating only when the critic is sufficiently accurate on a specific problem), using reference solutions to guide the critic, and evaluating chunks of 10,000 tokens to provide a meaningful signal.

Main Findings

AC2 significantly improves compute efficiency and performance compared to GRPO. On the IMO-ProofBench benchmark, AC2 exceeds GRPO's peak validation score of 18.5% while using 2.5 times fewer decoding FLOPs. This efficiency gain stems from two factors: AC2 requires 25% fewer training steps to reach peak performance, and it generates fewer tokens per update because it does not need to complete every trajectory. The authors demonstrate that the critic becomes trustworthy early in training and that the combination of local readiness, reference solutions, and action chunking is essential for these gains.

Why It Matters

This work challenges the prevailing view that learned critics are too inaccurate for LLM reinforcement learning. By demonstrating that LLMs can effectively trust a learned critic to perform credit assignment on partial trajectories, the authors open a new design space for RL algorithms. This shift allows for more flexible training strategies, such as learning from off-policy initial states and performing more efficient, fine-grained updates, which are critical for scaling reasoning capabilities in models.

Key Terms and Definitions

Action Chunking — A technique that segments a trajectory into fixed-length sequences of tokens, allowing the critic to evaluate intermediate states rather than just terminal outcomes.
Local Readiness — A mechanism that restricts the use of critic-based updates to specific problems where the critic has demonstrated sufficient accuracy, reverting to terminal rewards otherwise.
Auditing — A process where a subset of critic-ready problems are still rolled out to completion to monitor the critic's accuracy and provide ground-truth targets for further training.
Decoding FLOPs — A hardware-agnostic metric used to measure the computational cost of generating tokens during policy rollouts, serving as a proxy for training efficiency.
Group Relative Policy Optimization (GRPO) — A baseline reinforcement learning algorithm that uses the mean reward of a group of complete trajectories as a baseline to estimate advantages.

Trust the Critic More: Research Paper Podcast