Yujin Kim, Namgyu Ho, Sangmin Hwang, Joonkee Kim, Yongjin Yang, Sangmin Bae, Seungone Kim, Jaehun Jung, Se-Young Yun, Hwanjun Song
4 min
Abstract
Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.
Sam: So the tutor detects that collapse and adds a constraint to bring the variation back.
Alex: Exactly. It restores the conditions the model needs to keep learning. And because the whole process runs automatically, you don't need a human to sit down and design a curriculum from scratch. The difficulty scales on its own.
Sam: That said—who's checking whether the AI actually met the new constraint? If there's no clear right-or-wrong answer to verify against, how does the system know if the model succeeded?
Alex: That's a genuine challenge the paper addresses. In some domains—like math—you can check an answer automatically. But in others, like writing or reasoning tasks, there's no simple programmatic check. In those cases, the tutor model also serves as the judge. It evaluates whether the student model's output actually satisfied the constraint.
Sam: So the tutor is doing two jobs: setting the difficulty and grading the result.
Alex: Correct. And the paper acknowledges that this creates a dependency. The quality of the whole system rests on how accurately that tutor model can judge the outputs. If the tutor's judgments are off, the curriculum it builds will be off too.
Sam: So it's a meaningful advance, but the reliability of the tutor model is still a real constraint on how well this works.
Alex: That's a fair summary. What the framework does well is solve the structural problem—keeping the training signal alive as the model improves, without requiring constant human intervention. Whether it fully delivers depends on the quality of the model doing the tutoring.
Sam: It's a self-adjusting system, but only as good as the judgment at its center.
Alex: Precisely. And that tension—between automation and reliability—is probably where the most interesting follow-on work will happen. Thanks for listening to ResearchPod.