Reinforcement learning (RL) for non-verifiable instruction following increasingly relies on LLM judges with prompt-specific rubrics as reward signals. While recent methods adapt these rubrics to the evolving policy during training, the training prompts themselves remain static, drawn from fixed corpora. This static approach often results in a critical misalignment between prompt difficulty and policy capability, leaving the judge unable to recover a discriminative reward signal when prompts fail to elicit quality variance among rollouts. To address this misalignment, we introduce LLM-as-a-Tutor, a framework that extends the LLM's role from judge to tutor: a single model serves as an examiner that pairwise-compares policy rollouts to detect non-challenging prompts, and as a generator that appends atomic constraints to them. This append-only design monotonically raises difficulty in step with the policy's capability, producing a self-calibrating training signal without external difficulty schedules. On three complex instruction-following benchmarks, our method consistently outperforms both policy-unaware baselines and prior policy-adaptive methods that adapt rubrics or rewrite prompts, suggesting prompt adaptation as a missing axis of policy-awareness in non-verifiable RL.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a framework called "LLM-as-a-Tutor," and it tackles a surprisingly fundamental problem in how we train AI models.
Sam: What's the problem?
Alex: Think about what happens when a student only ever practices problems they've already mastered. They stop growing. The same thing happens to AI. When a model is trained on a fixed set of tasks, it gets good at them quickly—and then there's nothing left to learn from. Researchers call this the plateau problem.
Sam: So the AI hits a ceiling because the training material isn't keeping up with its progress?
Alex: Exactly. And the key insight in this paper is that difficulty isn't a fixed property of a task. It's a relationship—between the task and the AI's current skill level. A problem that was genuinely hard last week might be trivial today.
Sam: So the solution is to keep adjusting the difficulty as the model improves?
Alex: That's the idea. The researchers built a system where a second AI model acts as a tutor. It watches how the student model is performing, and when it detects that the tasks have become too easy, it steps in and makes them harder.
Sam: How does it actually make them harder? Does it swap in completely new problems?
Alex: No—and this is where the design gets interesting. Rather than replacing the task, the tutor adds a single new requirement on top of it. Think of it like a teacher who says, "Okay, you can solve the equation—now solve it without using a calculator, and show every step." The core problem stays the same, but there's a new constraint layered on.
Sam: And they keep adding these one at a time?
Alex: Right. The paper calls them "atomic constraints"—small, individual requirements that stack up over time. The approach is append-only, meaning you never strip away the original task. You only add to it. That way, the model is always building on what it already knows, not starting over.
Sam: That's a clean design. But how does the tutor know when to add a new constraint? How does it detect that the model has plateaued?
Alex: It runs what the paper calls a pairwise judgment. The tutor looks at two different attempts the student model made on the same task and asks: are these two answers essentially identical in quality? If the answer is yes—if the model is producing the same result every time with no variation—that's the signal that the task has become too easy.
Sam: Why does identical output mean the task is too easy?
Alex: Because of how reinforcement learning works. The system learns by comparing outcomes—it needs to see that some attempts are better than others in order to figure out what to do more of. When every attempt looks the same, that signal disappears. There's nothing to compare, so there's nothing to learn from. The researchers call this a collapse in the reward signal.
Sam: So the tutor detects that collapse and adds a constraint to bring the variation back.
Alex: Exactly. It restores the conditions the model needs to keep learning. And because the whole process runs automatically, you don't need a human to sit down and design a curriculum from scratch. The difficulty scales on its own.
Sam: That said—who's checking whether the AI actually met the new constraint? If there's no clear right-or-wrong answer to verify against, how does the system know if the model succeeded?
Alex: That's a genuine challenge the paper addresses. In some domains—like math—you can check an answer automatically. But in others, like writing or reasoning tasks, there's no simple programmatic check. In those cases, the tutor model also serves as the judge. It evaluates whether the student model's output actually satisfied the constraint.
Sam: So the tutor is doing two jobs: setting the difficulty and grading the result.
Alex: Correct. And the paper acknowledges that this creates a dependency. The quality of the whole system rests on how accurately that tutor model can judge the outputs. If the tutor's judgments are off, the curriculum it builds will be off too.
Sam: So it's a meaningful advance, but the reliability of the tutor model is still a real constraint on how well this works.
Alex: That's a fair summary. What the framework does well is solve the structural problem—keeping the training signal alive as the model improves, without requiring constant human intervention. Whether it fully delivers depends on the quality of the model doing the tutoring.
Sam: It's a self-adjusting system, but only as good as the judgment at its center.
Alex: Precisely. And that tension—between automation and reliability—is probably where the most interesting follow-on work will happen. Thanks for listening to ResearchPod.