From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement | Qinsi Wang et al. | ResearchPod

Transcript: From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Alex: Welcome to another episode of ResearchPod. Today, we're looking at how to train AI on tasks that lack a clear "right" or "wrong" answer.

Sam: That's a significant hurdle. Most AI training works by checking whether a model solved a math problem correctly or wrote functional code. So this paper is asking: how do we teach AI to get better at subjective tasks—like creative writing—without needing a human to judge every single output?

Alex: Exactly. The researchers propose a framework they call Reinforcement Learning with Self-Verifiable Rewards, or RLSVR. The core idea is that even when quality is hard to measure directly, you can sometimes engineer an environment where quality is forced to reveal itself.

Sam: So if you can't measure "good writing" directly, you build a situation where good writing becomes the only way to win?

Alex: That's a good way to put it. The specific method they use is called SpyRL, and it works like a social deduction game—think Among Us or Werewolf. Picture a group of AI models all given the same writing task, but one of them secretly receives an incomplete brief. It's missing key details the others have. Then the whole group votes on who they think got the bad brief—who's the "spy." To avoid being voted out, every model, including the spy, is pressured to write as clearly and completely as possible.

Sam: Oh, that's clever. The spy can't just write a vague or mediocre response, because that would give them away. So the game itself forces quality.

Alex: Right. And here's the key: the system already knows who the spy is. So the vote result is a hard, measurable fact—not a subjective opinion. That gives the AI a clear signal about whether its writing was good enough to pass as fully informed. It's a verifiable outcome for an open-ended task.

Sam: But wait—couldn't the models just learn to copy each other's style rather than actually improving? Like, find some "safe" generic writing that doesn't stand out, without getting any better at the underlying task?

Alex: That's a legitimate concern, and the researchers address it. They found that outputs which received more suspicion votes in the game were, in fact, lower quality by independent measures. So the game's rules appear to track actual writing quality reasonably well—it's not just rewarding conformity, it's rewarding the kind of clarity that only comes from genuinely understanding the task.

Sam: So the game works as a proxy. It can't perfectly measure "good writing," but it correlates with it closely enough to be useful.

Alex: That's the paper's argument, yes. And the learning process itself uses a technique called Group Relative Policy Optimization—GRPO—which scores each model's output relative to how the others in the group performed. Rather than comparing against some fixed standard, the models are always learning from each other, which the researchers say keeps the training stable.

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao

Jul 26, 2026·3:37·arxiv·Original Paper

RLSVR (Reinforcement Learning with Self-Verifiable Rewards)RLVR (Reinforcement Learning with Verifiable Rewards)SpyRLTask TransformationRLSVRSelf-supervised Learning

Listen

4 min

Briefing

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has driven recent progress in reasoning-oriented large language models (LLMs) by enabling large-scale optimization. However, its applicability remains largely limited to domains such as mathematics and coding, where correctness can be deterministically verifiable. Open-ended tasks instead often rely on human preferences, reward models, or LLM-based judges, introducing evaluation bias, judge capability bottlenecks, and additional inference costs. Drawing on the principle of self-supervised learning, which constructs pretext tasks to derive supervision from the data itself, we propose Reinforcement Learning with Self-Verifiable Rewards (RLSVR), a task-transformation-based training paradigm for extending RLVR to open-ended tasks. RLSVR transforms open-ended tasks into verifiable proxy environments whose internal rules and interaction outcomes automatically generate reward signals. We instantiate RLSVR with SpyRL, a Self-PlaY Reinforcement Learning method inspired by social deduction game Who Is the Spy?. Agents receive asymmetric information, complete the same target task, and vote to identify a designated spy. Because the spy identity is predetermined, voting outcomes provide fully verifiable rewards, while successful identification remains closely related to output quality. Experiments on text summarization, creative writing, and mathematical reasoning show that SpyRL outperforms existing self-improvement methods on non-verifiable tasks and yields consistent gains on verifiable reasoning tasks. These results demonstrate that task transformation can extend scalable RLVR-based self-improvement beyond inherently verifiable domains. Models and code have been released at https://github.com/wangqinsi1/RLSVR/tree/SpyRL.

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement: Research Paper Podcast

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Listen