Peiyang Xu, Bangzheng Li, Sijia Liu, Karthik R. Narasimhan, Pramod Viswanath, Prateek Mittal, Xingyu Fu
5 min
Large language models often struggle with context unawareness, where they fail to ground their outputs in decisive, sparse evidence within long trajectories or dense images. The authors propose ContextRL, a post-training framework that augments standard reinforcement learning (GRPO) with an auxiliary context-selection objective. Instead of only rewarding the final answer, the model is trained on contrastive pairs—a query and answer paired with both a supporting context and a superficially similar but incorrect 'confounder' context. The model is explicitly rewarded for assigning higher probability to the supporting context, forcing it to learn a more robust grounding mechanism.
The authors developed domain-specific pipelines to generate these contrastive pairs. For coding agents, they mined 1,000 trajectory pairs from SWE-smith by filtering for trajectories that share the same repository, file, and function but differ in the specific code edit required. For multimodal tasks, they created 7,000 image pairs using generative editing (for natural images) and similarity-based retrieval (for structured images like charts), ensuring that the 'negative' context is visually similar to the 'positive' one but supports a different answer.
ContextRL consistently outperforms standard GRPO baselines across 5 long-horizon agentic benchmarks and 12 multimodal reasoning tasks. In agentic coding, the method allows smaller 8B models to outperform significantly larger models like Qwen3-32B. In multimodal tasks, it provides broad improvements across mathematical, scientific, and general visual reasoning categories. Crucially, the authors demonstrate that these gains are not merely due to the additional contrastive data; when they used the same data for standard supervised fine-tuning or outcome-based RL, the performance improvements vanished, confirming that the specific context-selection objective is the primary driver of success.
This work provides a lightweight, modality-agnostic way to improve model grounding without requiring architectural changes or massive human annotation. By decoupling the task of generating an answer from the task of identifying the evidence that supports it, ContextRL offers a scalable path toward more reliable agentic and multimodal systems that are less prone to hallucinating or ignoring critical contextual cues.
Large language models (LLMs) often fail when answering requires identifying a small but decisive piece of evidence within a long or complex context, such as a single line in a tool trace or a subtle detail in an image. We propose ContextRL, a context-aware reinforcement learning (RL) method that improves long-horizon reasoning and multimodal performance through an \emph{indirect} auxiliary objective. Instead of supervising only the final answer, ContextRL presents the model with a query, an answer, and two highly similar contexts, and rewards it for selecting the context that supports the query--answer pair, thereby encouraging fine-grained grounding. We construct contrastive context data in two domains: for coding agents, trajectories serve as contexts, yielding 1k pairs built via condition filtering; for multimodal reasoning, images serve as contexts, yielding 7K pairs built via generative editing and similarity search. ContextRL achieves average gains of +2.2% over standard GRPO on 5 long-horizon benchmarks, and +1.8% across 12 diverse visual question answering benchmarks. To disentangle the effect of the proposed objective from that of additional data, we compare against data-augmentation baselines that repurpose the same contrastive contexts as standard query--context--answer examples. These baselines provide little to no improvement, showing that the gains arise from the proposed context-selection objective rather than from the contrastive data alone.
Sam: Precisely. And they were careful to build these evidence-and-confounder pairs for both written text and images, so the model couldn't just learn a simple surface pattern. It has to do the genuine work of identifying what actually supports the answer.
Alex: Does this translate into better performance on real tasks?
Sam: The paper reports consistent gains across seventeen benchmarks. Crucially, the researchers also tested a simpler alternative: just giving the model more training data, without changing what it was being rewarded for. That approach provided little improvement. The gains came specifically from the new objective—from changing *what the model was asked to learn*, not just how much data it saw.
Alex: So it's not about feeding the model more information. It's about changing the standard it's held to during training.
Sam: That's the core insight. And the researchers are transparent about the boundaries of what they've shown. They only tested models of a certain size, and they focused primarily on one model family. Whether this approach holds up for much larger models, or across a wider range of architectures, remains an open question.
Alex: So it's a meaningful result, but not a universal solution yet.
Sam: That's a fair characterisation. What they've demonstrated is a proof of concept: that you can improve a model's reasoning by changing what you reward during training. Rather than only asking "did you get the answer right?", you also ask "can you show me the evidence?" That shift in incentive is what drives the improvement.
Alex: It's the difference between a student who memorises the answer key and one who understands how to work through the problem. One falls apart the moment the question changes slightly; the other can adapt.
Sam: And that distinction matters enormously for practical applications. The authors point specifically to what are called "agentic" systems—AI tools that act on your behalf to do things like write code or analyse long documents. If an agent can be trained to ground its decisions in the actual context it's been given, rather than relying on pattern-matching, the risk of it making errors because it ignored a crucial detail goes down considerably.
Alex: Which is really the goal—not just a model that's powerful, but one you can actually trust with a complex, multi-step task.
Sam: That's the direction this points toward. By ensuring the model genuinely reads the needle in the haystack, rather than guessing where it might be, we move closer to systems that are not just capable, but consistently reliable. Thanks for listening to ResearchPod.