Shufan Li, Konstantinos Kallidromitis, Akash Gokul, Yuta Kyuragi, Aditya Grover
6 min
Abstract
Group-advantage-based reinforcement learning methods, such as GRPO and DAPO, have demonstrated strong performance across diverse domains, including mathematical reasoning and text-to-image generation. However, their reliance on sample-level rewards introduces a key limitation as uniform credit assignment across all tokens fails to capture fine-grained, token-level contributions. To address this issue, we propose Guidance Contrastive Policy Optimization (GCPO), a novel algorithm that enables per-token credit assignment by contrasting model predictions under positive and negative prompts. Rather than uniformly broadcasting sample-level advantages, GCPO assigns token-level advantages proportional to the difference between these contrastive predictions, allowing more precise and informative learning signals. Empirically, we find that GCPO emphasizes semantically relevant regions such as visual areas aligned with textual prompts in text-to-image generation, and critical keywords within reasoning traces for chain-of-thought tasks. Through extensive experiments, GCPO consistently outperforms GRPO and DAPO baselines on both text-to-image generation and chain-of-thought reasoning benchmarks, demonstrating its effectiveness as a general and scalable optimization strategy for discrete policy learning.
Sam: That's where the team found a practical workaround. These models are designed to follow instructions, so they don't naturally have an "empty" mode to contrast against. Instead, the researchers simply add the phrase "please generate a wrong answer" to the end of the prompt. This forces the model to show what it thinks an incorrect response looks like — and by comparing that to the correct response, the system can identify which parts are actually doing the important work.
Alex: That's a clever trick. It's a bit like asking a student to deliberately show the wrong method, so you can see which steps really separate the right approach from the wrong one. But does the model need that "wrong answer" prompt when it's actually being used in the real world?
Sam: No — and that's an important point. The "wrong answer" prompt is only used during training, not during actual use. It's like training with extra weights at the gym. You don't wear them during the game, but the strength you built stays with you. The model learns where to focus its attention during training, and that focus carries over into real-world performance.
Alex: That makes sense. Now, you mentioned that these importance scores — the measurements of how much each word changes — can vary wildly. Why is that a problem, and how do they fix it?
Sam: It's a problem because without any adjustment, the system might fixate on a single outlier word — often just the very first word of a response, which tends to score unusually high for technical reasons unrelated to meaning. That would be like a teacher giving an A just because a student used a fancy font on the title page. To prevent this, the researchers use a technique called histogram equalization. Imagine you have a pile of test scores ranging from ten to ten thousand — all over the place. This method lines them up by rank and maps them onto a consistent, predictable scale. A word in the top ten percent of importance always gets the same weight in training, regardless of whether its raw score was huge or tiny.
Alex: So it's less about the absolute size of the score and more about where it falls relative to everything else. That sounds much more stable than just cutting off scores above a fixed limit.
Sam: Exactly. It ensures the spotlight is spread fairly across the whole response, rather than being dominated by one unusual data point. And the evidence from the paper suggests this stability translates to real improvements — particularly on tasks involving complex reasoning, like solving math problems or interpreting logic in images.
Alex: We've covered how the method works. But I'm curious about where it falls short. If the whole system depends on that "wrong answer" prompt, what happens if the model doesn't respond to it the way you expect?
Sam: That's a meaningful limitation the paper acknowledges. The effectiveness of this approach is sensitive to how that negative prompt is designed. It needs to accurately reflect the model's own internal sense of what an incorrect answer looks like. If the prompt fails to trigger that specific "wrong" belief state — maybe because it's too vague, or the model interprets it differently — then the comparison becomes noisy, and the spotlight loses its precision.
Alex: So the quality of the whole method depends on the quality of that one prompt. You're essentially relying on the model to already have some sense of what "wrong" looks like, just so you can teach it what "right" looks like.
Sam: Exactly. It requires careful engineering of that negative prompt. If it's poorly chosen, the importance weights assigned to tokens won't actually align with the logical steps needed to solve the problem. It's a real trade-off — the method is powerful, but it takes effort to set up correctly.
Alex: Do you think the field will eventually move past needing humans to craft these specific negative prompts?
Sam: That seems like the logical next step. Future work could focus on automating the discovery of optimal prompts — essentially allowing models to generate their own contrastive training data. That would remove the human engineering step and make the entire training loop more self-contained.
Alex: So GCPO is less about changing what the model is, and more about changing how precisely we teach it — pointing to the specific moments in its reasoning that actually matter. That's a meaningful shift in how we think about training these systems. Thanks for listening to ResearchPod.