Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized. \textbf{Contamination mitigation evaluation} intervenes in the decoding process to suppress memorization and restore a contaminated model's genuine capability, but its prevailing metric, the \textbf{G-AP} (\textbf{G}ap of \textbf{A}ggregate \textbf{P}erformance), is flawed. Discrete correct/incorrect readouts cannot characterize per-question performance, averaging before differencing lets over- and under-suppression cancel out, and uniform per-question weighting invites strategies to push solve probabilities onto the clean model's high-frequency values. We propose \textbf{SA-PPG} (\textbf{S}tratified \textbf{A}ggregate of \textbf{P}er-question \textbf{P}robability \textbf{G}aps): estimate each question's solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model's solve probability. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate. \textbf{RailCap} instead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner-up, accumulating suppression until the response distribution becomes sufficiently dispersed. Across multiple contaminated models and benchmarks, SA-PPG reveals that prior strategies' restoration is substantially overestimated, while RailCap attains the lowest SA-PPG.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a challenge in how we measure artificial intelligence. Sam, what exactly is this paper investigating?
Sam: We're discussing something called benchmark contamination. Imagine you're studying for a test, and by accident, your teacher hands you the actual exam questions a week early. You memorize the answers—not because you understand the material, but because you've seen it before. That's essentially what happens to AI models. Test questions get accidentally included in the massive pile of data the model learns from during training. So when the model is later graded on those same questions, its scores look artificially high—masking whether it can actually reason through new problems.
Alex: So this paper argues our current grading is flawed because it rewards memorization rather than actual thinking?
Sam: Exactly. Researchers have been using a metric called the Gap of Aggregate Performance, or G-AP. Think of it like grading a class by looking at the average score. If half the students cheat and the other half fail, the average might look fine—but it hides the truth entirely.
Alex: The average masks what's really going on. So how does this new metric—the Stratified Aggregate of Per-question Probability Gaps—fix that?
Sam: The authors start by moving away from simple pass-or-fail grades. Instead, they look at what they call "solve probability"—if you ask a model the same question ten times, how often does it get it right? That gives you a richer picture than a single attempt. But here's the key insight: if you just average those probabilities across all questions, a model can still hide its weaknesses. It can ace the easy questions and fail the hard ones, and the average looks decent.
Alex: So you still have the same masking problem, just one level deeper.
Sam: Right. To fix that, they use something called stratification. Think of it like a school that sorts students into different reading levels before tracking their progress. Instead of one big average, you group questions by difficulty and measure performance within each group separately. That way, a model can't hide its failures on hard problems by getting easy ones right.
Alex: Oh—so you're comparing the model's performance against a "clean" model's performance within the same difficulty tier. It has to prove itself at every level.
Sam: Precisely. It forces the model to demonstrate genuine understanding across the board, rather than just memorizing paths to the most common answers.
Alex: That sounds like a much stricter teacher. So that's the measurement side. How does the "RailCap" strategy actually intervene to stop the memorization from happening in the first place?
Sam: Most existing methods try to guess where contamination occurred before the model even runs—which is difficult, because training data is enormous and messy. RailCap takes a different approach: it acts in real time, while the model is generating its answer. The system watches each word the model produces. When it detects the model following what the researchers call a "greedy trajectory"—meaning it's picking the single most predictable, well-worn word at every step, like reciting a memorized script—it steps in and lowers the probability of those words, nudging the model toward its second or third best options instead.
Alex: But wait—if you're suppressing the model's preferred answer, doesn't that risk pushing it toward wrong or low-quality responses?
Sam: That's exactly the right concern, and it's why the design of the intervention matters so much. A hard ban—just blocking certain words entirely—actually increases errors, because the model loses its footing and can't find its way to the right answer. RailCap instead just lowers the probability of those memorized words, keeping other options available. Think of it like a teacher telling a student, "Try putting that in your own words," rather than crossing out their answer entirely. The student still knows the material—they just can't copy the phrasing they memorized.
Alex: So the system is keeping the model's options open, just making the shortcut harder to take.
Sam: Exactly. And the trigger for when the system steps in—based on detecting repeated memorized word sequences, what the paper calls an "n-gram" threshold—turns out to be surprisingly robust. If it fires too often, it disrupts the model's genuine reasoning. If it fires too rarely, it misses the contamination. The researchers found a setting that catches the memorized patterns while leaving real problem-solving intact.
Alex: That balance sounds delicate. Is there a fundamental limit to where this can be applied?
Sam: There is. RailCap requires access to the model's internal probability scores at each step of generating an answer—what are technically called "logits"—and the ability to adjust them on the fly. If you're using a model through an API, where you only see the final output and have no visibility into the internal workings, this kind of intervention simply isn't possible. It requires what researchers call "white-box" access: being able to open the model up and observe its reasoning as it happens.
Alex: So for this to be widely adopted, model developers would need to build these guardrails directly into their systems.
Sam: That's the likely path forward. The paper suggests models could eventually include a native sense of "dynamic uncertainty"—where the system itself recognizes when it's reciting a memorized pattern versus genuinely working through a problem. That's a meaningful shift: rather than grading models on a curve and hoping the scores reflect real ability, you'd be building systems that can flag their own limitations.
Alex: It's a shift from measuring the final score to measuring the quality of the reasoning behind it. Thanks for walking me through this, Sam.
Sam: It's been a pleasure. Benchmark contamination is one of those problems that's easy to overlook precisely because the scores look fine on the surface. This work is a meaningful step toward evaluation that actually holds up. Thanks for listening to ResearchPod.