Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at how errors in AI tests can quietly change which models look best. One popular test is called Humanity's Last Exam, or HLE for short. It's a set of tough questions from fields like math, science, engineering, and humanities, designed to push the limits of the smartest AI language models—these are programs that read and write human-like text to solve problems.
Alex: So it's like a super-hard final exam for AI. But why does that matter if there are mistakes in it?
Sam: Community checks found issues like unclear questions, wrong official answers, or explanations that don't match the answer. These flaws aren't spread evenly—they can tip scores in ways that don't show true AI skills, flipping leaderboard rankings and misleading what researchers build next. The paper notes this distorts comparisons between models.
Alex: A model might seem worse just because the test has a bad answer key. How bad is the effect?
Sam: They tested seven top models on the original HLE and a cleaned-up version called HLE-Verified. On average, scores rose 7 to 10 percentage points overall—and up to 30 to 40 points on the flawed items. This suggests the noise was hiding real capabilities.
Alex: Fixing the test gives a fairer picture. But how did they clean it up without changing what it tests?
Sam: They used a two-stage process. First, they broke each question into parts—the problem statement, the final answer, and the explanation—and checked each for errors with experts and AI cross-checks. Flawed but fixable ones got repaired in stage two by two experts, keeping the original goal intact, while uncertain ones were set aside. Think of it like fixing a broken recipe: check ingredients, steps, and outcome separately, then patch without rewriting the dish.
Alex: They split each question into its main parts—like the setup, the step-by-step thinking, and the final choice—and checked those separately. Walk me through what happens in the first stage exactly.
Sam: They check three parts separately: first, is the problem clear and complete, like does it give all the info needed without confusion or wrong facts? Second, is the answer right and properly written? Third, does the explanation make logical sense and line up with that answer? Experts review these, and they test top AI models by having them try solving it eight separate times to see if they reliably get the correct result. Only questions where everything checks out go straight into the reliable set.
Alex: Multiple checks to catch slip-ups. But what about questions that have issues—do they just toss them?
Sam: No, the second stage fixes ones that are flawed but savable. Two teams of experts suggest targeted repairs—fix the problem wording first, then the explanation, then the answer—while keeping the core test the same, like correcting a math homework without changing the question's point. They stick to areas like math or physics where right and wrong are clear-cut. A final review picks the best fix, and the item gets rechecked.
Alex: Repairs are careful, not rewrites. And the ones they can't fix?
Sam: Those go into an uncertain group—about 689 items where things are too debatable, like needing rare expert knowledge. Each gets a note on why it's unclear. They also tag errors across all fixed items using 19 types, split by part. This tracking shows most flaws were in answers and explanations.
Alex: By breaking it down like that, they not only clean the test but make future fixes easier. Does that explain why model scores jumped so much on the bad ones?
Sam: Yes—the paper suggests targeted fixes on those clustered errors cut the noise, letting models show what they really know. It was a clear improvement.
Alex: Clustered errors... so not random slip-ups. What did the analysis show about where the problems piled up most?
Sam: The paper breaks errors into types for each part. For setups, flaws were rare, mostly missing key details or wording mix-ups. Choices had way more issues, with about 90 percent just flat-out wrong, like picking the opposite sign from what solves it right. Think chains: the end link snaps most often because earlier weak spots build up. Explanations topped the trouble list, with gaps like skipping needed steps, making the logic hard to follow.
Alex: Setups hold up best, but thinking chains and answers crumble under incompleteness or plain mistakes. Does that vary by topic, like math versus physics?
Sam: Yes, patterns shift across fields. In math and biology, answers were often clearly wrong. Physics and engineering had more uncertain cases, especially in explanations. Chemistry stayed solid on setups, but humanities leaned uncertain overall. Checking parts separately matters—lumping them hides field-specific snags.
Alex: Math errors are decisive wrongs, while others blur into "maybe." That explains the big score jumps on fixes—they targeted the weak spots precisely. To see if it really worked, they ran the cleaned test against top AI models?
Sam: Yes. They compared seven leading language models on the original test and the verified version. On the full set, scores rose 7 to 10 percentage points. The bigger shift came on the 624 flawed items they fixed: gains of 30 to 40 points, showing those errors had masked what models could do.
Alex: Twice the improvement on the bad ones. Why such a clear jump there specifically?
Sam: Flaws like wrong answers or mismatched explanations tricked models into failing items they might otherwise solve. Fixing them let the AI's logic shine through without distortion.
Alex: The noise wasn't everywhere, but where it hit, it hit hard. They also looked at something called calibration?
Sam: Models often guess how sure they are about an answer, like saying 'I'm 80% confident.' Calibration checks if that matches reality. Verified items cut the mismatch sharply. Cleaner tests give truer readouts of both skill and self-awareness. Physics and biology saw the sharpest rises, as those had more fixable mismatches.
Alex: Models weren't just missing answers—they were fooling themselves too. So this cleaning process doesn't just tweak scores—it changes which models rank highest. Like, the leaderboards flip around?
Sam: Yes, the verified version rearranges the top spots entirely. One model that trailed at around 25 percent on the raw test climbed to lead near 50 percent after fixes. This shows flaws weren't neutral—they favored certain styles of reasoning over others. On flawed problems, models' stated confidence drops noticeably, but jumps after repairs. The paper suggests this self-doubt could flag noisy items automatically.
Alex: The 'best' AI depended on the test's weak spots. Cleaner tests make the models' self-checks more trustworthy too.
Sam: The uncertain set needs more specialized input, and they skipped subjective areas to avoid bias. This keeps the benchmark honest. Overall, it highlights how noise distorts cross-field comparisons, and verification evens that out.
Alex: A solid start, but room for community help on the rest. The real payoff is this template for keeping tough tests reliable as AI gets smarter.
Sam: Precisely. It turns benchmarks into living tools—audited with metadata and open for fixes—tracking real progress without hidden flaws distorting the view.
Alex: That's a meaningful way to cut through the noise in these high-stakes tests. Thanks for breaking it down, Sam. And thanks to everyone for listening to this ResearchPod episode on cleaning up AI benchmarks.