Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li, Xiang Xu, Bohan Wang, Peng Wang, Xingzhe Wu, Anfeng Li, Qiyuan Feng, Yuhao Zhou, Shoulin Han, Wenjie Luo, Yiyuan Li, Yaxuan Wang, Ruixian Luo, Guojie Lin, Peiyao Xiao, Chengliang Xu, Ben Wang, Zeyu Wang, Zichao Chen, Jianan Ye, Yijie Hu, Jialong Chen, Zongwen Shen, Yuliang Xu, An Yang, Bowen Yu, Dayiheng Liu, Junyang Lin, Hu Wei, Que Shen, Bing Zhao
7 min
Abstract
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy. Our construction follows a two-stage validation-and-repair workflow resulting in a certified benchmark. In Stage I, each item undergoes binary validation of the problem and final answer through domain-expert review and model-based cross-checks, yielding 641 verified items. In Stage II, flawed but fixable items are revised under strict constraints preserving the original evaluation intent, through dual independent expert repairs, model-assisted auditing, and final adjudication, resulting in 1,170 revised-and-certified items. The remaining 689 items are released as a documented uncertain set with explicit uncertainty sources and expertise tags for future refinement. We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points on HLE-Verified. The improvement is particularly pronounced on items where the original problem statement and/or reference answer is erroneous, with gains of 30--40 percentage points. Our analyses further reveal a strong association between model confidence and the presence of errors in the problem statement or reference answer, supporting the effectiveness of our revisions. Overall, HLE-Verified improves HLE-style evaluations by reducing annotation noise and enabling more faithful measurement of model capabilities. Data is available at: https://github.com/SKYLENAGE-AI/HLE-Verified
Sam: No, the second stage fixes ones that are flawed but savable. Two teams of experts suggest targeted repairs—fix the problem wording first, then the explanation, then the answer—while keeping the core test the same, like correcting a math homework without changing the question's point. They stick to areas like math or physics where right and wrong are clear-cut. A final review picks the best fix, and the item gets rechecked.
Alex: Repairs are careful, not rewrites. And the ones they can't fix?
Sam: Those go into an uncertain group—about 689 items where things are too debatable, like needing rare expert knowledge. Each gets a note on why it's unclear. They also tag errors across all fixed items using 19 types, split by part. This tracking shows most flaws were in answers and explanations.
Alex: By breaking it down like that, they not only clean the test but make future fixes easier. Does that explain why model scores jumped so much on the bad ones?
Sam: Yes—the paper suggests targeted fixes on those clustered errors cut the noise, letting models show what they really know. It was a clear improvement.
Alex: Clustered errors... so not random slip-ups. What did the analysis show about where the problems piled up most?
Sam: The paper breaks errors into types for each part. For setups, flaws were rare, mostly missing key details or wording mix-ups. Choices had way more issues, with about 90 percent just flat-out wrong, like picking the opposite sign from what solves it right. Think chains: the end link snaps most often because earlier weak spots build up. Explanations topped the trouble list, with gaps like skipping needed steps, making the logic hard to follow.
Alex: Setups hold up best, but thinking chains and answers crumble under incompleteness or plain mistakes. Does that vary by topic, like math versus physics?
Sam: Yes, patterns shift across fields. In math and biology, answers were often clearly wrong. Physics and engineering had more uncertain cases, especially in explanations. Chemistry stayed solid on setups, but humanities leaned uncertain overall. Checking parts separately matters—lumping them hides field-specific snags.
Alex: Math errors are decisive wrongs, while others blur into "maybe." That explains the big score jumps on fixes—they targeted the weak spots precisely. To see if it really worked, they ran the cleaned test against top AI models?
Sam: Yes. They compared seven leading language models on the original test and the verified version. On the full set, scores rose 7 to 10 percentage points. The bigger shift came on the 624 flawed items they fixed: gains of 30 to 40 points, showing those errors had masked what models could do.
Alex: Twice the improvement on the bad ones. Why such a clear jump there specifically?
Sam: Flaws like wrong answers or mismatched explanations tricked models into failing items they might otherwise solve. Fixing them let the AI's logic shine through without distortion.
Alex: The noise wasn't everywhere, but where it hit, it hit hard. They also looked at something called calibration?
Sam: Models often guess how sure they are about an answer, like saying 'I'm 80% confident.' Calibration checks if that matches reality. Verified items cut the mismatch sharply. Cleaner tests give truer readouts of both skill and self-awareness. Physics and biology saw the sharpest rises, as those had more fixable mismatches.
Alex: Models weren't just missing answers—they were fooling themselves too. So this cleaning process doesn't just tweak scores—it changes which models rank highest. Like, the leaderboards flip around?
Sam: Yes, the verified version rearranges the top spots entirely. One model that trailed at around 25 percent on the raw test climbed to lead near 50 percent after fixes. This shows flaws weren't neutral—they favored certain styles of reasoning over others. On flawed problems, models' stated confidence drops noticeably, but jumps after repairs. The paper suggests this self-doubt could flag noisy items automatically.
Alex: The 'best' AI depended on the test's weak spots. Cleaner tests make the models' self-checks more trustworthy too.
Sam: The uncertain set needs more specialized input, and they skipped subjective areas to avoid bias. This keeps the benchmark honest. Overall, it highlights how noise distorts cross-field comparisons, and verification evens that out.
Alex: A solid start, but room for community help on the rest. The real payoff is this template for keeping tough tests reliable as AI gets smarter.
Sam: Precisely. It turns benchmarks into living tools—audited with metadata and open for fixes—tracking real progress without hidden flaws distorting the view.
Alex: That's a meaningful way to cut through the noise in these high-stakes tests. Thanks for breaking it down, Sam. And thanks to everyone for listening to this ResearchPod episode on cleaning up AI benchmarks.