Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
Sam: A language model can look human-like while making very different mistakes. CogBattery finds that shared patterns on cognitive tasks can hide large differences in errors and confidence.
Alex: So seeing a familiar human bias isn't enough to say a model behaves like us?
Sam: That's the central warning from this benchmark study by Dengzhe Hou and colleagues at Tohoku University and collaborating institutions. They tested 54 open-weight language models, whose weights are publicly available. The battery covered 13 cognitive tasks, each compared with published human data.
Alex: That matters if you're using psychological tasks to make claims about what models can do. A familiar effect might make the evidence look stronger than it is.
Sam: And it matters when you're deciding whether a model's confidence is useful. We'll unpack the comparisons, separate the robust findings from the cognitive interpretations, and ask how much the task adaptations limit the conclusions.
Alex: Researchers already give models cognitive tests. What does this study add beyond another collection of scores?
Sam: Earlier work, including CogBench and CogEval, already used cognitive tasks to investigate model behavior. The authors argue that previous evaluations covered few tasks or few models, and often used human data only as reference scores. Here, the emphasis is on which conditions produce which errors, across many model families.
Alex: Give me a case where that changes the conclusion. What would an overall score or a familiar effect miss?
Sam: Consider n-back, where you decide whether the current item matches one several positions earlier. In the version we'll focus on, the current letter must match the letter two positions back. A lure is a tempting match at the wrong position.
Alex: So a model could be especially vulnerable to those tempting letters, just as humans are. But that doesn't tell us how it treats letters that aren't tempting.
Sam: Both populations responded “match” more often to lures than to ordinary non-target letters. But humans falsely accepted about seven percent of ordinary non-targets. Models accepted about thirty-three percent.
Alex: That's not just the same selective mistake becoming slightly stronger. Were a few models that barely understood the task driving the average?
Sam: The excess appeared across most models and most family averages. It remained after removing models that gave nearly constant answers. It also survived a minimal-performance check requiring evidence that a model distinguished targets from non-targets.
Alex: Though clearing a minimal bar doesn't necessarily mean doing the task well.
Sam: The authors make that distinction explicitly. Many models passing the n-back check still performed poorly. The check shows that the false-alarm pattern isn't confined to completely unresponsive models; it doesn't establish task comprehension.
Alex: Does another task show the same broader lesson without relying on sequence matching?
Sam: In Stroop, you name a color while ignoring a conflicting word. The text adaptation used color labels rather than colored ink. Models made a much larger accuracy loss under conflict than humans, often answering with the distracting word itself.
Alex: That adaptation seems consequential. Naming a written label isn't the same experience as naming ink color.
Sam: That's the biggest limitation across the battery: no humans took these text versions. The observed gaps therefore combine population differences with changes to the tasks. Timing and motor demands disappeared, and materials often stayed visible.
Alex: So we shouldn't hear “working memory task” and assume the study measured a model's memory capacity.
Sam: The authors explicitly reject that inference for tasks with visible histories. In the repeated word-list task, models could consult the list throughout. Their strong initial recall and lack of improvement aren't estimates of human-like learning or retention.
Alex: That puts boundaries around the interpretation. What makes the comparisons useful despite those boundaries?
Sam: They expose response patterns that aggregate accuracy hides, under clearly specified conditions. Each task has a source-specific human reference, usually with participant-level records. But there's no single human cohort covering the entire battery, so this isn't a matched comparison of complete cognitive profiles.
Alex: Let's turn to confidence. Does the problem come down to models sounding too certain?
Sam: There's more to it. On the calibration questions, models were less accurate on average but more confident. More tellingly, their stated confidence distinguished correct answers from wrong answers less well than human confidence did.
Alex: How did they measure that distinction? Average confidence alone wouldn't tell you whether it rises on the right answers.
Sam: They used type-two area under the receiver operating characteristic curve: a measure of whether confidence ranks correct answers above incorrect ones. The average was about point five seven for models, compared with point seven one for humans. The shortfall appeared in every model family.
Alex: But a stated confidence rating is another generated answer. Did they compare it with the probabilities the model assigned to its choices?
Sam: They did, in a rerun of the direct-answer models. Those answer-token probabilities distinguished correct from incorrect responses about as well as human confidence reports overall. A subset with better probability coverage did somewhat better than humans.
Alex: That supports a reporting gap rather than a complete absence of information about correctness. It doesn't establish that models use that information to guide decisions.
Sam: That's also the authors' interpretation. The wagering task makes the distinction concrete: models classified grammar strings, then chose a low or high wager. About two-thirds always chose the same wager, including some accurate classifiers.
Alex: Could that simply be a preference for one response label rather than a failure to adjust confidence?
Sam: A follow-up swapped the labels attached to the wager amounts. Constant wagering mostly tracked the response labels, supporting label preference rather than a confidence interpretation. The practical lesson is to counterbalance those labels before treating wagers as a window into confidence.
Alex: What happened as models got larger? If size closes the gaps, this might describe a passing stage.
Sam: Larger models narrowed some gaps, including Stroop interference and confidence discrimination. But the excess n-back false alarms remained among the larger models. Across tasks, distance from human performance did not shrink consistently with size.
Alex: The title also promises something about cognitive structure. Do tasks labelled memory or control actually cluster together in model performance?
Sam: Model scores shared a broad dimension strongly associated with size, plus a weaker, unstable second dimension. The authors then asked whether cognitive task groupings helped predict scores for model families left out of training. Those groupings added no reliable predictive advantage over random groupings of the same sizes.
Alex: So the group labels describe where the tasks came from, not established divisions of model ability. But that's not proof that models lack those abilities.
Sam: Both qualifications matter. The authors also say a lack of reliable advantage doesn't establish equivalence to random groupings. These are results for this battery, with uncertainty, not a map of every possible model capability.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Researchers evaluating models or making cognitive claims should start with Behavioral Results, especially n-back and confidence. Then read Battery and Administration alongside Limitations to see what changed from the human tasks. If your interest is cognitive organization, read Score Structure for the held-out-family prediction test and its uncertainty.
Alex: And for a researcher who only needs the central lesson?
Sam: In this model panel, the recurring errors are more convincing than claims about underlying cognitive abilities. Closed models weren't tested, and each step used a single deterministic response. Keep this distinction: matching a human effect is not the same as making human mistakes.
Alex: Ask which mistakes sit underneath the score.