Dengzhe Hou, Lingyu Jiang, Fangzhou Lin, Kazunori D Yamada
7 min
Abstract
Human cognitive tasks let us compare language models with humans under controlled conditions. Yet a model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms, comparing each with human data from its source study. Like humans, models respond more to n-back lures than to controls, yet they make far more false alarms on ordinary non-targets. Most models' reported confidence separates correct from incorrect answers less well than humans' does, and most always choose the same wager, including models that classify accurately. These patterns span many model families. Across tasks, model scores share two dimensions. The first is closely tied to scale, yet larger models are not consistently closer to humans. Predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size. These results show that matching human effect directions does not establish human-like behavior. CogBattery provides condition-level human references for testing errors and confidence reports in new models.
Alex: Though clearing a minimal bar doesn't necessarily mean doing the task well.
Sam: The authors make that distinction explicitly. Many models passing the n-back check still performed poorly. The check shows that the false-alarm pattern isn't confined to completely unresponsive models; it doesn't establish task comprehension.
Alex: Does another task show the same broader lesson without relying on sequence matching?
Sam: In Stroop, you name a color while ignoring a conflicting word. The text adaptation used color labels rather than colored ink. Models made a much larger accuracy loss under conflict than humans, often answering with the distracting word itself.
Alex: That adaptation seems consequential. Naming a written label isn't the same experience as naming ink color.
Sam: That's the biggest limitation across the battery: no humans took these text versions. The observed gaps therefore combine population differences with changes to the tasks. Timing and motor demands disappeared, and materials often stayed visible.
Alex: So we shouldn't hear “working memory task” and assume the study measured a model's memory capacity.
Sam: The authors explicitly reject that inference for tasks with visible histories. In the repeated word-list task, models could consult the list throughout. Their strong initial recall and lack of improvement aren't estimates of human-like learning or retention.
Alex: That puts boundaries around the interpretation. What makes the comparisons useful despite those boundaries?
Sam: They expose response patterns that aggregate accuracy hides, under clearly specified conditions. Each task has a source-specific human reference, usually with participant-level records. But there's no single human cohort covering the entire battery, so this isn't a matched comparison of complete cognitive profiles.
Alex: Let's turn to confidence. Does the problem come down to models sounding too certain?
Sam: There's more to it. On the calibration questions, models were less accurate on average but more confident. More tellingly, their stated confidence distinguished correct answers from wrong answers less well than human confidence did.
Alex: How did they measure that distinction? Average confidence alone wouldn't tell you whether it rises on the right answers.
Sam: They used type-two area under the receiver operating characteristic curve: a measure of whether confidence ranks correct answers above incorrect ones. The average was about point five seven for models, compared with point seven one for humans. The shortfall appeared in every model family.
Alex: But a stated confidence rating is another generated answer. Did they compare it with the probabilities the model assigned to its choices?
Sam: They did, in a rerun of the direct-answer models. Those answer-token probabilities distinguished correct from incorrect responses about as well as human confidence reports overall. A subset with better probability coverage did somewhat better than humans.
Alex: That supports a reporting gap rather than a complete absence of information about correctness. It doesn't establish that models use that information to guide decisions.
Sam: That's also the authors' interpretation. The wagering task makes the distinction concrete: models classified grammar strings, then chose a low or high wager. About two-thirds always chose the same wager, including some accurate classifiers.
Alex: Could that simply be a preference for one response label rather than a failure to adjust confidence?
Sam: A follow-up swapped the labels attached to the wager amounts. Constant wagering mostly tracked the response labels, supporting label preference rather than a confidence interpretation. The practical lesson is to counterbalance those labels before treating wagers as a window into confidence.
Alex: What happened as models got larger? If size closes the gaps, this might describe a passing stage.
Sam: Larger models narrowed some gaps, including Stroop interference and confidence discrimination. But the excess n-back false alarms remained among the larger models. Across tasks, distance from human performance did not shrink consistently with size.
Alex: The title also promises something about cognitive structure. Do tasks labelled memory or control actually cluster together in model performance?
Sam: Model scores shared a broad dimension strongly associated with size, plus a weaker, unstable second dimension. The authors then asked whether cognitive task groupings helped predict scores for model families left out of training. Those groupings added no reliable predictive advantage over random groupings of the same sizes.
Alex: So the group labels describe where the tasks came from, not established divisions of model ability. But that's not proof that models lack those abilities.
Sam: Both qualifications matter. The authors also say a lack of reliable advantage doesn't establish equivalence to random groupings. These are results for this battery, with uncertainty, not a map of every possible model capability.
Alex: Given those limits, who should read the full document, and where should they start?
Sam: Researchers evaluating models or making cognitive claims should start with Behavioral Results, especially n-back and confidence. Then read Battery and Administration alongside Limitations to see what changed from the human tasks. If your interest is cognitive organization, read Score Structure for the held-out-family prediction test and its uncertainty.
Alex: And for a researcher who only needs the central lesson?
Sam: In this model panel, the recurring errors are more convincing than claims about underlying cognitive abilities. Closed models weren't tested, and each step used a single deterministic response. Keep this distinction: matching a human effect is not the same as making human mistakes.
Alex: Ask which mistakes sit underneath the score.