Decision models read a document and return a probability for each of a fixed set of options instead of generating text. Several appeared in 2026, but none has published results outside English. We describe Zuse, three decision models (4B, 9B and 27B) that train a low-rank adapter and a pointer head on frozen open-weight bases, targeting 27 European languages, and use them to study how this design behaves across languages and tasks. On a broad European suite of 11 sources excluded from training, covering 9 task groups and 27 languages, a development set that also guided model selection, the 27B model reaches 87.5% accuracy with a pooled expected calibration error of 0.016, and 0.035 when weighted per source. Post-hoc analyses of these single-seed results give three findings. First, miscalibration is structured by task rather than language: on parallel reading and topic items in 26 languages the confidence gap varies about as much as sampling noise, while across sources it spans -0.07 to +0.09, with knowledge tasks overconfident and acceptability judgements underconfident at all three sizes, so one fitted temperature cannot correct both. Second, the share of decisions that can be automated at a 5% error budget follows chance-corrected skill (Spearman 0.97 over 33 source and model pairs) more closely than raw accuracy (0.91). Third, reach beyond the training languages is set by the base model: across 210 languages, changing the training run moved scores by at most about 2 points within a size, while languages outside the base's documented list trail by 23 to 28 points at every size. On English public suites Zuse is behind Jev and Kev, including Kev models on the same bases. We also report two pitfalls in building multi-source training data: label-word collision between topic schemes, and an injection augmentation that taught a shortcut. Zuse can be tried at https://bineric.com/zuse.
Alex: A model’s confidence can look reliable overall and still mislead on your task. This study of multilingual decision models finds that confidence errors follow the task more than the language.
Sam: That matters if confidence determines which decisions people review. What kind of study produced that finding?
Alex: It’s a methods and evaluation preprint by Umair Mehmood Imam, founder of Bineric AI and an assistant professor at OsloMet. Called “Zuse: A Decision Model for European Languages,” it studies models that choose from supplied options instead of generating text.
Sam: So the audience includes anyone routing documents or checking policies across languages. They need more than a correct answer; they need to know when to trust it.
Alex: That’s the practical question. The study examines whether confidence can support automation, and where that promise breaks down. We’ll unpack the design, the evidence behind the confidence finding, and why the headline score needs caution.
Sam: Start with what makes this different from asking a language model to pick an answer. What changes inside the system?
Alex: The underlying language model stays frozen: its existing weights aren’t updated. Training changes an adapter, a small addition inside that model, plus a component called a pointer head. That head scores the supplied options and returns a probability for each.
Sam: So it cannot invent an answer outside the menu. But that doesn’t mean it chooses correctly, or knows when it’s wrong.
Alex: Those are separate questions. The fixed output format prevents free-form text, not mistakes. For a concrete example, the paper evaluates banking-intent classification: the model reads the input and assigns probabilities across the available intents.
Sam: Was the novelty this architecture, or taking it into languages that earlier work hadn’t evaluated?
Alex: Mainly the multilingual evaluation and the analysis around it. Earlier systems, including Jev and its open reproduction Kev, already used decision-model designs. Zuse follows Kev’s adapter approach, while the author reports finding no published per-language evaluations for those earlier systems.
Sam: How broad was the European evaluation? A multilingual claim could still rest on a narrow task.
Alex: The suite covered twenty-seven European languages. It combined reading, topics, knowledge, sentiment, banking intents, legal judgments, grammatical acceptability, and other tasks. There were about twenty thousand evaluation rows, drawn from sources the author says were excluded from training.
Sam: Excluded from training sounds reassuring. Was this also an untouched test used only after development ended?
Alex: No, and that is the central caveat. The suite guided model selection, error diagnosis, and later training decisions. Its largest model reached eighty-seven point five percent accuracy, but that is a development result, not an independent final test.
Sam: So we should treat the headline as performance on a suite the researcher learned from. What can we still learn from the confidence analysis?
Alex: First, calibration means how closely confidence matches observed correctness. The model looked well calibrated when all European decisions were pooled. But the row-weighted average of calibration errors measured separately within sources was more than twice as large.
Sam: Because being too confident on one task can cancel being too cautious on another. The combined score hides both problems.
Alex: That’s what the source-level results show. Knowledge questions tended toward overconfidence, while grammatical acceptability judgments tended toward underconfidence. Those directions held across model sizes. Adjusting all probabilities with one shared setting therefore could not correct both patterns together.
Sam: But languages had different task mixes. How did the author separate a language effect from a task effect?
Alex: Reading and topic datasets supplied parallel items across nearly all the target languages. On those matched tasks, differences in the confidence gap across languages were close to sampling noise. Across sources, the gaps were much wider.
Sam: That supports a narrower conclusion than saying language never affects confidence. Here, on matched reading and topic items, task differences stood out more.
Alex: That is the useful scope. It suggests checking calibration when moving to a new task, rather than assuming a good overall multilingual score transfers. It does not establish that every language or every future task behaves this way.
Sam: Let’s connect that to automation. If I only accept the model’s most confident decisions, how much work could it take over?
Alex: The paper sorts decisions by confidence and accepts the largest share whose observed error stays within a chosen budget. At a five percent error budget, the largest model could accept seventy-seven percent of the European suite. But the cutoff was selected on those very same decisions.
Sam: So that’s an optimistic estimate, not a promised operating point. Did the paper test whether a cutoff survives a change of evaluation split?
Alex: It did, on an English public suite. A cutoff chosen on one split produced roughly nine percent error on the other, exceeding the intended budget. That makes the distinction concrete: ranking confidence retrospectively is not the same as setting a reliable deployment threshold.
Sam: And high accuracy alone might be misleading if most examples share the same answer. Did that show up here?
Alex: On Swiss court judgments, the model scored below the baseline of always answering no. Very little of that task met the automation budget. Across sources and model sizes, automation tracked skill beyond simple guessing or majority answers more closely than raw accuracy.
Sam: What about the multilingual reach itself? Could extra decision training compensate for a weaker underlying language model?
Alex: The training comparisons suggest the base model mattered more than parameter count. Languages outside the base’s documented coverage also performed substantially worse. But these were mostly single-run comparisons, and some training settings differed, so the evidence is suggestive rather than controlled.
Sam: There’s also a safety angle. Does forcing the output into fixed options protect against instructions hidden inside the document?
Alex: It does not remove that risk. The paper studies prompt injection: instructions planted in untrusted input. An early training scheme always planted a note naming a wrong option, and the model appeared almost completely resistant.
Sam: But that construction gives away information. Avoiding whichever answer the note names could improve performance without teaching the model to ignore instructions.
Alex: That was the shortcut the author identified. Accuracy actually improved under attack. After training sometimes made the named option correct, the apparent protection weakened. The lesson is to investigate robustness results that look better under attack than under clean conditions.
Sam: Before deciding who should read this, how does the model compare with existing systems? Multilingual coverage alone doesn’t establish an overall improvement.
Alex: On shared English aggregate measures, Zuse trailed Jev and Kev. The author did not evaluate those competitors on the European suite, so relative European performance remains unknown. Training data and the full recipe are also undisclosed, preventing reproduction from the paper.
Sam: That leaves useful diagnostics, but not a clean ranking or a deployment guarantee. Who should read the full document, and where should they start?
Alex: Researchers evaluating multilingual classifiers or confidence-based automation should start with the European-suite results. Then read the automation metric definition and the limitations together. That pairing shows both the practical question and why the reported coverage is an upper bound.
Sam: And for someone building training data, rather than evaluating deployment?
Alex: Read “Analysis of Training Choices,” especially the injection shortcut and conflicting label vocabularies. Those are concrete failure modes to check in merged datasets. For everyone else, remember this: confidence that looks trustworthy in the average still needs testing on your task.
Sam: Thanks for listening.