Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.
Sam: A model answering German questions can benefit from reasoning in German, but English remains hard to beat. This study found that matching reasoning to German evidence helped compared with forcing French, not compared with default English.
Alex: That distinction matters if you're building a system over non-English documents. Who tested it, and what kind of evidence are we talking about?
Sam: It's an experimental preprint by Oliver Hauck, Mario Sanz-Guerrero, and Katharina von der Wense. They built a German question-answering testbed, then changed the language of the model's intermediate reasoning. The questions, source documents, and final answers stayed German.
Alex: So this is about language inside the process, not simply translating the output. For researchers choosing how a document-based assistant should work, that could change a design decision.
Sam: Yes, though the evidence supports testing that decision, not a universal rule. We'll unpack the comparison that makes the result meaningful, how document structure changes it, and where the evidence stops.
Alex: Start with what was unresolved. Why wouldn't matching the language already be the obvious choice?
Sam: Current reasoning models are trained to reason mostly in English. A reasoning trace is the intermediate text they generate before answering. Earlier work by Qi and colleagues found that forcing another language reduced accuracy, even when it matched the question.
Alex: But a question isn't the same as a stack of source documents. Was that earlier result tested while the model was reading retrieved evidence?
Sam: No, it was a short-prompt setting. Here the authors study retrieval-augmented generation: answering with passages fetched from an external collection. When those passages are German, much of what the model must interpret is German evidence, not just a German question.
Alex: That creates a different possibility. English reasoning might work well generally, while introducing an extra translation step between the evidence and the answer.
Sam: That's the authors' hypothesis. Their opening example is a German river name, Südask. An answer produced with English reasoning drops the umlaut, while German reasoning preserves it. It's a concrete illustration of a language artifact, not proof of the overall mechanism.
Alex: To test the broader claim, they'd need questions the model couldn't answer from memory. Otherwise retrieval might barely matter. How did they get that?
Sam: They used The Dark Eye, a tabletop role-playing game with a richly documented fictional world. Its German regional sourcebooks supplied the evidence. Without retrieval, both the study model and a larger model from the same family scored near the minimum.
Alex: So the larger model didn't rescue the answers. That makes this a useful test of reading retrieved material rather than recalling familiar facts.
Sam: That's the design's main strength. The benchmark has 585 human-checked questions answerable from a single section. A separate set contains 30 hand-written multi-hop questions, meaning questions that require combining evidence across passages.
Alex: How does the model collect those passages? A fixed bundle of evidence could behave differently from a system choosing what to search.
Sam: This is an agentic system: it can choose searches before answering. It retrieves relevant passages, adds them to its context, and can search again. With the structured collection, it can also fetch a section's parent or child by identifier.
Alex: Document structure sounds central here. What does keeping it actually give the model that a plain text slice doesn't?
Sam: Consider a short section describing market stalls and taxes. On its own, it doesn't identify the city. The authors preserve ancestor headings, which place that section in Al’Anfa. Those headings supply context that the paragraph itself lacks.
Alex: So a chunk isn't just more text. It carries the document's explanation of where that text belongs. Did they compare that against ordinary fixed-size chunks?
Sam: They did, using comparable average chunk sizes and the same source information. The structured version helped forced German reasoning most. Its improvement was significant, while the smaller improvement for unconstrained reasoning was not.
Alex: Before we interpret that, how did they force the reasoning language? Were these separately trained models?
Sam: No. They used the same Qwen model, adding an instruction and a short reasoning prefix in the chosen language. Final answers still had to be German. They also tested unconstrained reasoning, which defaults to English, and a setting with reasoning disabled.
Alex: French seems like an unusual comparison for German documents. Why not simply compare German with English?
Sam: English is the model's default reasoning language, so that comparison mixes alignment with its native reasoning preference. French is another forced, non-default language. The model's reported language benchmarks are better in French than German, making French a useful check against a simple proficiency explanation.
Alex: Then the decisive question is whether German beats French despite that proficiency disadvantage. What happened?
Sam: On the structured collection, forced German averaged about 3.70 on a five-point answer-quality scale. Forced French averaged about 3.49. That difference was statistically significant after correction for multiple comparisons, supporting the authors' alignment explanation.
Alex: That's an advantage, but it isn't evidence that German reasoning beats everything else. How did it compare with letting the model choose?
Sam: Unconstrained English averaged about 3.72. German, forced English, and unconstrained English were statistically tied on the single-passage questions. The result is that forced German reached default English performance here, not that it surpassed it.
Alex: And what about disabling reasoning? If the answer sits in one passage, extra intermediate text might just add opportunities to wander.
Sam: Disabling reasoning scored highest on those single-passage questions. On the harder questions, it tied with French for the lowest retrieval score. The authors suggest enabling reasoning according to question difficulty, but they haven't established an exact threshold.
Alex: Those results also depend on who scored the answers. Was the same evaluator judging both sets?
Sam: No. The single-passage answers were scored by a larger language model given the question, reference answer, and source section. The multi-hop answers were rated by domain experts, blind to the configuration. That harder set is small, and the reviewers disagreed enough that its rankings need caution.
Alex: An automated evaluator could prefer a certain answer style rather than better evidence use. Did the authors check that?
Sam: A human spot-check agreed exactly with the automated ratings about two-thirds of the time. There were also larger disagreements, including correct source quotations scored as hallucinations. Higher-rated answers tended to be shorter, so some advantage from disabling reasoning may reflect the judge's preference for brevity.
Alex: That weakens a blanket claim that reasoning hurts simple questions. Does it also undermine German's advantage over French?
Sam: Not in the same way. German answers were slightly longer than French answers, so the observed length bias worked against German. German also ranked above French on the human-rated harder set. That supports the ordering, although it doesn't make the small set conclusive.
Alex: Is alignment only about interpreting passages, or can the reasoning language affect which evidence the system retrieves?
Sam: The authors observed French reasoning sometimes producing French search queries against the German collection. That could hurt retrieval itself. They report it qualitatively, without measuring its frequency or separating its contribution from other effects. So the exact route from alignment to improvement remains open.
Alex: Given those caveats, who should read the full document, and where should they start?
Sam: Researchers building multilingual document assistants should start with Results and Discussion, especially German versus French and Judge Reliability and Language Artifacts. Benchmark builders should read Structure-Preserving Knowledge Base too. The authors release the benchmark and construction tools, but not the copyrighted source collection.
Alex: And what should readers resist carrying beyond this experiment?
Sam: Don't assume the result transfers to other models or low-resource languages. The system experiments use one model family and German evidence, with prompt-based control rather than training for multilingual reasoning. For everyone else, remember this: matching the evidence language can help, without making default English reasoning obsolete.
Alex: Keep the language choice testable, rather than automatic.