Oliver Hauck, Mario Sanz-Guerrero, Katharina von der Wense
9 min
Abstract
Reasoning traces improve large language models (LLMs), but current models are trained to reason mostly in English. It has been shown that forcing a model to reason in another language degrades accuracy, even when the reasoning language matches the language of the prompt -- but only for a setting where the model reasons over a short prompt. Here, we ask whether the same holds for retrieval-augmented generation (RAG), where the model must read and integrate a large amount of retrieved evidence in the target language. To study this, we build a fully monolingual German RAG question-answering testbed over the fictional world of the tabletop role-playing game The Dark Eye, a domain that is richly documented in German but too niche for the model to answer from memory, so that it has to rely on retrieval. Varying the forced reasoning language of an agentic RAG system on this testbed, we find that aligning the reasoning language with the language of the query and the retrieved documents helps. Forced German reasoning outperforms forced French, although the model benchmarks higher in French, so the benefit comes from alignment and not from language proficiency. The advantage grows when the retrieved context is richer and structure-aware. However, forced German only reaches the level of the model's native, unconstrained English reasoning without surpassing it, showing that native multilingual reasoning is needed. We publicly release the testbed and QA benchmark.
Alex: So the larger model didn't rescue the answers. That makes this a useful test of reading retrieved material rather than recalling familiar facts.
Sam: That's the design's main strength. The benchmark has 585 human-checked questions answerable from a single section. A separate set contains 30 hand-written multi-hop questions, meaning questions that require combining evidence across passages.
Alex: How does the model collect those passages? A fixed bundle of evidence could behave differently from a system choosing what to search.
Sam: This is an agentic system: it can choose searches before answering. It retrieves relevant passages, adds them to its context, and can search again. With the structured collection, it can also fetch a section's parent or child by identifier.
Alex: Document structure sounds central here. What does keeping it actually give the model that a plain text slice doesn't?
Sam: Consider a short section describing market stalls and taxes. On its own, it doesn't identify the city. The authors preserve ancestor headings, which place that section in Al’Anfa. Those headings supply context that the paragraph itself lacks.
Alex: So a chunk isn't just more text. It carries the document's explanation of where that text belongs. Did they compare that against ordinary fixed-size chunks?
Sam: They did, using comparable average chunk sizes and the same source information. The structured version helped forced German reasoning most. Its improvement was significant, while the smaller improvement for unconstrained reasoning was not.
Alex: Before we interpret that, how did they force the reasoning language? Were these separately trained models?
Sam: No. They used the same Qwen model, adding an instruction and a short reasoning prefix in the chosen language. Final answers still had to be German. They also tested unconstrained reasoning, which defaults to English, and a setting with reasoning disabled.
Alex: French seems like an unusual comparison for German documents. Why not simply compare German with English?
Sam: English is the model's default reasoning language, so that comparison mixes alignment with its native reasoning preference. French is another forced, non-default language. The model's reported language benchmarks are better in French than German, making French a useful check against a simple proficiency explanation.
Alex: Then the decisive question is whether German beats French despite that proficiency disadvantage. What happened?
Sam: On the structured collection, forced German averaged about 3.70 on a five-point answer-quality scale. Forced French averaged about 3.49. That difference was statistically significant after correction for multiple comparisons, supporting the authors' alignment explanation.
Alex: That's an advantage, but it isn't evidence that German reasoning beats everything else. How did it compare with letting the model choose?
Sam: Unconstrained English averaged about 3.72. German, forced English, and unconstrained English were statistically tied on the single-passage questions. The result is that forced German reached default English performance here, not that it surpassed it.
Alex: And what about disabling reasoning? If the answer sits in one passage, extra intermediate text might just add opportunities to wander.
Sam: Disabling reasoning scored highest on those single-passage questions. On the harder questions, it tied with French for the lowest retrieval score. The authors suggest enabling reasoning according to question difficulty, but they haven't established an exact threshold.
Alex: Those results also depend on who scored the answers. Was the same evaluator judging both sets?
Sam: No. The single-passage answers were scored by a larger language model given the question, reference answer, and source section. The multi-hop answers were rated by domain experts, blind to the configuration. That harder set is small, and the reviewers disagreed enough that its rankings need caution.
Alex: An automated evaluator could prefer a certain answer style rather than better evidence use. Did the authors check that?
Sam: A human spot-check agreed exactly with the automated ratings about two-thirds of the time. There were also larger disagreements, including correct source quotations scored as hallucinations. Higher-rated answers tended to be shorter, so some advantage from disabling reasoning may reflect the judge's preference for brevity.
Alex: That weakens a blanket claim that reasoning hurts simple questions. Does it also undermine German's advantage over French?
Sam: Not in the same way. German answers were slightly longer than French answers, so the observed length bias worked against German. German also ranked above French on the human-rated harder set. That supports the ordering, although it doesn't make the small set conclusive.
Alex: Is alignment only about interpreting passages, or can the reasoning language affect which evidence the system retrieves?
Sam: The authors observed French reasoning sometimes producing French search queries against the German collection. That could hurt retrieval itself. They report it qualitatively, without measuring its frequency or separating its contribution from other effects. So the exact route from alignment to improvement remains open.
Alex: Given those caveats, who should read the full document, and where should they start?
Sam: Researchers building multilingual document assistants should start with Results and Discussion, especially German versus French and Judge Reliability and Language Artifacts. Benchmark builders should read Structure-Preserving Knowledge Base too. The authors release the benchmark and construction tools, but not the copyrighted source collection.
Alex: And what should readers resist carrying beyond this experiment?
Sam: Don't assume the result transfers to other models or low-resource languages. The system experiments use one model family and German evidence, with prompt-based control rather than training for multilingual reasoning. For everyone else, remember this: matching the evidence language can help, without making default English reasoning obsolete.
Alex: Keep the language choice testable, rather than automatic.