Peixian Zhou, Yuxu Chen, Chaorui Zhang, Wei Han, Bo Bai, Xueyan Niu
5 min
This paper investigates whether the logical reasoning capabilities of large language models (LLMs), which are primarily benchmarked in English, remain robust when applied to Chinese. The authors introduce CHLOGIC, a novel, English-Chinese aligned benchmark designed to isolate the challenge of logical reasoning from the challenge of linguistic surface normalization. By using formal logical templates to generate identical logical structures in both English and five distinct Chinese surface realizations (standard, natural, colloquial, rhetorical, and perturbed), the researchers can determine if models maintain consistent logical judgments across different linguistic forms.
The experiments reveal a persistent performance gap between English and Chinese logical reasoning across all tested models, including Qwen3, Ministral, and GLM. While larger models generally show improved robustness, they still struggle significantly with Chinese-specific phenomena such as rhetorical questions, negation scope, and conditional markers. For instance, even high-performing models show substantial accuracy drops when moving from English to rhetorical Chinese variants. The authors also utilized back-translation as a diagnostic tool; while translating Chinese back into English often improved performance on general logical propositions, it produced mixed or negative results on more difficult problems, suggesting that translation itself can sometimes introduce new errors or simplify the logical complexity of the original input.
Most current logical reasoning benchmarks are English-centric and template-heavy, which may lead to an overestimation of a model's true reasoning ability. By demonstrating that models often fail to correctly interpret the underlying logical structure of Chinese expressions—even when they can solve the exact same problem in English—this study highlights that multilingual reasoning is not just about formal inference, but also about the ability to normalize diverse, pragmatically complex linguistic inputs. CHLOGIC provides a critical stress test for developers to identify specific linguistic failure modes in their models.
Large language models perform increasingly well on standardized logical reasoning benchmarks, but whether this ability remains robust beyond English is unclear. We introduce ChLogic, an English--Chinese aligned benchmark that tests whether models preserve logical reasoning performance when the same latent logical structure is expressed in English and diverse Chinese surface realizations. Built from formal logical templates, the benchmark contains three data sets: (i) the General aligned set, derived from 60 General Propositions across nine template families; (ii) the Difficult aligned set, derived from 40 Difficult Problems; and (iii) the Chinese-only set, covering 15 language-specific phenomenon types. Each aligned item pairs one English reference expression with five Chinese realizations. Experiments on Qwen3, Ministral, and GLM models reveal a persistent English--Chinese performance gap. Back-translation from standard Chinese into English often improves performance on the General aligned set, but produces mixed effects on the Difficult aligned set, where Qwen3-32B and GLM-5.1 perform worse after translation. These results indicate that Chinese surface realization, translation artifacts, and model-specific behavior jointly affect multilingual logical reasoning. Overall, ChLogic provides a useful stress test for the robustness of multilingual reasoning.
Sam: Right, so if a model takes a rhetorical question at face value, it reaches the wrong logical conclusion—even though its reasoning machinery is technically fine.
Alex: Precisely. The model's logic engine might be working perfectly, but it's been fed the wrong input because it misread the language. That's the core of what the researchers found: as linguistic complexity increases, the model's ability to translate the input into a stable logical form breaks down.
Sam: So the failure isn't in the reasoning itself—it's in the translation step that happens before the reasoning even starts.
Alex: That's the paper's central claim, yes. And to test it more directly, the researchers used a technique called back-translation. They took the Chinese questions, translated them back into English, and then asked the model to answer them again.
Sam: Did that fix things?
Alex: It helped, but not consistently. Which is actually informative. If the problem were purely about Chinese being an unfamiliar language, you'd expect back-translation to solve it almost entirely. The fact that it doesn't suggests the failure is also tied to specific structural features of Chinese—like the compact grammatical markers it uses to signal conditional relationships, which don't have direct equivalents in English.
Sam: So a model that scores well on an English logic benchmark might still be genuinely fragile in ways that only show up when you test it in a different language.
Alex: That's the key implication. A high score on an English test tells you the model can reason well under familiar linguistic conditions. It doesn't tell you whether that reasoning is robust when the language changes. The researchers' argument is that true logical robustness means navigating linguistic variety without losing the underlying truth—and by that standard, current models still have meaningful ground to cover.
Sam: It's a bit like a student who's great at maths when the problems are written clearly, but struggles when the same problem is wrapped in unfamiliar wording. The maths knowledge is there, but the reading comprehension is the weak link.
Alex: That's a fair analogy. And it points toward what better evaluation might look like—not just testing models on one language, but stress-testing the translation step itself across many linguistic styles. Thanks for listening to ResearchPod.