MathShikkha: A Controlled Study of Answer-Only and Chain-of-Thought Supervision for Bangla Mathematical Reasoning in Small Language Models

This paper investigates whether training small language models (SLMs) on step-by-step Chain-of-Thought (CoT) rationales provides a measurable benefit over simple answer-only fine-tuning in a low-resource language context (Bangla). The authors introduce MathShikkha, a dataset of Bangla mathematical problems with teacher-generated reasoning traces. To isolate the causal effect of CoT, they employ a strictly matched protocol where both the CoT and answer-only conditions share identical data splits, loss-masking strategies, and evaluation pipelines, differing only in the training target.

The study reveals that the value of CoT supervision is highly context-dependent. In-domain, CoT supervision provides no statistically significant improvement over answer-only fine-tuning for three out of four tested backbones, despite the significantly higher computational cost of generating longer reasoning sequences. However, this pattern shifts dramatically on out-of-domain benchmarks, where CoT supervision consistently outperforms answer-only training by 20–28 points. Furthermore, the authors find that answer-only fine-tuning can actually degrade out-of-domain performance compared to the base model, whereas CoT supervision acts as a protective mechanism, preserving or enhancing generalization.

For researchers and practitioners, these findings suggest that the primary utility of rationale-based distillation in low-resource settings may not be raw accuracy gains on in-distribution tasks. Instead, CoT supervision is most valuable for ensuring target-language adherence, producing auditable reasoning traces, and maintaining model robustness when applied to out-of-domain problems. The study highlights that accuracy alone can be a misleading metric, as a significant portion of correct answers in these models is supported by invalid reasoning.