Xiaofeng Shi, Xiaosong Qiu, Wenxin Ma, Qian Kou, Yiming Pan, Longbin Yu, Ying Liu, Haiping Wang, Hua Zhou
4 min
Enterprise question answering (QA) requires models to master proprietary internal knowledge without losing their general-purpose reasoning and instruction-following abilities. The authors introduce Wnuan, a three-stage post-training pipeline designed to adapt large language models to proprietary enterprise documents. The process begins by converting raw documents into structured QA pairs, followed by supervised fine-tuning (SFT) with general-data replay to preserve baseline capabilities. Finally, the pipeline applies reinforcement learning (RL) specifically to the residual errors—questions the model still fails to answer correctly after the SFT stage.
The primary 32B model trajectory demonstrates substantial gains: the acceptable-answer rate (AAR) on the custom WnuanBench increases from 52.76% (base) to 80.06% after SFT, and reaches 91.51% after residual-error RL. The authors find that while SFT drives the majority of accuracy gains, the RL stage is critical for reducing hallucinations and improving faithfulness. A controlled experiment comparing data-selection strategies confirms that focusing RL updates on residual errors outperforms both full-pool and random sampling by approximately 3 points. However, this specialization comes at a cost: general-benchmark performance drops by 5.17 points, with the most significant regressions occurring in instruction-following tasks.
This study provides a practical, modular recipe for organizations looking to deploy proprietary QA systems. By demonstrating that residual-error sampling is more efficient than training on full datasets, the authors offer a resource-conscious path for enterprise adaptation. The results highlight the inevitable tension between domain-specific accuracy and general-purpose performance, providing a clear roadmap for balancing these competing requirements in production environments.
Enterprise question answering requires models to acquire proprietary knowledge without discarding general capabilities. We present Wnuan, a three-stage pipeline that constructs task-oriented supervision from documents, performs supervised fine-tuning with general-data replay, and applies reinforcement learning to residual errors. On the 707-question WnuanBench, the primary 32B route raises acceptable-answer rate (AAR) from 52.76% before adaptation to 80.06% after SFT and 91.51% after RL. Under a matched 100-update protocol, residual-error sampling outperforms full-pool and size-matched random sampling by 3.11 and 2.97 points, respectively. Source-cluster bootstrap intervals remain above zero for both contrasts, and a same-domain validation set preserves the ordering. The general-benchmark average decreases by 5.17 points across the route, concentrated in instruction following. The automatic evaluation ensemble agrees with an authoritative domain expert on 90.5% of a stratified Wnuan-Inst response sample. These results characterize both the gains and the general-capability cost of staged enterprise adaptation.
Sam: Does that targeted focus actually move the needle?
Alex: It does, meaningfully. On their internal benchmark, the measure they call the "Acceptable-Answer Rate"—essentially, the share of questions the model answers well enough to be useful—rose from around 53% before this stage to over 91% after it.
Sam: That's a substantial shift. Did they check whether the targeted approach was actually responsible, or whether just doing more training of any kind would have produced the same result?
Alex: They did run that comparison. Working within the same fixed training budget, focusing exclusively on the failure cases outperformed training on the full question pool by around three percentage points. So it isn't simply a matter of doing more work—it's about directing the effort where it's most needed.
Sam: The strategy of which data you choose matters as much as the quantity of data. Are there any downsides to this approach?
Alex: There is a real trade-off, and the researchers are clear about it. While the model improves substantially on enterprise tasks, it does lose some of its general instruction-following ability—dropping around five points on standard public benchmarks. That's a measurable cost.
Sam: So it becomes a more specialized tool: well-suited to a company's internal needs, but somewhat less flexible for general-purpose conversation. For a dedicated enterprise system, that might be an acceptable trade.
Alex: That is exactly how the researchers frame it. It's a deliberate choice to prioritize accuracy within a specific domain, made with clear awareness of what is given up in the process. Whether that trade is worth making depends entirely on what the system is being built to do. Thanks for listening to ResearchPod.