Parsa Bakhtiari, Hassan Bashiri, Alireza Khalilipour, Masoud Nasiripour, Moharram Challenger
5 min
Industrial technical reports contain vital engineering knowledge for maintenance, troubleshooting, and product compliance, but their heterogeneous structure—combining dense prose, specifications, and complex tables—makes them difficult to index and reason over using standard QA pipelines. Existing large language models often struggle with specialized industrial data, yet few public datasets or benchmarks exist in this domain. This paper addresses the gap by introducing Industrial-Instruction, an end-to-end framework and two open datasets for building instruction-tuning and benchmark corpora from real-world industrial technical documents.
The authors use 906 publicly available Panasonic technical documents spanning 7,525 pages as a case study. The extraction pipeline leverages Dots.OCR, a layout-aware vision-language model, to preserve text and tabular content while maintaining reading order. Following extraction, the framework constructs a semantic retrieval index and synthesizes multiple-choice QA instances grounded in retrieved evidence. Each dataset explicitly models five realistic query-document relationships: irrelevant retrieval, single-document support, multi-document support, single-document answer, and multi-document answer. After automated filtering of 23.9k initial samples, each resulting dataset contains approximately 13.6k high-quality QA pairs.
To study data generation trade-offs, the authors release two parallel versions of the dataset produced by the same pipeline: one generated using the open-weight Qwen3-30B-A3B-Instruct model and one using the proprietary Claude-Opus-4.6 model. Experiments demonstrate that fine-tuning small open models (<10B parameters) on these datasets yields consistent performance gains both with and without retrieval-augmented generation. Furthermore, the Claude-generated corpus yields larger downstream fine-tuning improvements and retains general knowledge on MMLU evaluations, whereas the Qwen-generated data exhibits a slight forgetting effect, albeit at a significantly lower financial cost.
This work provides a practical, reproducible blueprint for bridging the gap between general-purpose language models and specialized industrial environments. By releasing open datasets and an end-to-end extraction and synthesis pipeline, the research enables organizations to effectively transform complex, tabular-heavy technical documentation into actionable training data for deployable, small-scale industrial language models.
Industrial technical reports contain high-value knowledge for maintenance, troubleshooting, and product engineering, but their heterogeneous structure (dense prose, specifications, tables) makes them difficult to index and reason over with standard retrieval and QA pipelines, and no public instruction-tuning or benchmark datasets are built from such documents. We address this gap with Industrial-Instruction, contributing (i) two open QA datasets built from real industrial technical reports and (ii) the end-to-end pipeline that produces them. Using 906 public Panasonic documents (7,525 pages), we apply layout-aware extraction, build a semantic retrieval index, and synthesize multiple-choice QA grounded in retrieved evidence under five query-document relationships (irrelevant retrieval, single-/multi-document support, single-/multi-document answer). After filtering an initial 23.9k generated samples, each dataset provides approximately 13.6k QA pairs with source documents and a held-out benchmark split. Fine-tuning small open LLMs (under 10B parameters) improves Set-Match Accuracy from 28.5% to 42.0% and F1 from 46.6% to 63.5% on the Panasonic benchmark. We release two parallel versions built by the same pipeline: one generated with the open-weight Qwen3-30B-A3B-Instruct model and one with the closed, API-based Claude-Opus-4.6 model, enabling a direct comparison of open- versus frontier-model data generation. The Claude-Opus-4.6 dataset yields a cleaner raw corpus and larger fine-tuning gains, at roughly two orders of magnitude higher cost. MMLU evaluation shows models trained on the Claude-Opus-4.6 data retain essentially all general knowledge, versus a small but measurable forgetting effect for the Qwen-generated data. Together, these datasets and pipeline offer a practical, reproducible path toward scalable industrial benchmarks and training data from real-world documentation.
Sam: That's a useful distinction for anyone thinking about build-versus-buy on the data side.
Alex: It is. But here's where a careful referee would push hardest: both fine-tuned models scored zero on perturbed questions in the FailureSensorIQ benchmark. Not low — zero.
Sam: That's a significant failure mode. What's actually happening there?
Alex: The models learn the content of the manuals well enough to answer direct queries, but they're brittle to any rephrasing of those queries. Change the surface form of the question without changing its meaning, and performance collapses. That's a distribution shift problem — the pipeline builds domain knowledge but doesn't teach the model to handle natural query variability. On a factory floor, where a technician might phrase the same question a dozen different ways, that's a deployment blocker.
Sam: So the gap isn't in what the model knows — it's in how flexibly it can access what it knows.
Alex: Precisely. The authors flag this directly and point toward a clear remediation path: explicitly generating adversarial and paraphrased variants of each QA pair during synthetic data construction. If the training distribution includes rephrased questions, the model should generalize better to them at inference time. It's a straightforward fix in principle, though it adds complexity to the data generation pipeline.
Sam: And presumably increases the cost of the frontier-generated data option even further.
Alex: That tension is real and the paper doesn't fully resolve it. What the study does offer is a reproducible framework — the extraction approach, the multi-scenario retrieval structure, the benchmark design — that organizations can adapt without depending entirely on closed-source APIs. The brittleness result is the honest limitation that defines how far the current version can actually be trusted.
Sam: So the contribution is a credible starting point, with a clearly identified gap that points toward the next iteration.
Alex: That's a fair read. The performance gains on smaller models are meaningful, the catastrophic forgetting analysis adds nuance, and the zero-score on perturbed questions is the finding that most constrains what you can claim. Thanks for listening to ResearchPod.