OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining

Medical coding—assigning standardized codes to clinical documents—requires an understanding of both medical terminology and the complex relational structure of medical ontologies (e.g., hierarchical, causal, and exclusion rules). Standard language models pretrained on flat text corpora often fail to capture these relationships. The authors investigate whether synthetic, ontology-grounded textbooks can be used to inject this structural knowledge into encoder language models.

OntoBook follows a three-stage pipeline to convert ontology structure into pretraining signal:

1. Graph Traversal: The authors perform random walks through medical ontologies (CIM-10, CCAM, ATC) to capture hierarchical and causal relationships between codes.
2. Textbook Generation: A large language model (Qwen3-235B) reformulates these structured walks into fluent, textbook-style prose, ensuring the content remains grounded in the original ontology data.
3. Multi-Task Pretraining: The authors train a ModernCamemBERT encoder using two aligned objectives on this synthetic data: masked language modeling (MLM) and relation prediction between code pairs.

The authors emphasize that these two objectives must be trained on the same data (alignment) for the model to learn effectively.

OntoBook significantly outperforms standard MLM-only pretraining on three French medical coding benchmarks (FRACCO, Cantemist-FR, and Distemist-FR). The most notable improvement occurs on the Distemist-FR benchmark, where the model achieves an 8.0 micro-F1 gain. The authors demonstrate that this performance is not merely due to the synthetic text, but specifically to the multi-task alignment; misaligning the MLM and relation prediction tasks leads to a 30-point performance drop. Furthermore, the authors find that the model is robust to the loss weight assigned to the relation prediction task, but performance is sensitive to the number of training epochs, peaking at two epochs before beginning to degrade.