Real Data Closes Synthetic-to-Real Gap in Optical Chemical Structure Recognition

Optical Chemical Structure Recognition (OCSR) is a critical task for converting chemical drawings from patents and journals into machine-readable formats like SMILES. While modern Vision-Language Models (VLMs) like Qwen2.5-VL-7B achieve high accuracy on synthetic datasets (often >90%), they struggle significantly when applied to real-world documents, with accuracy frequently dropping below 16%. This study investigates how to close this gap by systematically varying training data composition, base model architecture, and adaptation strategies.

The researchers found that the most reliable way to improve performance on real-world documents is to include labeled real-world images in the training mixture. In a controlled experiment, adding real data consistently outperformed other interventions such as increasing synthetic data volume, using more diverse rendering toolkits, or applying reinforcement learning. For instance, increasing the real-data fraction from 0% to 50% significantly boosted exact-match accuracy on real-world benchmarks like ACS and USPTO.

The study highlights that there is no "one-size-fits-all" approach to model adaptation. While applying LoRA to the vision tower is a common strategy, its impact varies wildly by base model. For some models, vision-tower adaptation provides substantial gains, while for others, it provides no benefit or even hinders performance. Furthermore, the researchers observed that the relative ranking of different base models shifts as the proportion of real training data increases. This suggests that researchers cannot simply pick a "best" base model in isolation; instead, the model architecture and the training data mixture must be optimized together.

As chemical research relies increasingly on automated data extraction from vast archives of literature and patents, the ability to accurately recognize chemical structures is paramount. This paper demonstrates that relying solely on synthetic data for training is insufficient for real-world deployment. By identifying that real-world supervision is the primary driver of performance, this work provides a clear roadmap for building more robust OCSR systems that can handle the noise and stylistic variability of actual scientific documents.