ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams

As AI-assisted database engineering becomes more prevalent, a critical bottleneck is the inability of AI systems to interpret conceptual database schemas—typically represented as Entity-Relationship Diagrams (ERDs)—which are usually distributed as images rather than machine-readable code. This paper investigates whether modern Vision-Language Models (VLMs) can accurately reconstruct structured, executable database schemas from these visual diagrams.

The authors introduce ERUnderstand, a large-scale benchmark containing 2,960 ER diagrams. The dataset is composed of three sources: curated educational diagrams, real-world production schemas, and synthetically generated diagrams designed to test specific structural complexities. Each diagram is paired with a standardized JSON representation. The researchers evaluate state-of-the-art VLMs using a structure-aware fractional F1-score, which provides granular insight into how well models recover specific schema elements like entities, relationships, and integrity constraints.

The study reveals a significant performance gap between simple and complex schema elements. While VLMs perform well on basic components (F1 > 0.74), their accuracy drops sharply for advanced constructs: weak entities (0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). Although reasoning-augmented models show a 15-25% improvement in overall performance, they remain highly sensitive to diagram complexity and spatial layout. The authors also identify a "complexity collapse," where performance degrades rapidly as the number of entities and relationships increases, suggesting that current models struggle with the global structural reasoning required for large-scale database design.

ERUnderstand provides the first standardized framework for evaluating how well multimodal models can perform structural reasoning in the context of database engineering. By highlighting the specific failure modes of current VLMs, this work establishes a baseline for future research aimed at building reliable AI assistants for database migration, documentation, and automated design.