Metric-Guided Synthetic Image Data Rendering for Deep Learning compatible with Agentic AI

Deep learning for scientific image analysis is often bottlenecked by the high cost and labor required to create large, annotated datasets. While synthetic data generation via 3D rendering offers a potential solution, the process is often subjective, lacking systematic quantitative guidance to bridge the domain gap between synthetic and real-world images. This paper investigates how to quantitatively measure and optimize synthetic data realism and diversity to improve model performance.

The researchers developed a procedural pipeline using Blender to generate synthetic images of virological plaque assays. To guide the generation process, they introduced the GraNatPy Python package, which provides metrics for realism (Fréchet Inception Distance, image naturalness) and diversity (Shannon entropy). They evaluated these metrics by training an instance segmentation model on synthetic datasets of varying quality and testing its zero-shot performance on real, human-annotated images. Finally, they implemented an agentic skill, SynthClaw, to automate the optimization of procedural rendering parameters via natural language.

The study confirms that increasing dataset size, realism, and diversity generally improves zero-shot performance. However, the researchers discovered that for small object detection, gradient similarity (measured by the naturalness factor) is a critical predictor of model success. Most notably, mixing a small fraction of real data with synthetic data yielded superior performance compared to using large synthetic datasets alone, suggesting that realism and gradient alignment are more impactful than sheer dataset size.