MythraGen: Two-Stage Retrieval Augmented Art Generation Framework

Text-to-image models often struggle to accurately capture specific artistic styles, genres, or the nuanced techniques of individual artists when prompted. The authors seek to address this by creating a framework that combines the generative power of diffusion models with the precision of retrieval-augmented generation to better align outputs with complex artistic descriptions.

MythraGen operates in two distinct stages. First, an Art Retrieval module uses BLIP-2 to encode images, captions, and metadata (genre, style, artist) into a unified feature vector. These vectors are indexed using FAISS to allow for rapid retrieval of reference images that match the user's prompt. Second, an Art Generation module uses these retrieved images to fine-tune Stable Diffusion via Low-Rank Adaptation (LoRA). The authors train two separate LoRA models—one for genre-style combinations and one for artist-style combinations—which are then mixed to produce the final image. To support this, the authors also used a VQA model to automatically label 16,452 previously unclassified images in the WikiArt dataset.

Experimental results demonstrate that MythraGen consistently outperforms baseline Stable Diffusion, BingAI, and Midjourney across standard metrics. Specifically, the framework achieves higher textual similarity (CLIP-T) and style similarity (CLIP-I) scores. The authors highlight that integrating caption and genre information into the retrieval process significantly improves the relevance of the retrieved images compared to using visual embeddings alone. User studies confirm that the generated artworks more faithfully reflect the intended content, genre, and artist style.

This work provides a practical, resource-efficient way to incorporate specific artistic knowledge into generative models without the need for full-model retraining. By leveraging retrieval, the framework allows for flexible, high-quality artistic generation that respects the unique characteristics of various artists and genres, making it a valuable tool for digital artists and designers.