PRISM: Prompt Refinement via Image-grounded Self-rewarding Mechanism for Text-to-Image Generation

Text-to-image (T2I) generation models are highly sensitive to prompt formulation, yet existing optimization methods often rely on text-only rewriting or scalar rewards that lack deep visual understanding. This paper asks: can we create a closed-loop system where a model learns to diagnose its own generated images and use that visual feedback to iteratively refine prompts for better semantic alignment and aesthetic quality?

The authors propose PRISM, a framework that integrates prompt rewriting and visual feedback assessment into a single Vision-Language Model (VLM). The process occurs in two stages:

1. Supervised Initialization: The VLM is fine-tuned on a multi-task dataset to perform prompt rewriting, visual diagnosis, and scoring across three dimensions: semantic consistency, aesthetic quality, and human preference.
2. Self-Rewarding Optimization: The model generates multiple candidate prompts, renders them using a frozen T2I generator, and evaluates the results. It then uses a hybrid reward function—combining ideal-point and Chebyshev objectives—to update the prompt policy. This hybrid approach encourages overall quality improvements while specifically penalizing the weakest performance dimension.

Experimental results on benchmarks like BeautifulPrompt and T2I-CompBench demonstrate that PRISM consistently outperforms existing text-only and visual-feedback-based methods. By incorporating structured visual diagnosis, the model produces prompts that lead to higher CLIP scores, better aesthetic ratings, and improved human preference alignment. The study highlights that learning a reusable optimization policy via self-rewarding is more effective than relying on test-time-only feedback or fragmented evaluation metrics.

PRISM bridges the gap between text-based prompt engineering and visual evaluation. By enabling models to "see" and critique their own output, it provides a scalable way to improve image generation quality without needing constant human intervention or external reward models. This approach offers a path toward more controllable and faithful T2I generation systems that can adapt to specific user preferences.