Jiahao Zhao, Xiaomin Yu, Zhongxiang Sun, Fengwei Teng, Chengwei Qin, Xiaobin Hu, Jun Xu, Shuicheng Yan
4 min
Text-to-image (T2I) models can produce visually compelling images, yet they remain limited on open-world tasks that require complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing efforts introduce agent capabilities into image generation, but they either prescribe a fixed workflow or place only a subset of the open-world image generation process under agent control. Consequently, reasoning, tool invocation, and image generation are not coordinated by a single policy. We propose ToolArtist, a fully agentic image generation model obtained by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within one unified policy. During Supervised Fine-Tuning (SFT), we equip a teacher agent with search tools alongside an image-generation tool. We then convert the collected trajectories into a UMM compatible format, where the image-generation tool is concealed while the resulting generated images are retained. During Reinforcement Learning (RL), we develop an agentic RL infrastructure for UMMs and introduce Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model. Experiments show that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. We release the training data and the complete post-training infrastructure.
Traditional text-to-image (T2I) models often struggle with open-world tasks that require factual accuracy, multi-step reasoning, or external knowledge. Existing agentic approaches typically rely on rigid, predefined workflows or separate the reasoning/search process from the actual image generation. ToolArtist addresses this by creating a fully agentic model where the Unified Multimodal Model (UMM) itself autonomously decides when to reason, search for information, and generate images.
ToolArtist is built on the Emu3.5 UMM architecture and trained in two distinct stages. First, the researchers performed Supervised Fine-Tuning (SFT) using 7,132 high-quality trajectories. In this stage, a teacher agent used external search and generation tools; the researchers then converted these trajectories into a unified format where the model learns to perform the entire process—reasoning, tool invocation, and image generation—natively. Second, they introduced Reason-Act-Draw GRPO (RAD-GRPO), a reinforcement learning framework that optimizes the policy using two complementary rewards: an Intent Reward (evaluating the generation prompt) and a Quality Reward (evaluating the final image's faithfulness and aesthetics).
ToolArtist demonstrates that unifying the entire generation pipeline under a single policy leads to superior performance on complex, knowledge-intensive tasks. In evaluations on the WISE and WorldGenBench-Humanities benchmarks, ToolArtist consistently outperformed both general-purpose image generators and models that only partially incorporate agentic capabilities. By allowing the model to dynamically decide when to search and when to draw, it effectively bridges the gap between factual knowledge retrieval and visual synthesis.
This work shifts the paradigm of image generation from static prompt-to-pixel mapping to an interactive, agentic process. By enabling models to autonomously gather evidence and refine their own visual output, ToolArtist provides a scalable path toward reliable, fact-aware image generation that can handle diverse, real-world queries that are impossible for standard T2I models to solve.
Sam: That's a useful way to put it. It also sounds like the difference between a student who guesses on a test and one who actually looks up the answer. Is this approach more effective than simply using a more powerful model?
Alex: It's a different kind of improvement. A more powerful model still draws only on what it was trained on, and even very capable models can confidently produce incorrect details—what researchers call "hallucination." By forcing the model to search for current, specific information and verify it before drawing, you're grounding the output in real-world data rather than trained intuition. That's a more reliable approach for factual or historically specific requests.
Sam: You mentioned the search step—how does that actually work in practice?
Alex: The model queries external sources and then uses a language model to summarize and filter those results into something usable. The paper tested what happened when they removed that summarization step entirely. Factual accuracy dropped noticeably. That finding suggests the "thinking and filtering" stage is doing real work—it's not just decoration around the image generation.
Sam: So the comprehension step is load-bearing. Without it, you're back to guessing.
Alex: That's a good way to frame it. And there's one more piece worth understanding—how the teacher's demonstrations get converted into something the model can actually train on. The process takes the teacher's full sequence of steps—the reasoning, the searches, the drawing decisions—and encodes images as special placeholder tokens within that sequence. This lets the model read the entire process as one long, coherent story, rather than treating the image as something separate that appears at the end.
Sam: So it's not just seeing a finished image. It's learning the whole chain of decisions that produced it.
Alex: Exactly. And that's what makes the approach meaningful. By training on those complete, detailed sequences, the model learns to plan—from the first search query all the way to the final output. It's a step toward AI systems that are more reliable partners for work that requires factual precision, not just visual fluency.
Sam: That's a genuinely interesting direction. Thanks for walking us through it.
Alex: Thanks for listening to ResearchPod.