Hengyu Liu, Jihyeon Je, Ruiqi Wang, Mingxuan Wu, Haven Feng, Qixing Huang, Georgios Pavlakos, Hao Zhang, Jiajun Wu, Maneesh Agrawala, Leonidas Guibas, R. Kenny Jones, Shangzhe Wu, Guandao Yang
7 min
Abstract
Computer graphics is built on code. Large language model (LLM) agents can now generate graphics code and iteratively refine it through various feedback. This capability opens the door to a potentially new paradigm of computer graphics, with graphics algorithms increasingly built around agentic systems to automate progressively complex tasks. Over the past years, research on agentic graphics has expanded rapidly across a diverse range of tasks. This rapid growth makes it increasingly difficult for manual literature reviews to keep pace with new developments. At the same time, similar ideas are often described using different terminology across these tasks, further complicating efforts to develop a systematic understanding of the field. In this report, we address these challenges through two contributions. First, we introduce a unified taxonomy for characterizing agentic systems across diverse computer graphics tasks. Second, we develop an agentic literature system that can assist human researchers in analyzing existing and newly published papers at scale. Building on these contributions, we analyze emerging research trends and discuss open challenges in coordinating agents and tools across graphics tasks, designing interfaces and representations for human-agent collaboration, sustaining long-horizon planning and refinement, and reliably evaluating interactive graphics systems.
Alex: Can we follow one repair? It’s easier to understand those distinctions when something actually goes wrong.
Sam: The review describes LL3M creating a red fire hydrant through editable Blender programs. Blender is the graphics software used for construction and rendering. The initial hydrant lacks its top cap. Visual criticism prompts the system to add one, but another inspection finds it detached. A further edit lowers it into place.
Alex: Adding the requested component wasn’t sufficient. The system needed to inspect whether the repair actually solved the problem.
Sam: And that separates successful execution from successful construction. The program can run while the cap remains wrong. The review distinguishes rendered observations from assessments of those observations: an image supplies evidence; a diagnosis identifies what needs fixing.
Alex: But a model looking at an image can miss things too. How does this map handle errors that aren’t visually obvious?
Sam: It includes geometric checks, execution diagnostics, simulation tests, and human judgments alongside visual criticism. In computer-aided design, engineering geometry with explicit dimensions and feature relationships, CADSmith supplements images with measurements such as volume. A deterministic check rejects invalid solids regardless of the model’s assessment.
Alex: That suggests feedback and editing controls need to fit each other. Finding a defect doesn’t help much if the system cannot target it.
Sam: That connection runs through the review. A predefined content schema limits edits to what it can express. Code gives broader control, but can introduce programming errors. Mouse and keyboard actions reach existing interfaces, but their effects depend on application state, such as the current selection.
Alex: Let’s turn to the evidence. How did the authors organize a literature this broad without relying entirely on manual reading?
Sam: They built a literature agent that discovers papers, annotates them using the taxonomy, and retains evidence for later revision. It compiled a corpus of 350 papers, followed by manual verification of relevance. The collection extends through September 2026. Records include design labels, limitations, and supporting source locations.
Alex: That makes the annotations inspectable. Does the system also establish that its classifications are correct?
Sam: Not through the checks described here. It verifies required fields and whether labels follow the taxonomy. The authors explicitly say those checks do not verify classification correctness. Manual relevance checking is therefore not the same as validating every assigned label.
Alex: So I’d treat this as a structured literature map, not a definitive measurement of every design choice. What patterns does that map reveal?
Sam: Among the 304 works using observations as feedback, about three quarters use visual observations. Vision–language models, which interpret images as well as text, help make that possible. About three fifths combine multiple observation categories, rather than relying on one kind of evidence alone.
Alex: Combining evidence sounds sensible, but does the review show that it improves performance?
Sam: Those frequencies don’t establish a performance advantage. They describe system designs. The authors suggest additional measurements and simulations can reveal structural or physical problems that images miss. They also find different revision patterns: engineering design more often regenerates whole programs, while scene generation more often patches local elements.
Alex: The authors suggest dependencies between engineering operations could make local changes harder. Do those editing patterns change over the period they reviewed?
Sam: Local patching rises from half of records to about eighty-three percent across the analyzed period. That suggests a shift toward targeted revisions, not proof that patching is always better. Some systems also reject unsuccessful updates or restore checkpoints, rather than assuming each new version improves things.
Alex: We’ve mostly discussed geometry and appearance. Does the same reasoning carry over when objects move or interact?
Sam: It does, but motion adds another distinction. Artisan uses collision and stability feedback when building articulated objects, meaning objects with connected moving parts. An independent reviewer checks whether repairs preserve the intended function. For example, reducing a door’s opening range can avoid collisions while sacrificing required motion.
Alex: So passing a physical check can still mean failing the task. And matching a video needn’t mean recovering the mechanism behind it.
Sam: The review makes that explicit. A close trajectory match can leave the underlying mechanism underdetermined. It also describes failures arising from geometry, material properties, numerical settings, or the imposed interaction. An implausible simulation should not automatically trigger a material change.
Alex: Who should read the full document, and where should they begin?
Sam: Graphics-agent builders should start with Design Space of Agentic Graphics Systems, then read their application section. Evaluators should read Datasets and Benchmarks, followed by the open problem on reliable evaluation. The useful shift is to assess interaction histories and repairs, not just final outputs. For everyone else: a convincing picture is not proof of a correct construction.
Alex: Keep that distinction in mind when judging the result.