ReDesign: Recovering Editable Design Structures from Images via Agentic Decomposition

Recovering editable design files (like Figma or Illustrator files) from static raster images is a fundamental bottleneck in design workflows. Because rasterization is a many-to-one mapping, recovering the original structure—including typography, vector geometry, and layer ordering—is an ill-posed inverse problem. This paper asks how an agentic system can reliably decompose a flat image into a structured, editable hierarchy that behaves as a designer would expect.

ReDesign treats reconstruction as a structured tree-expansion task rather than a linear sequence of operations. A Vision-Language Model (VLM) controller acts as an agent that selects from a library of specialized tools (e.g., OCR for text, vectorization for shapes, and segmentation for objects) to expand the design hierarchy.

To ensure reliability, the authors introduce "graceful verification." At every node expansion, a verifier checks if the proposed children accurately represent the parent region. If the expansion is flawed, the system can accept, prune, or retry the operation. This local feedback loop prevents the accumulation of errors that typically plague long, serial tool-use pipelines, allowing the system to correct mistakes before they propagate through the entire reconstruction process.

The authors introduce the Figma Edit Replay Benchmark, which includes 909 raw design files and nearly 15,000 controlled edit instructions (e.g., recoloring, repositioning, and text rewriting). ReDesign consistently outperforms existing layered decomposition baselines and standard serial tool-use agents. By maintaining a structured hierarchy, the model ensures that edits are localized to the correct elements, preventing the "bleeding" effects or structural corruption common in methods that do not explicitly recover layer relationships.

This work moves beyond simple image-to-image translation by focusing on the functional requirements of professional design software. By enabling the automatic recovery of editable layers, ReDesign significantly reduces the manual labor required to repurpose static assets, providing a path toward more efficient, automated design handoff and modification workflows.