Haoran Feng, Ruiyang Zhang, Longyi Zhang, Dizhe Zhang, Lu Qi
6 min
Abstract
In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/
Sam: How does it handle combining different kinds of instructions? Like, what if you want to say "make this room look like a Japanese tea house" and also show it a reference photo?
Alex: They use a method called token-level concatenation. Here's a way to think about it: imagine you're giving someone directions, and instead of saying the address and the landmark separately, you hand them a single note that has both pieces of information written together in order. The model reads text descriptions and reference images as one combined sequence of data, so it can weigh both sources of guidance at the same time rather than switching back and forth between them.
Sam: That sounds like it gives the system a lot more flexibility. What did the experiments actually show?
Alex: The study reports meaningful improvements on metrics specifically designed to evaluate panoramic images—particularly boundary consistency, which is how well the left and right edges of the panorama match up. That's a practical gain for anyone building virtual reality environments or digital mockups of physical spaces.
Sam: There's one more piece you mentioned earlier—something about giving the AI a built-in compass?
Alex: Right. They also use a technique called Rotary Positional Embeddings. Every pixel in a flat image has a coordinate, but in a panorama, those coordinates need to map back to a position on a sphere. This technique injects that spherical coordinate information directly into the model's calculations, so it always knows where it is in 3D space, not just where it is on the flat image. Without it, the model could lose its spatial bearings in the most distorted parts of the panorama.
Sam: So it's like giving the AI a GPS coordinate for every part of the room, not just a grid reference on a flat map.
Alex: That's a good way to put it. And it's especially important at the edges and the top and bottom of the panorama, where the distortion from flattening the sphere is most severe.
Sam: The paper mentions some remaining limitations, though. What does the model still struggle with?
Alex: The authors are candid about this. The model still has difficulty with two specific categories: human faces and text—things like signs or labels on objects. Both tend to appear in the most geometrically distorted regions of the panoramic map, and the training dataset simply doesn't contain enough high-quality panoramic examples of those things for the model to have learned them well. It's a dataset imbalance problem more than a fundamental flaw in the approach.
Sam: So it's not that the method is wrong, it's that the model hasn't seen enough of the right examples yet?
Alex: Exactly. The authors flag expanding and rebalancing the training data as the primary direction for future work. They also note the broader responsibility that comes with tools like this—the ability to generate convincing synthetic environments raises real questions about misuse, and they emphasize the need for safeguards like watermarking and human review in any real-world deployment.
Sam: It's a measured view of the work—clear about what it achieves and honest about where it falls short. The core insight seems to be that if you want an AI to edit a space convincingly, you first have to teach it to understand that space as a three-dimensional object, not just a flat picture.
Alex: That's a good summary. Geometry-first thinking turns out to be the key to visual consistency. Thanks for listening to ResearchPod.