In this work, we present Canvas360, a two-stage framework for in-context panoramic generation that combines geometry-aware pretraining with downstream task-specific fine-tuning. To address the lack of large-scale, high-quality training data tailored to in-context panoramic tasks, we propose Canvas360Dataset, a collection of 1M high-quality paired panoramic samples for style transfer, inpainting, outpainting, and editing, enabling effective supervision across diverse in-context generation scenarios. On the modeling side, Canvas360 enhances text-to-panorama generation through parallel depth generation, velocity circular padding, and similarity loss regularization, enabling the model to learn geometry-aware representations, capture object distortion details, and improve geometric consistency and global coherence. Furthermore, empowered by strong panoramic priors, Canvas360 enables a unified in-context panoramic generation framework that supports diverse downstream tasks via token-level concatenation, surpassing prior methods in both task coverage and modeling flexibility. Extensive experiments show that Canvas360 improves panoramic image fidelity, achieving particularly strong performance on the panorama-specific FAED metric and competitive or leading results across the reported quantitative evaluations. More information can be found on our project page: https://zry000.github.io/Canvas360/
Alex: Welcome to another episode of ResearchPod. Today, we're discussing Canvas360, a new two-stage framework for generating and editing 360-degree panoramic images using artificial intelligence.
Sam: So, this paper is basically asking how we can make AI-generated panoramas look more realistic and consistent, especially when we try to edit them?
Alex: Exactly. Think about how a panoramic photo works: you're taking a full spherical view of a room and flattening it onto a rectangle, the way you might peel an orange and try to press the skin flat. That flattening process creates distortions. Traditional AI image tools weren't designed with that in mind, so when they try to edit a panorama, you get warped furniture, stretched walls, or visible seams where the edges don't quite match up.
Sam: I've definitely seen those weird, stretched-out edges in panoramic photos. So how does Canvas360 actually fix that?
Alex: They treat it as a geometry problem rather than just a visual one. Before the AI learns to paint a scene, it first learns to read the room's depth—how far away each surface is. It's like teaching an artist to sketch the floor plan and measure the distances before they pick up a paintbrush. That way, the furniture doesn't end up floating or melting into the walls.
Sam: That makes sense. So the model is building a mental map of the space first. How do they actually teach it to do that?
Alex: They train it on a large dataset of one million paired samples—each one containing both a panoramic image and a corresponding depth map. The model has to generate both at the same time, which forces it to connect what things look like with where they sit in 3D space. You can't fake the depth map if you don't actually understand the geometry.
Sam: But panoramas have a really specific quirk, right? The left edge and the right edge of the flat image are actually the same point in the real world. How does the model handle that?
Alex: That's one of the trickier problems. They solve it with a technique called velocity circular padding. Imagine you're tiling a floor with a pattern that has to match seamlessly at every edge—this technique does the same thing for the AI's internal calculations. It synchronizes the information at the left and right boundaries so the model understands the image wraps around into a continuous loop, rather than just stopping at a hard edge.
Sam: So it's essentially teaching the model that the image is a sphere, not a rectangle. Does that understanding then carry over to editing tasks, like if you wanted to remove an object from the scene?
Alex: Yes, and that's where the two-stage design pays off. The first stage—geometry-aware pretraining—builds that structural understanding of 3D space. The second stage then fine-tunes the model on specific tasks like inpainting, which is the technical term for removing something from an image and filling in what should be behind it. Because the model already understands the room's geometry, it can remove a sofa, for instance, and fill the gap with a plausible floor and wall, without the edges warping or the room's proportions going wrong.
Sam: How does it handle combining different kinds of instructions? Like, what if you want to say "make this room look like a Japanese tea house" and also show it a reference photo?
Alex: They use a method called token-level concatenation. Here's a way to think about it: imagine you're giving someone directions, and instead of saying the address and the landmark separately, you hand them a single note that has both pieces of information written together in order. The model reads text descriptions and reference images as one combined sequence of data, so it can weigh both sources of guidance at the same time rather than switching back and forth between them.
Sam: That sounds like it gives the system a lot more flexibility. What did the experiments actually show?
Alex: The study reports meaningful improvements on metrics specifically designed to evaluate panoramic images—particularly boundary consistency, which is how well the left and right edges of the panorama match up. That's a practical gain for anyone building virtual reality environments or digital mockups of physical spaces.
Sam: There's one more piece you mentioned earlier—something about giving the AI a built-in compass?
Alex: Right. They also use a technique called Rotary Positional Embeddings. Every pixel in a flat image has a coordinate, but in a panorama, those coordinates need to map back to a position on a sphere. This technique injects that spherical coordinate information directly into the model's calculations, so it always knows where it is in 3D space, not just where it is on the flat image. Without it, the model could lose its spatial bearings in the most distorted parts of the panorama.
Sam: So it's like giving the AI a GPS coordinate for every part of the room, not just a grid reference on a flat map.
Alex: That's a good way to put it. And it's especially important at the edges and the top and bottom of the panorama, where the distortion from flattening the sphere is most severe.
Sam: The paper mentions some remaining limitations, though. What does the model still struggle with?
Alex: The authors are candid about this. The model still has difficulty with two specific categories: human faces and text—things like signs or labels on objects. Both tend to appear in the most geometrically distorted regions of the panoramic map, and the training dataset simply doesn't contain enough high-quality panoramic examples of those things for the model to have learned them well. It's a dataset imbalance problem more than a fundamental flaw in the approach.
Sam: So it's not that the method is wrong, it's that the model hasn't seen enough of the right examples yet?
Alex: Exactly. The authors flag expanding and rebalancing the training data as the primary direction for future work. They also note the broader responsibility that comes with tools like this—the ability to generate convincing synthetic environments raises real questions about misuse, and they emphasize the need for safeguards like watermarking and human review in any real-world deployment.
Sam: It's a measured view of the work—clear about what it achieves and honest about where it falls short. The core insight seems to be that if you want an AI to edit a space convincingly, you first have to teach it to understand that space as a three-dimensional object, not just a flat picture.
Alex: That's a good summary. Geometry-first thinking turns out to be the key to visual consistency. Thanks for listening to ResearchPod.