ResearchPod Summary
Generative image models have demonstrated latent capabilities for visual understanding, but existing generalist perception frameworks like Vision Banana are proprietary, leaving their technical recipe and data composition opaque. This research introduces Open Vision Banana (OVB), a fully open-source and reproducible framework that turns pretrained image generators into unified perception models for tasks like semantic segmentation, depth estimation, and surface normal prediction.
OVB adapts pretrained flow-matching image generators (FLUX.2-Klein) by training them on a massive mixture of public datasets. It frames perception tasks as conditional image editing, where the generator produces RGB outputs representing labels or geometry, which are subsequently decoded via deterministic, task-specific readouts. The researchers performed systematic ablations to identify critical design choices, including the use of randomized, well-separated class-color palettes and full transformer parameter adaptation. They also introduced legend ensembling—a training-free test-time scaling method that fuses segmentation passes with varied color legends to resolve binding errors and improve accuracy.
OVB models (4B and 9B parameters) significantly outperform the zero-shot performance reported for the proprietary Vision Banana system on the Cityscapes benchmark and remain competitive across referring and reasoning segmentation tasks. Legend ensembling provides a robust, zero-retraining boost, improving mIoU by up to 8.7 points on Cityscapes and 15 points on COCO-Object. Importantly, these perception-tuned models largely retain their generative and image-editing performance, demonstrating that dense perception tuning is compatible with original creative capabilities.
Alex: An image generator can outline the right object and still give the wrong answer. Open Vision Banana shows that teaching open image generators to label scenes works, but their choice of colors can undermine accuracy.
Sam: That sounds like a distinction between recognizing something and reporting it correctly. Whose work is this, and what are they offering?
Alex: It’s a methods preprint by Arpit Jadon and colleagues, called Open Vision Banana: A Fully Open Recipe for Generative Visual Perception. They release training pipelines, code, and model weights, alongside experiments testing what makes the recipe work.
Sam: So for researchers, the attraction is not just another model score. It’s being able to examine and change an approach that previously depended on proprietary components.
Alex: That’s the practical contribution. If you work on visual understanding or image generation, this offers a reproducible starting point. We’ll unpack the painted answers, the evidence behind the headline result, and the limits that matter before using it.
Sam: Let’s start with the predecessor. What had Vision Banana already established, and what remained unresolved?
Alex: Vision Banana adapted an image generator to perform perception by rendering predictions as color images. It handled segmentation, meaning dividing an image into labeled regions, and geometry, while largely retaining image editing. But its base model, data, and code were proprietary.
Sam: So the open question wasn’t simply whether a generator could do perception. It was whether public generators could reproduce that capability, and which ingredients were responsible.
Alex: The authors tackle both. They adapt open-weight FLUX generators at different sizes, using millions of training pairs from public datasets. They also vary the output representation, training mixture, prompt design, and which model parameters get updated.
Sam: Walk me through an answer painted as an image. What turns those colors into a prediction rather than just an illustration?
Alex: For semantic segmentation, which assigns a category to each pixel, the prompt lists categories and their assigned colors. The target paints each requested category accordingly. Anything unrequested stays black. A decoding rule then maps generated pixels to the nearest requested color or background.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: The colors are carrying the labels, then. If the generator paints a good outline in the wrong color, the evaluation sees the wrong category.
Alex: And that distinction becomes central. For geometry, colors carry measurements instead. Depth uses inverse depth, where nearer points have larger values and appear brighter. Surface normals, vectors describing which direction surfaces face, put their components into separate color channels.
Sam: Does the generator need a different training system for each kind of answer?
Alex: Because every target is an image, the authors retain the base generator’s training objective. The final recipe updates the full transformer, the main trainable network, while leaving the text encoder and image-compression components frozen. Public generation and editing examples are mixed into perception training.
Sam: Before the headline score, clarify “zero-shot.” Does it mean the model has never been taught segmentation?
Alex: No. Here, zero-shot means the evaluation dataset was excluded from training. The tasks themselves were trained. The authors also train other versions using training splits from some evaluation datasets, so those results answer a different question.
Sam: With that distinction in place, what is the strongest result?
Alex: On Cityscapes, a street-scene segmentation benchmark, the larger zero-shot model reaches about seventy-four on a class-averaged overlap score. Vision Banana reported about seventy on that measure. But the Open Vision Banana result combines repeated predictions rather than relying on one generated answer.
Sam: That qualification sounds essential. How much performance is coming from repetition rather than the trained model alone?
Alex: Its mean single-pass score is about sixty-six. The reported combined score uses twenty passes. The method is called legend ensembling: repeat the task with different category-to-color assignments, decode each answer using its own legend, then take a pixel-by-pixel majority vote.
Sam: So they aren’t just generating repeatedly and hoping errors average away. They’re changing the output code while asking the same underlying question.
Alex: They keep the generation seed fixed while changing the legend. This helps correct unstable links between categories and colors. On the more category-rich COCO-Object benchmark, the reported result uses ninety passes, so the improvement carries a substantial repeated-inference cost.
Sam: What tells them this is really a color-label problem, rather than repeated generation simply producing better object boundaries?
Alex: They run a diagnostic that assigns each predicted connected region its majority ground-truth category. That removes color-label mistakes for analysis. Region quality is already high early in training, while ordinary decoding improves more gradually. After ensembling, ordinary accuracy rises, but the diagnostic region score barely changes.
Sam: That supports a narrower conclusion: much of the gain comes from reporting regions under the correct labels. It doesn’t show that repeated passes make the model understand every region better.
Alex: That’s the authors’ interpretation. Their examples of single-pass errors include whole regions assigned the wrong category, pixels left black, and small contour speckles. Ensembling reduces all of those, with larger benefits for some infrequent and thin categories.
Sam: Could they prevent the problem during training by always giving each category the same color?
Alex: Their controlled experiments show the drawback: models memorize those fixed assignments and struggle when colors change. Random assignments teach prompt following. A mixture of subset and full-category requests helps the model handle both selective questions and requests for all categories.
Sam: That makes the prompt part of the task definition, not just wording around it. What other design choice would you carry into a replication?
Alex: Use well-separated colors. The authors find that a carefully restricted palette works better overall than their full palette. They also warn that helpful choices can interact badly. The recipe is not a collection of independent improvements you can safely stack.
Sam: The semantic-segmentation story is strong, with that compute qualification. Does the rest support calling this a unified perception model?
Alex: It supports broad capability, not uniform leadership. Relative-depth results are competitive with specialists. But those predictions are aligned to ground truth separately for each image, unlike Vision Banana’s metric-depth predictions. The authors explicitly say those depth scores are not directly comparable.
Sam: And tasks that require finding a particular object or interpreting a description?
Alex: Versions trained on benchmark training splits come closer to Vision Banana on referring and reasoning segmentation, but instance segmentation remains further behind. Some evaluations also use auxiliary language-and-vision models. Reasoning queries can be rewritten into explicit descriptions, and an instance-presence gate checks whether the requested category exists.
Sam: So readers should inspect the whole evaluation pipeline, not attribute every score to the image generator. Does learning perception preserve its original generation skills?
Alex: Editing is largely retained, with some improvements and some regressions. Text-to-image scores decline across the fine-tuned models. The authors identify weaker perception in open base generators as their main limitation; these experiments don’t establish a universal replacement for specialists or proprietary generators.
Sam: Given those trade-offs, who should read the full document, and where should they begin?
Alex: Researchers adapting generators for perception should start with Analysis of Design Choices, then Test-time Scaling through Legend Ensembling. Those sections explain both how to build the recipe and why its output can fail. Before comparing scores, read the task-specific evaluation protocols for repeated passes, auxiliary models, and geometry alignment.
Sam: And for someone who just needs the central lesson?
Alex: Finding the right region and giving it the right label are different problems. A generator can learn the first before it reliably learns the second.
Sam: Keep that distinction in mind when reading the scores.