Arpit Jadon, Valentinos Pariza, Jona Ruthardt, Ryousuke Yamada, A. Sophia Koepke, Andrei Bursuc, Yuki M. Asano
8 min
Generative image models have demonstrated latent capabilities for visual understanding, but existing generalist perception frameworks like Vision Banana are proprietary, leaving their technical recipe and data composition opaque. This research introduces Open Vision Banana (OVB), a fully open-source and reproducible framework that turns pretrained image generators into unified perception models for tasks like semantic segmentation, depth estimation, and surface normal prediction.
OVB adapts pretrained flow-matching image generators (FLUX.2-Klein) by training them on a massive mixture of public datasets. It frames perception tasks as conditional image editing, where the generator produces RGB outputs representing labels or geometry, which are subsequently decoded via deterministic, task-specific readouts. The researchers performed systematic ablations to identify critical design choices, including the use of randomized, well-separated class-color palettes and full transformer parameter adaptation. They also introduced legend ensembling—a training-free test-time scaling method that fuses segmentation passes with varied color legends to resolve binding errors and improve accuracy.
OVB models (4B and 9B parameters) significantly outperform the zero-shot performance reported for the proprietary Vision Banana system on the Cityscapes benchmark and remain competitive across referring and reasoning segmentation tasks. Legend ensembling provides a robust, zero-retraining boost, improving mIoU by up to 8.7 points on Cityscapes and 15 points on COCO-Object. Importantly, these perception-tuned models largely retain their generative and image-editing performance, demonstrating that dense perception tuning is compatible with original creative capabilities.
Recent work suggests that pretrained image generators encode real-world understanding beyond image synthesis. Vision Banana exploits this by framing vision tasks as conditional image editing. However, its data, weights, code, and implementation details are not public. We present Open Vision Banana (OVB), a fully open and reproducible recipe for converting pretrained generators into unified multi-task perception systems. Our analyses identify the design choices that matter most for generalization, including color encoding, prompt design, parameter adaptation, and task balancing. These choices interact, so individually helpful choices can hurt when combined with others. Starting from 4B and 9B parameter open-weight base models and using 2.1M-3.8M training pairs from public datasets, OVB surpasses Vision Banana's reported Cityscapes zero-shot performance (73.9 vs. 69.9 mIoU) and rivals state-of-the-art affine-invariant depth estimators. Its non-zero-shot variants come within five points of Vision Banana on referring and reasoning segmentation but remain further behind on instance segmentation. All models largely retain the base model's image-editing abilities. With legend ensembling, we introduce a test-time scaling method for generator-based perception. It segments an image under several class-color legends and fuses the results by majority vote, correcting unstable color-class bindings and improving mIoU by up to 8.7 points on Cityscapes and 15.0 points on COCO-Object without retraining. We release the dataset construction and processing pipeline, the training and evaluation code, as well as the final, intermediate, and ablation checkpoints. Project website: https://openvisionbanana.github.io
Sam: Does the generator need a different training system for each kind of answer?
Alex: Because every target is an image, the authors retain the base generator’s training objective. The final recipe updates the full transformer, the main trainable network, while leaving the text encoder and image-compression components frozen. Public generation and editing examples are mixed into perception training.
Sam: Before the headline score, clarify “zero-shot.” Does it mean the model has never been taught segmentation?
Alex: No. Here, zero-shot means the evaluation dataset was excluded from training. The tasks themselves were trained. The authors also train other versions using training splits from some evaluation datasets, so those results answer a different question.
Sam: With that distinction in place, what is the strongest result?
Alex: On Cityscapes, a street-scene segmentation benchmark, the larger zero-shot model reaches about seventy-four on a class-averaged overlap score. Vision Banana reported about seventy on that measure. But the Open Vision Banana result combines repeated predictions rather than relying on one generated answer.
Sam: That qualification sounds essential. How much performance is coming from repetition rather than the trained model alone?
Alex: Its mean single-pass score is about sixty-six. The reported combined score uses twenty passes. The method is called legend ensembling: repeat the task with different category-to-color assignments, decode each answer using its own legend, then take a pixel-by-pixel majority vote.
Sam: So they aren’t just generating repeatedly and hoping errors average away. They’re changing the output code while asking the same underlying question.
Alex: They keep the generation seed fixed while changing the legend. This helps correct unstable links between categories and colors. On the more category-rich COCO-Object benchmark, the reported result uses ninety passes, so the improvement carries a substantial repeated-inference cost.
Sam: What tells them this is really a color-label problem, rather than repeated generation simply producing better object boundaries?
Alex: They run a diagnostic that assigns each predicted connected region its majority ground-truth category. That removes color-label mistakes for analysis. Region quality is already high early in training, while ordinary decoding improves more gradually. After ensembling, ordinary accuracy rises, but the diagnostic region score barely changes.
Sam: That supports a narrower conclusion: much of the gain comes from reporting regions under the correct labels. It doesn’t show that repeated passes make the model understand every region better.
Alex: That’s the authors’ interpretation. Their examples of single-pass errors include whole regions assigned the wrong category, pixels left black, and small contour speckles. Ensembling reduces all of those, with larger benefits for some infrequent and thin categories.
Sam: Could they prevent the problem during training by always giving each category the same color?
Alex: Their controlled experiments show the drawback: models memorize those fixed assignments and struggle when colors change. Random assignments teach prompt following. A mixture of subset and full-category requests helps the model handle both selective questions and requests for all categories.
Sam: That makes the prompt part of the task definition, not just wording around it. What other design choice would you carry into a replication?
Alex: Use well-separated colors. The authors find that a carefully restricted palette works better overall than their full palette. They also warn that helpful choices can interact badly. The recipe is not a collection of independent improvements you can safely stack.
Sam: The semantic-segmentation story is strong, with that compute qualification. Does the rest support calling this a unified perception model?
Alex: It supports broad capability, not uniform leadership. Relative-depth results are competitive with specialists. But those predictions are aligned to ground truth separately for each image, unlike Vision Banana’s metric-depth predictions. The authors explicitly say those depth scores are not directly comparable.
Sam: And tasks that require finding a particular object or interpreting a description?
Alex: Versions trained on benchmark training splits come closer to Vision Banana on referring and reasoning segmentation, but instance segmentation remains further behind. Some evaluations also use auxiliary language-and-vision models. Reasoning queries can be rewritten into explicit descriptions, and an instance-presence gate checks whether the requested category exists.
Sam: So readers should inspect the whole evaluation pipeline, not attribute every score to the image generator. Does learning perception preserve its original generation skills?
Alex: Editing is largely retained, with some improvements and some regressions. Text-to-image scores decline across the fine-tuned models. The authors identify weaker perception in open base generators as their main limitation; these experiments don’t establish a universal replacement for specialists or proprietary generators.
Sam: Given those trade-offs, who should read the full document, and where should they begin?
Alex: Researchers adapting generators for perception should start with Analysis of Design Choices, then Test-time Scaling through Legend Ensembling. Those sections explain both how to build the recipe and why its output can fail. Before comparing scores, read the task-specific evaluation protocols for repeated passes, auxiliary models, and geometry alignment.
Sam: And for someone who just needs the central lesson?
Alex: Finding the right region and giving it the right label are different problems. A generator can learn the first before it reliably learns the second.
Sam: Keep that distinction in mind when reading the scores.