Haoxiang Cao, Yuxuan Zhang, Penghui Du, Bo Li, Jiajiong Cao, Yingying Fan, Zimeng Wu, Yifu Luo, Mingzhe Zheng, Junwen Miao, Huaisong Zhang, Yang Yao, Yu Wang, Dongfu Jiang, Ping Nie, Wenhu Chen, Chaoqun Wang, Changqian Yu, Kelsey R. Allen
8 min
Abstract
We introduce LongBanana, to our knowledge the first expert-designed and expert-verified benchmark for long-context multi-reference image synthesis in the age of agent-driven image generation. The task requires retrieving dispersed visual evidence, rejecting distractors, binding entities and relations, and composing one coherent final image from multiple references. LongBanana contains 90 carefully designed tasks organized by a 3-by-3 taxonomy; each task includes about 32 reference images, 20 constraints, and 50 evaluation items, yielding 4,670 expert-specified atomic evaluation items in total. Its graph-grounded evaluation design instantiates each task with reference images and an entity-relation graph, then converts node, edge, and whole-graph obligations into checklist items for Content Fidelity, Relational Binding, and Global Constraint Satisfaction. Reviewed expert labels include a 15% double-labeled subset with 89.4% exact agreement and 0.83 quadratic weighted kappa. We further introduce LongBanana-Gen, an agent-harness framework for long-horizon image generation; experiments show that minimal agent harnesses improve over direct generation, while a ReAct-Plan-Exec-Reflect (RPR) workflow yields harness-dependent gains by strengthening evidence grounding and repair. Together, LongBanana provides an auditable testbed for measuring dense visual evidence use and a shared target for developing stronger agent-driven image-generation systems. Code and data are available at the project page: https://long-banana.com/.
Alex: Yes. Content Fidelity checks required objects and attributes, plus the absence of forbidden content. Relational Binding checks connections, including whether a particular appearance belongs to the correct object. Global Constraint Satisfaction checks scene-wide obligations such as counts and layout.
Sam: The forbidden content seems important. Does the benchmark actually provide tempting alternatives, or just ask systems not to invent things?
Alex: Some cases contain hard negatives: visually plausible references that must not be used. A typical case has about thirty-two reference images. Distinctive visual details are omitted from the text where possible, so the generator has to inspect the images.
Sam: Who decides that the checklist really matches those images? A shared graph could still encode an ambiguous task.
Alex: Experts author scenarios, and a language model structures them into graphs. Human curators then inspect the instructions, references, and checks together. They revise or reject requirements that aren't visually supported, feasible, or individually verifiable.
Sam: That makes the benchmark more inspectable. But evaluating all those checks sounds like another difficult visual reasoning task.
Alex: It is, and the authors build a separate judging system. Instead of one all-at-once assessment, it assembles evidence for each checklist item. Specialized critics inspect content, relationships, or scene-wide requirements, calling text-reading, counting, layout, or matching tools when useful.
Sam: Does a tool's answer become the verdict, or does the critic still interpret it?
Alex: The tools provide observations, not final labels. Each item is judged satisfied, partial or uncertain, or unsatisfied. For a forbidden object, satisfaction means absence. The system averages those judgments and retains an evidence trail.
Sam: An evidence trail helps auditing, but it doesn't establish accuracy. What did they compare the judge against?
Alex: Reviewed human judgments covering nine hundred generated outputs, across every benchmark case and reported generation condition. Labels went through annotation, review, and expert adjudication. Independent annotators agreed exactly on about eighty-nine percent of a double-labeled subset.
Sam: How close did the automatic judge get? Aggregate scores can agree even when individual decisions are wrong.
Alex: That distinction matters here. Exact item agreement rose to about sixty-seven percent, compared with fifty-one percent for the single-pass visual-model judge. Both used the same underlying judging model and scoring criteria, so the comparison tests the evidence organization and tool-assisted procedure.
Sam: Better, but still far from substituting for expert inspection on every requirement.
Alex: And roughly a quarter of its item judgments reversed complete satisfaction and complete failure relative to humans. Its overall scores correlated more closely with human scores than the baseline did. My reading is: useful for aggregate diagnosis, but individual explanations still need scrutiny.
Sam: Let's turn to generation. Did agents beat direct prompting because they reasoned better, or because the direct baseline was weak?
Alex: The authors address that with a strong fixed-plan baseline. They give it the construction graph, including which references belong to which entities. It optimizes how to group and order the references, then builds the image in stages without adaptively revising the plan.
Sam: That's privileged information. The agents presumably have to recover those connections themselves.
Alex: They do. They don't receive the construction graph or evaluation checklist. An agent harness—the software organizing references, prompts, generation calls, and revisions—manages that work. The image generator itself produces the pixels.
Sam: What does the more elaborate workflow add beyond that basic organization?
Alex: The authors call it RPR. It combines ReAct, which retrieves and reasons over evidence, with Plan–Exec, which divides generation into executable steps. Reflect then inspects drafts and guides further retrieval or revision. The intended benefit is coordinated grounding, execution, and repair.
Sam: With the image generator held fixed, how much did that coordination buy?
Alex: Using Nano Banana Two, OpenClaw with the full workflow scored about seventy-eight, versus seventy-five for the graph-informed direct baseline. Those are average checklist scores on a hundred-point scale. The agent configuration also passed more tasks, though still fewer than half.
Sam: Does that make the full workflow the general recommendation, or did extra revision sometimes get in the way?
Alex: No single configuration led every task category. Adding the workflow improved average scores across the tested harnesses, but task pass rates fell for some. OpenClaw's simpler version also remained stronger on temporal and state reasoning; the authors suggest over-revision can shift an intended state.
Sam: So the result isn't that more reasoning automatically helps. It's that evidence must survive the whole generation process.
Alex: That's the paper's central interpretation. The component experiments also show that retrieval, planning, and reflection aren't individually reliable improvements. And holding the workflow fixed didn't erase differences between image generators. The underlying model still determines how faithfully the plan becomes an image.
Sam: Given the judging errors and the constructed tasks, who should read the full document, and where should they begin?
Alex: Evaluation researchers should start with Checklist Scoring Protocol and Evaluator Validation, including the item-level disagreements. Agent researchers should read Generation Results and Analysis, especially the graph-informed direct comparison and workflow ablation. Those distinguish planning the right composition from controlling what actually gets generated.
Sam: And how far should everyone else carry the conclusion beyond these tasks?
Alex: Keep it scoped to dense, multi-reference final-image synthesis—not video, three-dimensional scenes, or unrestricted planning. The benchmark offers a useful diagnostic target, not proof of general reliability. The line to remember is: a realistic image is not necessarily a faithful use of evidence.
Sam: Thanks for listening.