LONGBANANA: An Expert-Verified Benchmark for Long-Context Multi-Reference Image Synthesis

Alex: A convincing image can still use the wrong evidence. A new benchmark tests that gap, finding that adaptive generation workflows help, but polished pictures remain unreliable evidence of correctness.

Sam: If the picture looks convincing, what would make it wrong in a way researchers should care about?

Alex: It might copy the wrong handbag, attach a material to the wrong object, or include something explicitly excluded. This is LONGBANANA, a benchmark preprint by Haoxiang Cao and colleagues. It tests making one image from many visual references.

Sam: So this concerns anyone evaluating image generators, not just people trying to make them look better.

Alex: Especially researchers building agents that select references and revise images. The benchmark asks whether those systems preserve evidence, not just appearance. We'll unpack its scoring idea, compare fixed plans with adaptive workflows, and ask how much to trust the results.

Sam: What was missing from existing benchmarks? Reference-based image generation already had evaluations.

Alex: The authors describe earlier work on editing, personalization, and following text instructions. More recent benchmarks do combine references, but typically with fewer images or less dense cross-image requirements. Here, correctness depends on retrieving, rejecting, and connecting evidence scattered across a large reference set.

Sam: Give me a task where that distinction becomes unavoidable.

Alex: Their opening example is a handbag advertisement. A retailer wants structured leather bags with dual parallel top handles, arranged on a boutique shelf. The system must select the relevant bags, exclude similar-looking alternatives, and preserve their individual appearance.

Sam: A beautiful shelf of leather handbags could therefore fail because it contains the wrong products. The rendering isn't necessarily the problem.

Alex: The authors call these provenance errors: mistakes about where generated content comes from. A system can reproduce an object category while borrowing its details from the wrong reference. Their benchmark contains ninety cases, each designed to expose specific failures like that.

Sam: How do they turn that into a reproducible test rather than an expert saying something feels wrong?

Alex: The central device is an entity–relation graph: a map of required things and their connections. It also records requirements applying to the whole scene. Each task instruction, reference set, and evaluation checklist comes from that shared structure.

Sam: For the handbags, that would distinguish the bags themselves from their attributes and the requested arrangement.

Alex: Yes. Content Fidelity checks required objects and attributes, plus the absence of forbidden content. Relational Binding checks connections, including whether a particular appearance belongs to the correct object. Global Constraint Satisfaction checks scene-wide obligations such as counts and layout.

Sam: The forbidden content seems important. Does the benchmark actually provide tempting alternatives, or just ask systems not to invent things?

Alex: Some cases contain hard negatives: visually plausible references that must not be used. A typical case has about thirty-two reference images. Distinctive visual details are omitted from the text where possible, so the generator has to inspect the images.

Sam: Who decides that the checklist really matches those images? A shared graph could still encode an ambiguous task.

Alex: Experts author scenarios, and a language model structures them into graphs. Human curators then inspect the instructions, references, and checks together. They revise or reject requirements that aren't visually supported, feasible, or individually verifiable.

Sam: That makes the benchmark more inspectable. But evaluating all those checks sounds like another difficult visual reasoning task.

Alex: It is, and the authors build a separate judging system. Instead of one all-at-once assessment, it assembles evidence for each checklist item. Specialized critics inspect content, relationships, or scene-wide requirements, calling text-reading, counting, layout, or matching tools when useful.

Sam: Does a tool's answer become the verdict, or does the critic still interpret it?

Alex: The tools provide observations, not final labels. Each item is judged satisfied, partial or uncertain, or unsatisfied. For a forbidden object, satisfaction means absence. The system averages those judgments and retains an evidence trail.

Sam: An evidence trail helps auditing, but it doesn't establish accuracy. What did they compare the judge against?

Alex: Reviewed human judgments covering nine hundred generated outputs, across every benchmark case and reported generation condition. Labels went through annotation, review, and expert adjudication. Independent annotators agreed exactly on about eighty-nine percent of a double-labeled subset.

Sam: How close did the automatic judge get? Aggregate scores can agree even when individual decisions are wrong.

Alex: That distinction matters here. Exact item agreement rose to about sixty-seven percent, compared with fifty-one percent for the single-pass visual-model judge. Both used the same underlying judging model and scoring criteria, so the comparison tests the evidence organization and tool-assisted procedure.

Sam: Better, but still far from substituting for expert inspection on every requirement.

Alex: And roughly a quarter of its item judgments reversed complete satisfaction and complete failure relative to humans. Its overall scores correlated more closely with human scores than the baseline did. My reading is: useful for aggregate diagnosis, but individual explanations still need scrutiny.

Sam: Let's turn to generation. Did agents beat direct prompting because they reasoned better, or because the direct baseline was weak?

Alex: The authors address that with a strong fixed-plan baseline. They give it the construction graph, including which references belong to which entities. It optimizes how to group and order the references, then builds the image in stages without adaptively revising the plan.

Sam: That's privileged information. The agents presumably have to recover those connections themselves.

Alex: They do. They don't receive the construction graph or evaluation checklist. An agent harness—the software organizing references, prompts, generation calls, and revisions—manages that work. The image generator itself produces the pixels.

Sam: What does the more elaborate workflow add beyond that basic organization?

Alex: The authors call it RPR. It combines ReAct, which retrieves and reasons over evidence, with Plan–Exec, which divides generation into executable steps. Reflect then inspects drafts and guides further retrieval or revision. The intended benefit is coordinated grounding, execution, and repair.

Sam: With the image generator held fixed, how much did that coordination buy?

Alex: Using Nano Banana Two, OpenClaw with the full workflow scored about seventy-eight, versus seventy-five for the graph-informed direct baseline. Those are average checklist scores on a hundred-point scale. The agent configuration also passed more tasks, though still fewer than half.

Sam: Does that make the full workflow the general recommendation, or did extra revision sometimes get in the way?

Alex: No single configuration led every task category. Adding the workflow improved average scores across the tested harnesses, but task pass rates fell for some. OpenClaw's simpler version also remained stronger on temporal and state reasoning; the authors suggest over-revision can shift an intended state.

Sam: So the result isn't that more reasoning automatically helps. It's that evidence must survive the whole generation process.

Alex: That's the paper's central interpretation. The component experiments also show that retrieval, planning, and reflection aren't individually reliable improvements. And holding the workflow fixed didn't erase differences between image generators. The underlying model still determines how faithfully the plan becomes an image.

Sam: Given the judging errors and the constructed tasks, who should read the full document, and where should they begin?

Alex: Evaluation researchers should start with Checklist Scoring Protocol and Evaluator Validation, including the item-level disagreements. Agent researchers should read Generation Results and Analysis, especially the graph-informed direct comparison and workflow ablation. Those distinguish planning the right composition from controlling what actually gets generated.

Sam: And how far should everyone else carry the conclusion beyond these tasks?

Alex: Keep it scoped to dense, multi-reference final-image synthesis—not video, three-dimensional scenes, or unrestricted planning. The benchmark offers a useful diagnostic target, not proof of general reliability. The line to remember is: a realistic image is not necessarily a faithful use of evidence.

Sam: Thanks for listening.