Ming Wang, Yuqing Zhang, Tingna Xie, Xiangju Li, Xiaocui Yang, Daling Wang, Shi Feng, Yifei Zhang
7 min
Abstract
Creative capabilities of MLLMs matter in design, communication, education, and human--AI collaboration, yet remain difficult to evaluate because explicit targets and reward signals are scarce compared with accuracy-oriented tasks. Cross-concept understanding is a core cognitive capacity underlying receptive creativity. It enables a perceiver to recover intended meaning from non-obvious but meaningful conceptual relations. We operationalize item construction as cross-concept encoding and model inference as cross-concept decoding. We introduce C4, a cognition-inspired evaluation framework for Chengyu (Chinese idiom)-based Cross-Concept Creativity. Its encoding component maps target slots to imageable substitute concepts along bridge paths in a manually annotated and third-party-reviewed cross-concept network, enabling batch generation with explicit structure, difficulty indexed by bridge count and depth, and exact answers. Using this framework, we instantiate the C4 Evaluation Set (C4-Eval), comprising 184 synthetic items and 37 human-created cross-concept chengyu figures collected from online sources. We manually construct and review cross-concept relations, bridge paths, and reasoning processes for the collected figures. Each C4-Eval item is instantiated in five task settings, yielding 884 primary answer-recovery cases. Across ten evaluated MLLMs, the strongest closed models reach 50.7% and 48.0% primary accuracy, while open-source models remain substantially lower. Candidate constraints improve accuracy sharply, but bridge hints and explanation requests provide only modest gains. These results expose a substantial gap in how current MLLMs decode creatively encoded meaning through cross-concept relations. The code is in the supplementary material.
Alex: Which suggests that generating creative-sounding text and actually understanding the logic behind creativity are two very different things.
Sam: That's the core finding. The models aren't just missing the answer—they're missing the logic of the connection. And the researchers found a telling detail that helps explain why. When models were given a list of possible answers to choose from, their performance improved notably. That tells you something important: the knowledge is in there somewhere. The models have the right information stored away. What they can't do is actively navigate the chain of associations to reach it on their own.
Alex: So it's like having all the puzzle pieces in front of you, but not being able to see how they fit together without someone showing you the picture on the box.
Sam: That's a fair comparison. And it gets more interesting when you look at what happens when a model does guess the right answer. The researchers found that even then, models often struggled to reconstruct the correct bridge path—the actual chain of reasoning that leads from the image or clue to the idiom. They'd produce an explanation that sounds plausible, but it doesn't match the specific cultural or phonetic steps the puzzle was built on.
Alex: So it's like a student who writes down the right answer on a maths test but shows completely wrong working.
Sam: That's a very accurate comparison. The model is essentially generating a path that looks logical, without actually tracing the intended one. The researchers describe this as a kind of hallucination—not of facts, but of reasoning. The model invents a coherent-sounding explanation rather than recovering the real one.
Alex: Is this a problem that more training data could fix? Or is it something deeper in how these systems are built?
Sam: The paper suggests it's the latter. It's not that the models lack information—the candidate-list experiment showed they often have the right answer stored somewhere. The gap is in the reasoning process itself. Current systems are better at recognising patterns than at following a structured chain of logic from point A to point B to point C. That kind of step-by-step causal reasoning—tracing a bridge rather than guessing at a destination—is where they fall short.
Alex: And that's a meaningful distinction. Because if it were just a data problem, you could theoretically fix it by feeding the model more examples. But if it's a reasoning problem, the solution has to be more fundamental.
Sam: Right. And that's part of what makes this framework useful. By forcing models to show their work—to not just identify the idiom but reconstruct the path that leads to it—C4 gives developers a way to see exactly where the reasoning breaks down. It's not just a test of knowledge; it's a test of cognitive process.
Alex: Though I imagine this framework has its own limits. It's built on Chinese idioms, which are culturally very specific. Would it translate to other languages?
Sam: That's the main limitation the authors acknowledge. The strength of C4 comes from its manually annotated network of associations—people carefully mapped out which sounds, symbols, and cultural references connect to which idioms. That precision is what makes it a rigorous test. But it also means that applying this approach to English, or Arabic, or any other language would require building that entire annotation structure from scratch. It's a specialised instrument, and the authors are candid about that.
Alex: So the next challenge for the field is figuring out how to scale this kind of structured, culturally grounded benchmark across different languages and contexts.
Sam: That's where the research points. The framework itself is the contribution here—not a solution, but a more rigorous way of asking the question. Before this, measuring creative understanding in AI was largely subjective. C4 gives the field a structured, verifiable target. Whether a model truly grasps the logic of a metaphor, or is simply producing a convincing imitation of understanding—that distinction now has a way to be tested.
Alex: And that seems like a meaningful step. Not because it solves the problem, but because it makes the problem visible in a way it wasn't before.
Sam: Exactly. And the fifty percent ceiling these models hit is a useful reminder that fluency isn't the same as understanding. These systems can write, translate, and summarise with considerable skill. But tracing the inner logic of a creative leap—following the bridge from image to meaning—remains genuinely difficult for them. That gap is worth paying attention to. Thanks for listening to ResearchPod.