Contextualized Evaluation of Vision Language Models through Dynamic, Multi-turn Interactions

The Problem with Static Benchmarks

Most current evaluations of Multi-modal Large Language Models (MLLMs) rely on static, single-turn benchmarks. While these provide a standardized way to measure performance, they fail to capture the dynamic and interactive nature of real-world applications. In practice, users engage in multi-turn conversations where context, follow-up questions, and goal-oriented tasks are the norm. This misalignment often leads to inflated performance scores, as models may perform well on isolated, controlled tasks while failing in more complex, situated interactions.

The CEDI Framework

To address this, the authors propose CEDI (Contextualized Evaluations of MLLMs through Dynamic, multi-round Interactions). CEDI shifts the evaluation paradigm from a static test to a three-party interaction involving the evaluatee model, an automated examiner, and a grader. The examiner uses a graph-based representation of the image (a scene graph) to conduct semi-structured interviews. By navigating this graph, the examiner can adapt its strategy in real-time, moving from simple factual queries to adversarial probes and questions that contain false premises. This approach forces the model to maintain consistency over time and tests its ability to reject incorrect information.

Key Findings

Empirical testing across various models and domains demonstrates that CEDI is far more effective at uncovering visual hallucinations than traditional methods. The study shows that hallucinations are not just isolated errors but often accumulate throughout a conversation. Models frequently struggle when they must reject false premises or abstain from answering, often choosing instead to fabricate details to satisfy the conversational flow. Furthermore, the study highlights that the use of a scene-graph-based grader provides a more nuanced and accurate measure of model faithfulness compared to standard caption-based metrics.

By moving toward interactive, ecologically valid assessments, CEDI provides a more realistic diagnostic tool for developers. It highlights specific failure modes—such as vulnerability to confirmation pressure and difficulty with premise rejection—that are critical for building reliable MLLMs for high-stakes applications like autonomous driving or assistive technology.