MMGist: A Comprehensive Multimodal Benchmark for 2027

Research Question and Approach

As large vision-language models (LVLMs) advance, existing benchmarks have become increasingly noisy and inefficient. The authors audit 23,250 items from 18 popular benchmarks and identify three systematic flaws: weak visual dependency (models can answer without looking at the image), item saturation (models solve items too easily to distinguish performance), and pseudo-hardness (items are flawed due to ground-truth errors or ambiguity). To address these, the authors propose MMGist, a curated benchmark of 7,262 items constructed through a three-stage pipeline: text-only ablation, cross-model saturation filtering, and multi-model/human anomaly detection.

MMGist significantly improves the quality of model evaluation. By focusing on items that require genuine visual reasoning and provide high discriminative power, the benchmark reduces the required evaluation items by 69% while maintaining a high Spearman correlation ($ ho = 0.98$) with the original, larger pool. The authors find that current LVLMs struggle systematically with Visual Logic, while Expert Knowledge remains a key differentiator between open-source and closed-source models. Furthermore, the average performance of models on MMGist is lower than on raw benchmarks, suggesting that previous scores were inflated by non-visual shortcuts and low-quality items.

This work shifts the focus of benchmark design from quantity to quality. By demonstrating that a smaller, curated set of items can provide more reliable and discriminative signals than a large, noisy pool, the authors provide a blueprint for more efficient and trustworthy model evaluation. This is particularly critical as the computational cost of evaluating flagship models continues to rise, making it essential to ensure that every evaluation item contributes meaningfully to our understanding of model capabilities.