Yuyang Hong, Jinhui Guo, Jiaqi Gu, Lubin Fan, Ruixiang Wang, Kun Ding, Yue Wu, Shiming Xiang, Jieping Ye
6 min
Abstract
Visual instruction tuning is crucial for advancing the vision-language alignment and instruction-following capabilities of Vision-Language Models (VLMs). However, identifying optimal subsets under a fixed ratio constraint from rapidly expanding datasets remains a significant bottleneck. While existing methods largely depend on distribution diversity or heuristic filtering, they often overlook the internal coherence within individual samples. To bridge this gap, we propose Data Intrinsic Consistency (DIC), a self-scoring metric designed to quantify the sample-level inter-component consistency. DIC consists of two modules: Visual Information Consistency (VIC), evaluating the alignment between visual content and instructions, and Response Information Consistency (RIC), assessing response coherence relative to the instruction. Building upon DIC, we introduce Data Intrinsic Consistency Selection (DICS), an adaptive data selection method that optimizes the trade-off between high intra-sample consistency and global distributional diversity under varying data budgets. Extensive experiments demonstrate that DICS consistently outperforms state-of-the-art methods across diverse dataset scales and model architectures, surpassing full-dataset fine-tuning while using only 25% of the LLaVA-1.5-665K data. We further curate DICS-6M, a 6M-sample multi-modal instruction corpus that enables the largest-scale visual instruction selection study to date; remarkably, DICS reaches 94.52\% of the official InternVL3-8B-Instruct performance using less than 25\% of its reported training data. Code can be seen at https://github.com/cqu-student/DICS
Sam: And the headline performance number — maintaining over 94% of full-dataset performance with less than a quarter of the data — how load-bearing is that result?
Alex: It's the central empirical claim, and it holds across the benchmarks they report. But the important qualifier is that "94% of full-dataset performance" is a relative measure. The absolute gains depend heavily on how noisy the original dataset is. If you start with a relatively clean corpus, the filtering dividend will be smaller. The method is most compelling in the high-noise regime, which is where most large-scale multimodal datasets actually live — web-scraped image-caption pairs with weak alignment between modalities.
Sam: So the noisier the dataset, the more the DIC filter earns its keep.
Alex: Right. And the practical upside of the Weak-to-Strong result is significant — if a small model can curate data for a much larger one, you're front-loading compute into a relatively cheap scoring phase rather than burning it on training runs with degraded data. That's a meaningful shift in where the compute budget goes.
Sam: What are the honest limitations here?
Alex: Two main ones. First, the scoring cost scales linearly with dataset size — every sample needs inference-time forward passes, which isn't trivial at the scale of modern pretraining corpora. It's a favorable trade relative to wasted training compute, but it's not free. Second, the framework currently handles static image-text pairs only. Video and audio are the obvious next frontiers for multimodal learning, and it's not clear how the DIC signal would translate to temporal or sequential modalities where the grounding relationship is more complex.
Sam: There's also an interesting direction they gesture at — rather than discarding low-DIC samples, using the signal to trigger rewriting. If a sample is almost useful but the caption is misaligned, you could repair it rather than throw it away.
Alex: That's a natural extension, and it would change the framework from a filter into something more like an active curation pipeline. Whether that's practical at scale is an open question, but it's the right direction to think about — especially as the field moves toward synthetic data generation, where you have more control over what gets produced in the first place.
Sam: So the core contribution is a principled diagnostic for multimodal data quality — one that's grounded in whether visual information is actually doing inferential work, rather than just whether a sample looks diverse relative to the rest of the corpus.
Alex: That's the crux of it. The shift from "collect more data" to "verify that each sample is earning its place" is a meaningful reframing — and if the cross-architecture generalization holds up under broader scrutiny, it suggests the DIC signal is capturing something real about multimodal learning, not just overfitting to one experimental setup. Thanks for listening to ResearchPod.