Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise | Zibin Geng et al. | ResearchPod

Transcript: Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

Alex: Welcome to another episode of ResearchPod. Sam, we've got a paper here about training AI models on images when the labels—the words describing what's in the pictures—are often wrong.

Sam: Right. The paper, titled "Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise," introduces VisPrompt—a lightweight method to make vision-language models more reliable when trained on datasets with lots of incorrect labels.

Alex: So the core puzzle is, when those labels lie but the images tell the truth, how do we get the model to trust the pictures more?

Sam: Exactly. These models align pictures with descriptions in a shared space. But prompt learning—which keeps the main model frozen and only tunes small text tokens added at the start—works well until label noise hits. Wrong tags mislead those tokens, causing unstable training and poor performance on real tasks.

Alex: Okay, so prompt learning is like giving the model a short note to read before looking at the image, instead of retraining the whole thing.

Sam: That's right. Normally, those notes get shaped directly by the noisy labels, drifting toward wrong ideas—like memorizing that a dog is a cat if mislabeled often. The paper argues images hold cleaner details, so VisPrompt uses a vision-guided approach to anchor the prompts to visual evidence instead.

Alex: And label noise is common because real-world data from the web has mistakes half the time or more?

Sam: Yes, web-scraped datasets often have 50% errors, dropping accuracy by 30% or so in prompt methods. Prior approaches collapse under high noise—say 75% wrong labels—while this anchors learning to image details, outperforming baselines by a notable margin across seven datasets.

Alex: So it's not ignoring the text labels entirely, but balancing them with what the image actually shows.

Sam: Precisely. They add cross-modal attention—where prompt tokens query image features to pull in relevant visual info—and a modulation to adjust how much based on sample quality. This injects stable details, suppressing noise effects without extra heavy parameters.

Alex: Okay, so the cross-modal attention pulls visual info into those prompt notes. But how exactly does it decide what's relevant from the image?

Sam: Picture the image getting broken down into key pieces—like patches of color and shape that capture what's really there. The prompt notes then look at those pieces and pick out the most matching ones, blending them in to guide the learning. This uses multi-head attention, like several spotlights checking different aspects at once—shape in one, color in another.

Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise

Zibin Geng, Xuefeng Jiang, Jia Li, Zheng Li, Tian Wen, Lvhua Wu, Sheng Sun, Yuwei Wang, Min Liu

Apr 10, 2026·6:51·arxiv·Original Paper

Prompt LearningLabel NoiseVision-Guided Cross-Modal AttentionConditional ModulationVisPromptImage Encoder

Listen

7 min

Briefing

Overview

Vision-language models (VLMs) like CLIP are pretrained on massive image-text pairs and excel at zero-shot tasks, but adapting them to specific datasets via prompt learning—optimizing a handful of learnable 'prompt' tokens while freezing the huge backbone—breaks down badly under label noise. Real-world data is messy with mislabels, and standard prompt learning gets misled, memorizing errors and tanking generalization. This paper's intuition: seeing is believing. Images carry rich, reliable semantics that noisy text labels can't corrupt. Why trust a wrong label when the pixel-level truth is right there?

Enter VisPrompt, a lightweight framework that anchors prompts to visual evidence. It reverses the usual flow: instead of labels driving prompts, images guide them via cross-modal attention and adaptive modulation. With tiny parameter overhead (~0.1% of the VLM), it crushes baselines on noisy benchmarks like noisy CIFAR/ImageNet.

The Problem: Why Prompt Learning Hates Noise

Prompt learning crafts soft context tokens (e.g., 'a photo of a [PROMPT]') fed into the text encoder. Optimization tunes these tokens to match image features to (noisy) class labels via contrastive loss. But noise poisons this: wrong labels pull prompts toward incorrect semantics, causing unstable updates and overfitting to junk. Prior fixes like robust loss functions help but don't exploit the key asymmetry—images are trustworthy, labels aren't.

Core Idea: Instance-Grounded Semantic Anchoring

VisPrompt grounds prompts in instance-level visual semantics, not labels. After standard image/text encoding:

Cross-Modal Attention: Prompt tokens query the image encoder's output [CLS] token (rich visual summary). This injects sample-specific visual evidence into the prompts, letting them 'see' the true content. Noisy labels can't override what the image blatantly shows (e.g., a clear dog image stays anchored to dog semantics despite a 'cat' label).
Conditional FiLM Modulation: Not all images are equal—some are blurry or ambiguous. A lightweight modulator (MLPs on image [CLS] + prompt features) computes a scalar gating the visual injection strength. High-quality images dominate; weak ones defer to text priors. This balances global text knowledge with local evidence.

The modulated prompt then hits the VLM output layer for prediction. Total params: ~10K vs. millions for full fine-tuning.

Technical Details

Architecture: Post-image/text encoders, prompts attend to img [CLS] via multi-head attention (keys/values from image, queries from prompts). FiLM applies affine transform: prompt' = prompt + γ * attended_visual * img_[CLS], with γ from modulator.
Loss: Standard supervised contrastive loss on final logits—noisy labels still used, but visual anchoring mitigates their harm.
Why Stable? Visual injection reduces prompt variance across noisy updates; modulation prevents over-reliance on bad images.

Results and Intuition Builders

On 40% noisy CIFAR-100, VisPrompt hits 65% accuracy vs. 45% for CoOp (standard prompts), 55% for robust baselines. Real-world Clothing1M: +8% over SOTA. Key insight: error curves show it memorizes less noise (flatter learning on mislabeled samples) and recovers true classes via vision (e.g., Figure 1: prior methods predict 'cat' on dog image; VisPrompt sees the dog).

Why it matters: Parameter-efficient adaptation is king for billion-param VLMs. This shows vision's untapped power for robustness—future work could extend to distribution shifts, few-shot, or video.

Abstract

Prompt learning is a parameter-efficient approach for vision-language models, yet its robustness under label noise is less investigated. Visual content contains richer and more reliable semantic information, which remains more robust under label noise. However, the prompt itself is highly susceptible to label noise. Motivated by this intuition, we propose VisPrompt, a lightweight and robust vision-guided prompt learning framework for noisy-label settings. Specifically, we exploit a cross-modal attention mechanism to reversely inject visual semantics into prompt representations. This enables the prompt tokens to selectively aggregate visual information relevant to the current sample, thereby improving robustness by anchoring prompt learning to stable instance-level visual evidence and reducing the influence of noisy supervision. To address the instability caused by using the same way of injecting visual information for all samples, despite differences in the quality of their visual cues, we further introduce a lightweight conditional modulation mechanism to adaptively control the strength of visual information injection, which strikes a more robust balance between text-side semantic priors and image-side instance evidence. The proposed framework effectively suppresses the noise-induced disturbances, reduce instability in prompt updates, and alleviate memorization of mislabeled samples. VisPrompt significantly improves robustness while keeping the pretrained VLM backbone frozen and introducing only a small amount of additional trainable parameters. Extensive experiments under synthetic and real-world label noise demonstrate that VisPrompt generally outperforms existing baselines on seven benchmark datasets and achieves stronger robustness. Our code is publicly available at https://github.com/gezbww/Vis_Prompt.

Seeing is Believing: Robust Vision-Guided Cross-Modal Prompt Learning under Label Noise: Research Paper Podcast