Decompose, Compare, and Decide: Multimodal LLMs are Implicit Few-Shot Learners

Multimodal Large Language Models (MLLMs) excel at general vision-language tasks but often struggle with few-shot image classification when used off-the-shelf. Standard in-context prompting—where support images and labels are provided in a single prompt—frequently fails to leverage support examples effectively, often relying on memorized semantic priors rather than visual evidence. This paper investigates whether a structured inference approach can enable MLLMs to perform robust few-shot learning without additional training.

The authors propose DeCoDe (Decompose, Compare, Decide), a training-free framework that reformulates few-shot classification as a set of binary decisions. Instead of asking the model to classify a query image among N options, the system performs pairwise comparisons: for each support image, the MLLM is prompted to decide whether it and the query image belong to the same class. The logit score of the affirmative ('Yes') response is used as a similarity metric. The final class prediction is determined by aggregating these scores across all support examples for each candidate class. The authors further enhance this by providing high-level domain information (e.g., 'bird species' or 'yoga pose') to ground the comparison in the specific context of the dataset.

DeCoDe significantly improves few-shot performance across both standard benchmarks and novel, out-of-domain datasets. In settings where class labels are anonymized (e.g., 'Class 1' instead of 'dog'), standard in-context learning performance collapses, whereas DeCoDe maintains high accuracy, demonstrating that the method forces the model to rely on visual correspondence rather than textual memorization. The addition of domain-specific context further boosts performance, consistently establishing new state-of-the-art results for training-free few-shot classification. The authors also show that DeCoDe scales more reliably than standard prompting as the number of classes (N-way) increases.