Multimodal Concept Bottleneck Models

Traditional Concept Bottleneck Models (CBMs) are designed to make deep learning models more interpretable by forcing predictions through a layer of human-understandable concepts. However, they often suffer from two major drawbacks: they are restricted to a fixed set of output classes, and they are prone to 'information leakage,' where the model bypasses the bottleneck to rely on non-interpretable features. This paper asks whether a multimodal architecture can overcome these limitations to support flexible, open-vocabulary tasks while maintaining full interpretability.

The authors propose the Multimodal Concept Bottleneck Model (MM-CBM), which extends the CBM framework into the CLIP architecture. Instead of a single bottleneck, MM-CBM utilizes dual Concept Bottleneck Layers (CBLs)—one for images and one for text. By mapping both modalities into a shared, interpretable concept space, the model performs inference by calculating the similarity between these concept representations. The training process uses an interpretability loss to align these layers with human-understandable concepts and a task-specific loss to maintain classification accuracy. Crucially, the model uses a non-negative activation function (ReLU) to promote sparsity, ensuring that predictions are grounded in a small, meaningful subset of concepts.

MM-CBM demonstrates that it is possible to achieve high performance while remaining fully transparent. The model achieves up to a 51.26% accuracy improvement over existing CBM methods across standard benchmarks and maintains performance within 5% of black-box models. By operating entirely within the concept space, the model eliminates the need for a final linear classifier, which prevents the information leakage common in previous architectures. Furthermore, the dual-modality design allows for zero-shot classification and image retrieval using natural language queries, significantly increasing the flexibility of interpretable models.