The Importance of Encoder Choice: A Tabular-Image Study

Research Overview

This study investigates the critical, yet often overlooked, role of tabular encoders in multimodal learning. While image-tabular datasets are common, researchers have historically relied on simple Multi-Layer Perceptrons (MLPs) as tabular encoders. This paper evaluates state-of-the-art tabular models—specifically In-Context Learning (ICL) Tabular Foundation Models (TFMs)—as encoders in the image-tabular setting to determine if stronger tabular representations change our understanding of multimodal fusion.

The Encoder Challenge

Using ICL TFMs as encoders presents a unique technical hurdle: these models typically require labels to process instances, leading to a discrepancy between how training (context) and test (query) instances are embedded. The authors demonstrate that this 'context-query representation shift' is a structural property of ICL models, not just a data distribution issue. They evaluate three different feature extraction schemes to mitigate this shift, finding that the 'Non-partitioned' (NP) approach consistently outperforms the 'Vanilla' method.

Key Findings

1. Encoder Sensitivity: Multimodal model rankings are unstable across different tabular encoders; conclusions drawn from a single encoder often fail to generalize.
2. Fusion vs. Unimodal: In datasets where one modality is dominant, combining modalities can actually degrade performance below unimodal baselines.
3. Inflation of Benefits: Much of the performance gain attributed to complex multimodal fusion methods is actually a result of compensating for weak tabular encoders. When paired with a strong encoder, a simple bilinear fusion baseline performs on par with highly engineered, parameter-heavy multimodal architectures.

These findings suggest that the 'multimodal advantage' reported in many studies may be inflated by the use of suboptimal tabular encoders. By prioritizing the selection of a strong tabular encoder, researchers can achieve competitive results with significantly simpler and more efficient fusion pipelines, potentially reducing the need for complex, task-specific pretraining.