Dive into the implicit biases of low-rank vision-language alignment

This paper investigates the vision-language alignment stage, where pretrained vision encoders are connected to large language models (LLMs). While standard practice involves full-parameter fine-tuning, the authors propose using low-rank adaptation (LoRA, LoHa, and LoKr) instead. They conduct a systematic study across over 100 configurations, including various model scales (1.4B to 14B parameters) and diverse benchmarks, to determine how low-rank constraints influence the resulting visual representations and model behavior.

• Performance Gains: Low-rank alignment consistently outperforms full-parameter alignment across perception, knowledge, and reasoning benchmarks while significantly reducing computational costs.
• The LS-Curse: The authors identify the "LS-curse," where full-parameter alignment disrupts the per-token linear separability of visual features. In contrast, low-rank alignment preserves the structured knowledge inherited from the pretrained vision tower.
• Behavioral Shift: Models aligned with low-rank methods exhibit a shift from hallucinatory to conservative behavior, reducing overconfident, incorrect predictions without sacrificing performance on positive-ground-truth tasks.
• Geometric Stability: Geometric analysis shows that low-rank aligned models maintain homogeneous and structurally stable feature manifolds, whereas full-parameter models engage in aggressive, premature fusion of entity-level semantics.

To explain these empirical observations, the authors establish two theorems. Theorem 1 demonstrates that low-rank gradient flow acts as a noise-weighted smoother, favoring updates in feature subspaces that are robust to perturbations. Theorem 2 shows that low-rank parameters naturally concentrate in subspaces characterized by flat gradients. Together, these mechanisms explain why low-rank alignment maintains a more stable, general-purpose visual representation compared to the disruptive nature of full-parameter updates.