Structured Hyperedge Adaptation for Parameter-Efficient Fine-Tuning of Vision Transformers

Standard parameter-efficient fine-tuning (PEFT) methods for vision transformers (ViTs) typically refine each image token independently. This paper investigates whether this token-wise approach overlooks the inherent spatial and semantic structure of visual scenes, and whether modeling these relationships during adaptation can improve performance without significantly increasing the parameter budget.

The authors introduce HyperAdapter, a novel adapter architecture that shifts adaptation from the token space to a hyperedge space. Instead of applying bottleneck transformations to each token in isolation, HyperAdapter uses a soft hypergraph construction. It employs learnable prototype vectors to softly assign patch tokens to latent hyperedges based on representation similarity. These hyperedges aggregate information from related tokens, undergo lightweight bottleneck adaptation, and then diffuse the refined updates back to the original tokens via the hypergraph incidence structure. This design allows for group-aware feature updates while maintaining the modularity and efficiency of traditional adapter modules.

HyperAdapter consistently outperforms strong PEFT baselines, including standard adapters, LoRA, and prompt-based methods, across 24 diverse visual benchmarks. The authors demonstrate that HyperAdapter generalizes standard token-wise adapters, which emerge as a special case when each token is treated as an independent hyperedge. The performance improvements are particularly notable on tasks requiring structured reasoning, suggesting that injecting an explicit relational inductive bias into the adaptation process is a highly effective strategy for transfer learning in ViTs.

This work identifies the 'adaptation space' as a critical, under-explored dimension in PEFT research. By demonstrating that structured, group-level adaptation is superior to independent token refinement, the paper provides a new design principle for building more efficient and effective adaptation modules. Because HyperAdapter is a drop-in replacement that requires no changes to the frozen backbone, it offers a practical way to enhance the performance of large-scale vision models on downstream tasks.