BMFA: Boundary-Minority Free-Energy Adaptive Screening

Vision Transformers (ViTs) often rely on coarse token summaries to reduce computational complexity. However, these summaries frequently fail at object boundaries or in regions with high-response minority features. Because the softmax function used in attention is exponential, a spatially small region with high logits can dominate the attention mass, even if its contribution to a local block mean is negligible. The authors define this as "boundary-minority underestimation," a structural failure where the non-commutativity of averaging and exponential aggregation leads to a positive free-energy gap.

To address this, the authors introduce Boundary-Minority Free-Energy Adaptive Screening (BMFA). Instead of using standard importance scores or fixed budgets, BMFA constructs a hierarchical piecewise-constant approximation of the image. It recursively refines blocks based on a computable lower-bound increment of local free energy. If a block's internal heterogeneity causes the free-energy gap to exceed a threshold, the block is subdivided. This allows the model to maintain coarse, efficient representations over homogeneous background regions while allocating fine-grained resolution to critical boundaries and small objects.

BMFA was evaluated across a range of tasks, including synthetic tests, COCO and LVIS diagnostic probes, and ImageNet-1K classification. In synthetic tests, BMFA reduced the mean underestimate from 2.582 to 0.261. In ImageNet-1K experiments, the method preserved 71.520% Top-1 accuracy at a 55.861% leaf ratio, significantly outperforming random control baselines. These results confirm that model fidelity depends heavily on the ability to identify and preserve minority responses that are otherwise concealed by standard coarse summarization techniques.