Huachen Gao, Yang Yang, Xiaoqian Liang, Xingyu Chen, Youjia Zhang, Qi Zhang, Baorui Ma, Anpei Chen, Wei Yang
9 min
Abstract
Geometric Relative Positional Encoding has become the primary mechanism for injecting spatial inductive biases into multi-view transformers. Existing designs condition attention on camera-level frustums or ray-marched depth, associating cross-view tokens through the sensor geometry. Such sensor-centric formulations, however, describe only where a token is observed from, leaving the rich surface geometry of the scene region depicted by the token. In this paper, we advocate a shift toward surface-aware positional encoding that explicitly grounds patch tokens in the geometry of physical surfaces. We introduce HRoPE, which represents the local geometry of each patch as a slanted 3D plane, following the slanted support-window assumption of classical multi-view stereo. We inject token-level, plane-induced homographies into attention as relative transformations. This equips the transformer with a geometric prior for rectifying perspective distortions and aligning patch features across viewpoints. Our encoding requires no ground-truth geometry: the plane parameters can be predicted end-to-end under a view-synthesis objective alone, or derived from foundation models. Extensive experiments on RealEstate10K, CO3Dv2, and DL3DV show that our best variant of HRoPE consistently outperforms the evaluated geometric positional encodings for novel view synthesis. Extensive ablations and analysis show the effectiveness of the proposed encoding. Please refer to our project page at: https://gaohchen.github.io/HRoPE
Alex: What does “inside attention” mean operationally? Are the authors moving pixels around before the model sees them?
Sam: No. They transform the feature vectors that attention compares. Each patch gets its own transform, and the comparison uses one patch’s transform composed with the inverse of the other’s. The authors interpret this as an attention-level analogue of comparing neighborhoods after aligning them to a shared plane.
Alex: That sounds physically meaningful when both patches show the same flat surface. Does it still mean that for arbitrary patch pairs?
Sam: Not necessarily, and that distinction matters. For patches on the same physical plane, with consistent camera geometry, the composition gives the actual planar mapping. For patches on different planes, it is not guaranteed to describe a physical image warp.
Alex: Then should we regard the estimated planes as recovered geometry, or as useful internal parameters?
Sam: The fully predicted version uses latent transformation parameters, optimized for image synthesis. It even allows rotation and translation to vary by patch, although camera pose is shared within a view. The structure is geometrically motivated; every prediction need not be a literal scene measurement.
Alex: How do they train that without making surface annotations a new requirement?
Sam: A lightweight prediction head estimates the parameters from patch features, using the image-synthesis objective without ground-truth geometry supervision. The feature source can be trained from scratch or pretrained. Their strongest main version uses VGGT, a backbone pretrained on geometric prediction tasks.
Alex: That creates an obvious alternative explanation: better pretrained features, rather than better attention geometry. How do they separate those?
Sam: They hold the synthesis architecture fixed and vary the encoding. They also compare different transformations using the same feature source. The homography structure beats an unrestricted matrix, despite fewer degrees of freedom, and beats adding features from VGGT rather than using them to transform attention.
Alex: Put the main result in scale for us. What model and comparison should we remember?
Sam: The main benchmark comparisons use an LVSM image-synthesis backbone with about twenty-five million parameters. On RealEstate10K, the best H-rope version improves peak signal-to-noise ratio by about 1.27 decibels over P-rope. That metric measures image reconstruction quality; higher is better.
Alex: A reconstruction score alone could miss changes in visual appearance. Did the other measures point the same way?
Sam: Yes. Structural similarity and a learned perceptual-distance measure also improve. The strongest version beats the evaluated encodings on RealEstate10K, CO3Dv2, and Objaverse. On DL3DV, it also beats P-rope across the tested input-view counts, with the largest reconstruction gains under sparse observations.
Alex: Which comparison most directly supports the claim that local surface geometry matters, rather than just extra flexibility?
Sam: The calibrated-camera comparison. They retain P-rope’s ground-truth camera projections and add predicted local surface frames. That improves every reported image-quality metric on RealEstate10K. Because the camera projection stays unchanged, this is a concrete test of the added local geometry.
Alex: And if the geometry becomes more accurate, does the model actually benefit?
Sam: In a synthetic dataset with ground-truth geometry, replacing predicted parameters with ground-truth camera and plane parameters improves performance further. That supports the geometry interpretation under that test. It does not establish that all learned parameters are physically accurate.
Alex: What happens when the camera settings differ from training? That is where I would want a geometric prior to earn its keep.
Sam: They test stronger target-camera zoom, targets beyond the reference-view interval, and wider spacing between input frames. H-rope retains its advantage over the comparison encodings under those tested shifts. I’d read that as evidence of useful generalization, not robustness to every possible camera change.
Alex: Does the benefit survive outside their main image-synthesis architecture?
Sam: They also replace the camera encoding in EscherNet, a multi-view diffusion model, and improve every reported image metric across the tested view counts. In a video language model, adding H-rope improves the spatial-reasoning average by 1.8 points over the fine-tuned control. But performance on appearance order declines, so the transfer is not uniformly beneficial.
Alex: There is also a cost question. Does getting the best result require expensive features?
Sam: The learned-from-scratch version improves over the original backbone at the same reported training cost of thirty-three GPU-hours. The strongest VGGT version takes seventy-six GPU-hours under that schedule. So pretrained geometry helps, but its additional training cost belongs in the comparison.
Alex: What is your overall confidence, and who should read the full document?
Sam: I find the evidence persuasive for the evaluated settings: several datasets, fixed-backbone comparisons, and targeted ablations. But this is a preprint, and geometrically structured attention is not proof of recovered physical geometry. Researchers designing multi-view attention should start with the method’s per-patch transform subsection, then the transformation and calibrated-camera comparisons.
Alex: And what should everyone else carry away?
Sam: To compare what cameras see, model the surface’s orientation—not just its location.
Alex: Thanks for listening.