Tri-Efficient Transfer Learning for Point Cloud Videos

Point cloud video (PCV) understanding is currently hindered by three major bottlenecks: the high cost of annotating large-scale datasets, the parameter redundancy of task-specific models, and the excessive memory consumption of full fine-tuning. The authors investigate how to achieve "tri-efficient" transfer learning (TriETL)—optimizing for data, parameter, and memory efficiency simultaneously—to better adapt pretrained point cloud foundation models (PFMs) to downstream tasks.

The authors propose PoinTriE, a two-stage framework. In the pretraining stage, they introduce the Geometric-Motion Duality Network (GMD Net). Since real PCV data is scarce, they synthesize "pseudo-motion" trajectories from static point clouds using rigid transformations (rotations and translations). The model is then trained using a combination of multimodal contrastive learning (aligning 3D data with 2D projections and text) and motion prediction tasks.

In the fine-tuning stage, the backbone is frozen to prevent catastrophic forgetting. The authors implement a Spatio-temporal Side Network (STS Net) using LoRA (Low-Rank Adaptation) units. To address memory bottlenecks, they introduce a gradient flow masking strategy that selectively updates parameters, significantly reducing the GPU memory footprint compared to standard fine-tuning or existing parameter-efficient methods.

PoinTriE establishes new state-of-the-art results on standard benchmarks, including action recognition (e.g., 94.37% accuracy on MSR-Action3D) and semantic segmentation (e.g., 84.11% mIoU on Synthia 4D). The theoretical analysis provided by the authors demonstrates that data and parameter efficiency are often mutually exclusive in constrained pretraining, justifying their two-stage design as a necessary strategy to overcome these trade-offs.

This work provides a practical solution for researchers working with limited computational resources or smaller datasets. By decoupling the efficiency requirements into pretraining and fine-tuning stages, PoinTriE allows for the deployment of large-scale foundation models on downstream tasks without the prohibitive memory and data costs that typically limit 3D video understanding.