Polycepta: Object-Centric Appearance Estimation for Multi-Object Tracking

In multi-object tracking (MOT), appearance descriptors are typically extracted frame-by-frame, assuming that an object's visual identity is static. This approach is often computationally expensive and fails to account for the continuous visual evolution of objects under occlusion, perspective changes, or environmental noise. The authors ask: can appearance modeling be reformulated as a recursive estimation problem—similar to how Kalman filters handle motion—to improve tracking robustness and efficiency?

The authors introduce Polycepta, a framework that maintains an independent, evolving appearance state for each tracked object. Instead of relying on isolated, instantaneous features, Polycepta uses a Visual Relational Reasoning (VRR) module to correlate current observations with historical appearance states in the frequency domain. This allows the model to accumulate visual information over time. To ensure the model learns generalizable state-construction dynamics rather than memorizing specific object histories, the authors employ a state-erasure training strategy, where appearance states are reset at the end of every training epoch.

Polycepta demonstrates that appearance estimation quality improves as object states evolve during inference, providing a significant advantage over conventional static descriptors. When integrated into existing tracking-by-detection pipelines, the framework consistently reduces identity switches and improves overall tracking performance across benchmarks like KITTI, MOT17, and the Waymo Open Dataset. Notably, the system achieves state-of-the-art performance on the KITTI benchmark with a MOTA of 92.27% while maintaining a high operational speed of 90.57 Hz.