ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

Standard Rotary Position Embeddings (RoPE) are designed for language modeling, where they excel at capturing relative token order and enforcing long-term attention decay. However, this log-linear frequency schedule struggles to model the periodic, routine-based behavior common in user interaction sequences (e.g., watching videos at the same time every day). The authors investigate whether general query/key rotations can be adapted to better capture these complex temporal correlations.

The authors introduce Random Fourier Rotations (RFR), a theoretical framework proving that any normalized continuous positive-definite attention modulation function can be approximated by sampling rotation frequencies from its own Fourier transform. Building on this, they propose ClockRoPE, which applies RFR to periodic functions. By sampling frequencies from a periodic probability mass function—such as a periodic Gaussian or cosine prior—ClockRoPE modulates attention logits to prioritize interactions occurring at similar times of day or days of the week. The implementation maintains the efficient, multiplicative structure of standard RoPE, ensuring it remains computationally feasible for production-scale systems.

ClockRoPE successfully shifts the model's attention focus from simple recency to periodic routines. By using folded frequency distributions, the method also gains directional awareness, helping the model learn a more consistent representation of temporal causality. Empirical results from online A/B tests on a production-scale generative retrieval system at a major video-sharing platform demonstrate that ClockRoPE consistently improves engagement metrics compared to standard RoPE and other temporal encoding baselines.

This work bridges the gap between theoretical Fourier analysis and practical sequence modeling. It provides a principled way to inject domain-specific inductive biases—specifically periodicity—into transformer architectures without sacrificing the efficiency of multiplicative positional encodings. This is particularly valuable for recommendation systems where user behavior is inherently tied to daily or weekly cycles.