Guolin Ke, Di He, Tie-Yan Liu
4 min
Standard Transformer-based language models, such as BERT, typically incorporate positional information by adding absolute or relative positional embeddings directly to word embeddings. The authors investigate whether this additive approach is optimal, arguing that it forces heterogeneous information—semantic word content and structural sequence position—into a shared space, potentially introducing noise and limiting model expressiveness.
To address these limitations, the authors propose Transformer with Untied Positional Encoding (TUPE). The core innovation is to compute word-contextual correlations and positional correlations separately using distinct projection matrices, rather than adding them at the input layer. This decoupling prevents the mixing of heterogeneous information. Additionally, the authors identify that the [CLS] token, which serves as a global sentence representation, should not be treated identically to standard words. TUPE introduces a specialized mechanism to "untie" the [CLS] token from other positions, allowing it to better capture global information without being biased by local attention patterns.
The authors demonstrate that the standard additive approach in BERT creates noisy correlations between words and positions. By decoupling these components, TUPE provides a more expressive architecture that consistently outperforms standard BERT baselines across the GLUE benchmark. The ablation studies confirm that both the untied positional projection and the specialized handling of the [CLS] token contribute to improved performance. Furthermore, the authors show that TUPE is computationally efficient, adding only about 1% to the total parameter count of a BERT-Base model.
Positional encoding is a fundamental component of the Transformer architecture. This work suggests that the common practice of simply adding positional embeddings to word embeddings is a suboptimal design choice. By rethinking how positional information is injected into the self-attention mechanism, researchers can achieve better language understanding with minimal additional computational overhead, providing a more robust foundation for future pre-training efforts.
In this work, we investigate the positional encoding methods used in language pre-training (e.g., BERT) and identify several problems in the existing formulations. First, we show that in the absolute positional encoding, the addition operation applied on positional embeddings and word embeddings brings mixed correlations between the two heterogeneous information resources. It may bring unnecessary randomness in the attention and further limit the expressiveness of the model. Second, we question whether treating the position of the symbol \texttt{[CLS]} the same as other words is a reasonable design, considering its special role (the representation of the entire sentence) in the downstream tasks. Motivated from above analysis, we propose a new positional encoding method called \textbf{T}ransformer with \textbf{U}ntied \textbf{P}ositional \textbf{E}ncoding (TUPE). In the self-attention module, TUPE computes the word contextual correlation and positional correlation separately with different parameterizations and then adds them together. This design removes the mixed and noisy correlations over heterogeneous embeddings and offers more expressiveness by using different projection matrices. Furthermore, TUPE unties the \texttt{[CLS]} symbol from other positions, making it easier to capture information from all positions. Extensive experiments and ablation studies on GLUE benchmark demonstrate the effectiveness of the proposed method. Codes and models are released at https://github.com/guolinke/TUPE.
Sam: So it is like a student who was taught a slightly wrong method for solving problems. They might still get the right answer eventually, but they are working harder than they need to.
Alex: Exactly. And here is the underlying reason it works. Think of the part of the model that processes data as a translator — it takes raw information and shifts it into a form where the model can compare different things. When you jam word meaning and position indices into the same translator, it gets confused. It ends up trying to compare words to positions, which the paper shows produces essentially uniform noise — meaningless static.
Sam: So by untying them, you give the model two separate translators? One for the words and one for the positions?
Alex: That is the core mechanism. You remove the noisy, useless comparisons, and the model is left to focus only on the relationships that actually carry meaning. When you stop asking a system to learn from random noise, it finds the real signal much faster.
Sam: That explains the faster training. It is not doing extra work on data that was never going to teach it anything useful.
Alex: Precisely. And it is worth asking why this is not already universal. The primary limitation is that TUPE requires changing the internal architecture of the attention mechanism itself — the core processing layer of the model. You cannot simply patch an existing, already-trained model to use it.
Sam: So it is a meaningful improvement for building new models, but not a quick upgrade for the large models that already exist?
Alex: That is correct. It requires a commitment to a new training cycle from scratch. Which means the benefit is real, but the path to adoption takes time.
Sam: It is a reminder that some of the most important changes are structural. Not just adding more computing power, but questioning the assumptions built into the foundation.
Alex: That is the essence of this research. In deep learning, how we structure our initial assumptions is often just as important as the size of the model itself. Sometimes the most effective path forward is to simplify, not to scale. Thanks for listening to ResearchPod.