Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

As Large Language Models (LLMs) are increasingly used in dynamic environments, the ability to update factual knowledge without full retraining—known as knowledge editing—has become critical. However, existing batch editing methods often suffer from performance degradation as the number of edits increases. This paper investigates the hypothesis that this decline is caused by semantic representation entanglement, where shared syntactic patterns or overlapping concepts across different edit requests create interference in the model's hidden representation space.

The authors propose Orthogonal Representation Editing (ORE), a framework that treats knowledge editing as a geometric problem. ORE consists of two primary components:

1. Representation Subspace Orthogonalization: The authors construct a general semantic subspace by identifying shared patterns from structurally similar but factually unrelated samples. They then use Principal Component Analysis (PCA) to define this subspace and enforce an orthogonal constraint on edit vectors, ensuring that updates are independent of shared semantic noise.
2. Gated Non-linear Representation Head: To move beyond simple linear interventions, the authors introduce a non-linear bottleneck structure combined with a dynamic gating mechanism. This allows the model to adaptively select specific layers and token positions for intervention, ensuring precise knowledge injection while minimizing impact on unrelated model capabilities.

Experimental results on benchmarks like ZsRE and CounterFact demonstrate that ORE consistently outperforms existing methods such as MEMIT and AlphaEdit. The authors show that when edit requests share common templates (e.g., "The capital of [Country] is [City]"), traditional methods struggle with interference, whereas ORE’s orthogonalization effectively decouples these entangled representations. Furthermore, ORE shows superior robustness in cross-lingual settings, where shared multilingual semantic spaces often cause unintended interference in non-target languages.