LOKI: Memory-Free Null-Space Constrained Lifelong Knowledge Editing

Lifelong knowledge editing aims to update Large Language Models (LLMs) with new facts sequentially without forgetting previously learned information. Existing methods often suffer from two major limitations: they rely on a fixed set of layers for all edits, which limits flexibility, and they require expensive pre-processing or access to historical data to prevent catastrophic forgetting. The authors seek to develop a more efficient, memory-free approach that can adaptively select layers for each specific edit.

The authors introduce LOKI (Layer-adaptive Orthogonal Knowledge Insertion), which operates in three phases:

1. Dynamic Layer Selection: Instead of using a static set of layers, LOKI uses the Hilbert-Schmidt Independence Criterion (HSIC) to identify which layers are most critical for a specific input. It selects layers that maximize information flow between the input and the output.
2. Knowledge Consolidation: LOKI preserves past knowledge by projecting gradient updates onto the null-space of the model's weight matrices. This eliminates the need to store past data or compute expensive statistics, as the null-space is derived directly from the existing weights.
3. Knowledge Insertion: The model performs projected gradient descent to incorporate new facts, using the null-space projection to ensure that updates do not interfere with previously learned information.

LOKI demonstrates superior performance compared to existing state-of-the-art methods across various experimental settings. By moving away from static layer selection and avoiding the need for external memory or pre-processing, LOKI achieves up to a 14% improvement in average accuracy. The authors provide theoretical justification for using the weight null-space as a proxy for preserving past knowledge, suggesting that the model's internal weight structure inherently contains the information necessary to protect prior learning.