The Generalization Spectrum: A Chromatographic Approach to Evaluating Learning Algorithms

Traditional machine learning evaluation relies on aggregate scores, which conflate memorization with true generalization. This paper introduces the Generalization Spectrum, a framework that evaluates learning algorithms by measuring performance across a controlled axis of transfer distance. By tracking how well a model performs on variants of a training seed—ranging from exact recall to unpaired, category-matched problems—the authors expose the hidden "generalization radius" of different learning paradigms.

The authors define five levels of transfer distance (D0–D4) based on the amount of shared information between a training seed and a test variant:

• D0 (Exact Recall): The original problem.
• D1 (Implementation Transfer): Same logic, different programming language.
• D2 (Context Transfer): Same logic, completely different narrative/framing.
• D3 (Category-Matched): Different problem, same algorithmic family.
• D4 (Unpaired Baseline): General in-domain problem.

By comparing models at matched levels of D0 performance, the authors isolate "transfer efficiency," or how effectively a model converts memorization into broader generalization. This approach prevents the common pitfall of attributing superior transfer to simply having a better-memorized base.

The study reveals that algorithm choice fundamentally reshapes the generalization profile. While SFT often shows strong performance on near-transfer tasks (D1), its gains collapse rapidly as the distance increases. In contrast, RL maintains a more robust "far-transfer tail," performing better on D3 and D4 tasks when compared at equivalent D0 memorization levels. Furthermore, the study finds that in-context learning (ICL) is highly effective at context transfer (D2) but remains brittle, as its performance is strictly dependent on the quality of the provided demonstration.