Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Training

Large language models (LLMs) are increasingly using test-time training (TTT) to adapt to new information, such as user preferences, project-specific facts, or corrections. Current research typically evaluates these updates using proxy metrics like perplexity, future-token loss, or task reward. While these metrics are effective for measuring in-sequence adaptation, the authors argue they are inadequate for supporting claims about long-term deployment memory or personalization. A model can lower its loss on a support passage without actually acquiring the information in a way that is accessible for later, open-ended user interactions.

The authors propose a new framework to calibrate TTT claims to the evidence provided. They define a three-tier evidence ladder:

• Stream/Domain Adaptation (S-level): Evidence that the model adapts to recent tokens or target-domain data, supported by perplexity or task-specific accuracy.
• Bridge/Internalization (B-level): Evidence that information is stored or compressed within the model state, often tested via needle-in-a-haystack recall or answer-likelihood improvements.
• Deployment-Time Behavioral Learning (D-level): Evidence that sparse, post-deployment information changes later behavior after the original support context is removed, tested via recall, paraphrase robustness, and conflict resolution.

To validate this framework, the authors conducted a controlled diagnostic experiment using Qwen3 models. They applied one-step LoRA updates to inject sparse facts. While these updates consistently lowered the loss on the support and answer tokens, the models achieved zero percent success in free-form recall when queried after the support context was removed. Even when stronger updates were used to force recall, the models suffered from significant 'locality' failures, where unrelated knowledge was inadvertently corrupted. This highlights a critical trade-off: proxy improvements do not guarantee behavioral utility, and stronger behavioral recall can introduce dangerous interference.