We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Transformer with sliding-window attention. However, our model continues learning at test time via next-token prediction on the given context, compressing the context it reads into its weights. In addition, we improve the model's initialization for learning at test time via meta-learning at training time. Overall, our method, a form of Test-Time Training (TTT), is End-to-End (E2E) both at test time (via next-token prediction) and training time (via meta-learning), in contrast to previous forms. We conduct extensive experiments with a focus on scaling properties. In particular, for 3B models trained with 164B tokens, our method (TTT-E2E) scales with context length in the same way as Transformer with full attention, while others, such as Mamba 2 and Gated DeltaNet, do not. However, similar to RNNs, TTT-E2E has constant inference latency regardless of context length, making it 2.7 times faster than full attention for 128K context. Our code is publicly available.
Alex: Welcome to another episode of ResearchPod.
Sam: Today, we're looking at how language models handle really long pieces of text—like a whole book or a massive document. The challenge is that the best current systems slow down a lot as the text gets longer.
Alex: Why is that?
Sam: Most top-performing language models use a Transformer with full attention. It looks at every word in the context for each prediction. That works well, but the computing cost grows quadratically. As the text doubles in length, the work needed goes up four times or more. So for a 100-page document, it crawls to a halt.
Alex: Like trying to search your entire memory for one fact instead of summarizing the key points?
Sam: Exactly. Humans don't recall every word from a lecture years ago. We compress the important ideas into our understanding. The paper suggests doing something similar for models: keep using a standard Transformer setup, but with sliding-window attention. That limits the look-back to a nearby chunk of recent words—like reading the last few pages instead of the whole book. It keeps the cost steady per word.
Alex: Does it still capture the full context as well as full attention?
Sam: Not quite on its own—it misses some distant details. The core idea is to let the model keep learning during use, or test time. It practices predicting the next word step-by-step on the input text itself. It updates its internal settings a little each time through small adjustments based on errors. This compresses the whole context into the model's weights, much like taking notes during that lecture to build intuition without storing every word.
Alex: So it's like the model rehearses the material internally to summarize it?
Sam: Yes. They call this Test-Time Training, or TTT. With a special training setup called end-to-end, it matches the performance of full attention on contexts up to 128,000 words. It stays about 2.7 times faster because the speed doesn't grow with length.
Alex: How do they make that rehearsal efficient and stable?
Sam: They break the input text into small groups of words, called mini-batches. They update the model's settings using gradient descent—like gently nudging a bike's handlebars based on small errors. This compares what the model guessed to the actual next word. It happens step-by-step across the whole context. Researchers label this mini-batch Test-Time Training.
Alex: Does sliding-window attention tie in here?
Sam: Yes. Without it, inside each mini-batch, the model would predict later words without seeing the ones just before. That causes bigger errors that snowball. The sliding window gives local memory within the batch—like glancing at the last few sentences while noting key ideas. They set the window to 8,000 words and batches to 1,000.
Alex: What parts of the model get these nudges, and why not everything?
Sam: They only update specific sections called MLP layers. Those are like simple neural networks that process patterns without the complexity of attention. Updating everything else, like attention or normalization parts, causes instability, so those stay fixed. They limit nudges to the last quarter of the model's blocks. In those blocks, they add a second, unchanging MLP to hold pre-trained knowledge, preventing forgetting.
Alex: How does end-to-end training make the whole thing work as well as full attention?
Sam: Standard training ignores these test-time nudges, so the model starts from a poor spot. End-to-end fixes that with meta-learning. During training, they simulate the full sequence of nudges and optimize the starting weights so the final predictions match what you'd want after rehearsal. It's like practicing bike rides end-to-end, not just straight lines. The paper shows TTT-E2E nearly matches full attention's accuracy on long contexts.
Alex: Does that hold for even longer texts?
Sam: They tested different numbers of layers to update, focusing on the last ones. Updating too few means less space to store compressed info—like trying to summarize a book in one sticky note. But with the last quarter, it scales just like full attention, holding steady up to 128,000 words.
Alex: What about errors across a long text?
Sam: They looked at errors for each word position in texts of 32,000 and 128,000 words. TTT-E2E had lower errors than full attention at every spot. The average advantage came mostly from the early parts. That's because full attention has to guess well for every possible future word, spreading its skill thin. TTT-E2E focuses just on the current chunk, since later nudges handle what's ahead—like prepping for today's homework without worrying about next week's test.
Alex: What about tasks needing perfect recall of random details buried deep?
Sam: They tested that with needle-in-a-haystack—hide a key fact anywhere in a long passage, like a single phone number in a novel. Full attention keeps every detail handy. TTT-E2E drops to much lower success at longest lengths, confirming the trade-off: compression loses some fine details for speed.
Alex: Does this hold when the model generates its own long continuation?
Sam: Yes. They had it continue 8,000 words after an 8,000-word prompt from books. TTT-E2E's generated text earned lower errors overall than full attention's. It even self-trains on its own words once batches fill—like practicing what it just wrote to stay sharp.
Alex: What about costs?
Sam: Training is slower—about 3.4 times at short contexts, due to extra steps computing nudges. It evens out at longer lengths. Most training happens short, so it's a notable drawback needing fixes.
Alex: So it compresses long texts into its weights through step-by-step nudges, matching full attention's skill without the slowdown. But recall for buried details suffers, and training takes longer.
Sam: That's right. The paper positions this as continual learning, where the model adapts to each unique input like a person building personalized notes from a lecture, without forgetting core knowledge. It enables processing book-length documents in real time at steady speed, while nearing Transformer's accuracy. Thanks for listening to ResearchPod.