Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.
Alex: A language model can finish a long task without carrying its entire history. In these experiments, replacing that history with a carefully chosen brief usually preserved performance, even when the model started fresh after every action.
Sam: That challenges the push toward ever larger model memories. Is the claim that bigger memories are useless, or that we often use them unnecessarily?
Alex: The second, with qualifications. This is an experimental arXiv preprint by Peter Belcak and colleagues, titled “You don’t need long context.” They test coding, information-seeking, tool use, and document questions.
Sam: So this matters if my research involves agents working through many steps, or models reading large collections. The question is whether I need to pay for keeping everything available throughout.
Alex: And whether that habit slows local models unnecessarily. We’ll unpack what the brief preserves, what the comparisons show, and where the approach fails.
Sam: Before the method, what does “context” mean here? People use memory to mean several different things.
Alex: Context is the text and other tokens available during a model’s current inference step. It can contain instructions, the task, earlier outputs, and tool results. The context window is the maximum amount the model endpoint can process at once.
Sam: The authors distinguish a long task from a long document. Why does that distinction matter?
Alex: A long task builds history as the agent acts. A long document starts with lots of source material. Neither automatically requires everything to remain in the current context: history can be condensed, and documents can be read selectively.
Sam: What makes that worth the engineering effort? Does carrying extra context really change generation costs that much?
Alex: In their cloud-serving measurements, the largest improvement in output throughput was sevenfold when switching from their longest compared setting to the shortest. That was on identical hardware, under saturated serving load. They also measured local models becoming slower and using more memory as context increased.
Sam: That is a serving result, not yet evidence that a useful task becomes cheaper. How did they check whether shortening the context breaks the work?
Alex: Their baseline keeps the complete interaction history. The alternative accumulates history only until a threshold, then asks the same model for a bounded brief. A fresh conversation receives the original task and that brief, and the agent continues.
Sam: A summary can preserve the story while losing the detail needed to fix a bug. What did these briefs actually retain?
Alex: The authors found ordinary summarization suboptimal. Their compaction, meaning task-focused compression of the history, preserves the task almost verbatim, current intentions, planned steps, and relevant artifacts. It also carries completion requirements and pointers to information needed later.
Sam: So the brief is not an account of everything the agent did. It is a handover for whoever takes the next action.
Alex: That captures their distinction. They describe the best compactions as briefs for further execution, rather than narratives of execution history. They used the same basic structure across the different workflows.
Sam: How broad was the test? A handover that works for coding might fail when an agent must combine evidence across sources.
Alex: The long-horizon experiments covered software repair, terminal tasks, information retrieval, simulated user interactions, and tool coordination. One anchor was SWE-bench Verified: five hundred real repository issues, evaluated by whether proposed patches pass held-out tests. The main workers were Claude Opus 4.5 and GPT 5.6 Sol.
Sam: And what happened as the authors made the available context more restrictive?
Alex: Accuracy broadly stayed near the full-history baseline. The largest improvement across those main experiments was about six percentage points, though some configurations lost accuracy. The information-retrieval benchmark consistently showed modest losses, so shortening history was not uniformly beneficial.
Sam: Losing history sounds inevitable eventually. Suppose the task needs an exact observation from early in the run, not just the current plan.
Alex: The authors make that objection themselves, using a sequence of coin tosses. A bounded brief must eventually lose some outcomes. Their proposed escape is to keep the complete record outside the conversation and let the agent retrieve it when needed.
Sam: That is a claim about what must be actively present, not what can be thrown away forever. How did they apply it to long documents?
Alex: They stored documents as files and gave the agent tools for searching and reading selected passages. That is selective input reading. In one recorded run, the agent searched a large Kalamang grammar for a word, then read surrounding grammatical explanations and examples.
Sam: That example has a convenient search term. Did they test anything beyond finding a sentence with matching words?
Alex: Yes. The document benchmarks included questions requiring comparisons and reasoning across financial reports, legal material, and academic papers. There were also controlled retrieval and reasoning tests. The authors acknowledge that straightforward lexical overlap helped explain the especially strong needle-search results.
Sam: What result best supports this outside the artificial retrieval tests?
Alex: On the real-world document-reasoning benchmark called Artificial Analysis Long Context Reasoning, Opus improved by nineteen percentage points over receiving the full input. Sol improved slightly on that benchmark. Across the long-input tests, selective reading almost always matched or improved the full-input approach.
Sam: Can we credit that improvement to compaction, or is searching the files doing most of the work?
Alex: Often it is the selective reading. Compactions were rare in many document experiments, and some tests needed none. The evidence supports changing how inputs are accessed, not just compressing them after the model has already read everything.
Sam: Then the most extreme experiment is the reset after every action. What survives from one conversation to the next?
Alex: Only the task and an updated description of the current working state. The model takes an action, sees the outcome, updates that description, and starts fresh. Across the tested coding and terminal tasks, typical peak context was roughly eight thousand tokens.
Sam: Did that preserve accuracy for both models, or does the headline hide a trade-off?
Alex: Opus preserved or improved its baseline performance. Sol lost a little accuracy; its terminal-task reduction was about three percentage points. That supports the narrower claim that resetting after every action need not cause collapse.
Sam: But model choice could be decisive. What happened with the openly available models?
Alex: Their robustness varied substantially. At the tightest coding-task setting, Qwen Coder Next fell from fifty-three percent success to about four percent. The authors attribute much of that collapse to verbose trajectories leaving too little room for progress between compactions.
Sam: That makes this a strategy to test, not a universal setting to switch on. And are the advertised financial savings actual API bills?
Alex: The headline savings are estimates, not simply observed reductions in current bills. They extrapolate the measured open-model serving advantage into hypothetical context-sensitive prices for the closed models. At published API prices, some compacting runs cost more, because they generated more output or used more new input.
Sam: The authors also propose semantic state space models. Is that a trained system they demonstrate here?
Alex: It is a proposal: models post-trained to carry an explicit language description of state between turns. The experiments apply context management to existing models. They do not demonstrate the proposed training approach, or establish the authors’ suggested training-efficiency benefit.
Sam: With those boundaries, who should read the full document, and where should they start?
Alex: Researchers building agents or document-analysis systems should read the harness definitions first. Then compare detailed accuracy with nominal retail cost, and read the open-weight results before choosing a context budget. The authors explicitly do not prove that every task can use short context.
Sam: And what should everyone else remember when someone calls a workload inherently long-context?
Alex: Needing access to information is not the same as needing all of it in the current conversation.
Sam: Thanks for listening.