Peter Belcak, Saurav Muralidharan, Jindong Jiang, Yonggan Fu, Wonmin Byeon, Hanrong Ye, Ryan Angilly, Ali Taghibakhshi, Shih-Yang Liu, Siru Ouyang, Greg Heinrich, Pavlo Molchanov
8 min
Abstract
Later tokens are costlier to generate than earlier tokens. On frontier open weight models, the disparity in cost may rise to as much as 7x. Through frequent compaction, we show across a broad range of long-horizon and long-input tasks that frontier language models sacrifice little to no task accuracy when maintaining short contexts. We further demonstrate that LLM performance does not collapse on long-horizon software engineering tasks even when taking at most one action per context, and based on that we outline the concept of semantic state space models (SSSMs). Our findings question the need for long-context language models.
Alex: The authors found ordinary summarization suboptimal. Their compaction, meaning task-focused compression of the history, preserves the task almost verbatim, current intentions, planned steps, and relevant artifacts. It also carries completion requirements and pointers to information needed later.
Sam: So the brief is not an account of everything the agent did. It is a handover for whoever takes the next action.
Alex: That captures their distinction. They describe the best compactions as briefs for further execution, rather than narratives of execution history. They used the same basic structure across the different workflows.
Sam: How broad was the test? A handover that works for coding might fail when an agent must combine evidence across sources.
Alex: The long-horizon experiments covered software repair, terminal tasks, information retrieval, simulated user interactions, and tool coordination. One anchor was SWE-bench Verified: five hundred real repository issues, evaluated by whether proposed patches pass held-out tests. The main workers were Claude Opus 4.5 and GPT 5.6 Sol.
Sam: And what happened as the authors made the available context more restrictive?
Alex: Accuracy broadly stayed near the full-history baseline. The largest improvement across those main experiments was about six percentage points, though some configurations lost accuracy. The information-retrieval benchmark consistently showed modest losses, so shortening history was not uniformly beneficial.
Sam: Losing history sounds inevitable eventually. Suppose the task needs an exact observation from early in the run, not just the current plan.
Alex: The authors make that objection themselves, using a sequence of coin tosses. A bounded brief must eventually lose some outcomes. Their proposed escape is to keep the complete record outside the conversation and let the agent retrieve it when needed.
Sam: That is a claim about what must be actively present, not what can be thrown away forever. How did they apply it to long documents?
Alex: They stored documents as files and gave the agent tools for searching and reading selected passages. That is selective input reading. In one recorded run, the agent searched a large Kalamang grammar for a word, then read surrounding grammatical explanations and examples.
Sam: That example has a convenient search term. Did they test anything beyond finding a sentence with matching words?
Alex: Yes. The document benchmarks included questions requiring comparisons and reasoning across financial reports, legal material, and academic papers. There were also controlled retrieval and reasoning tests. The authors acknowledge that straightforward lexical overlap helped explain the especially strong needle-search results.
Sam: What result best supports this outside the artificial retrieval tests?
Alex: On the real-world document-reasoning benchmark called Artificial Analysis Long Context Reasoning, Opus improved by nineteen percentage points over receiving the full input. Sol improved slightly on that benchmark. Across the long-input tests, selective reading almost always matched or improved the full-input approach.
Sam: Can we credit that improvement to compaction, or is searching the files doing most of the work?
Alex: Often it is the selective reading. Compactions were rare in many document experiments, and some tests needed none. The evidence supports changing how inputs are accessed, not just compressing them after the model has already read everything.
Sam: Then the most extreme experiment is the reset after every action. What survives from one conversation to the next?
Alex: Only the task and an updated description of the current working state. The model takes an action, sees the outcome, updates that description, and starts fresh. Across the tested coding and terminal tasks, typical peak context was roughly eight thousand tokens.
Sam: Did that preserve accuracy for both models, or does the headline hide a trade-off?
Alex: Opus preserved or improved its baseline performance. Sol lost a little accuracy; its terminal-task reduction was about three percentage points. That supports the narrower claim that resetting after every action need not cause collapse.
Sam: But model choice could be decisive. What happened with the openly available models?
Alex: Their robustness varied substantially. At the tightest coding-task setting, Qwen Coder Next fell from fifty-three percent success to about four percent. The authors attribute much of that collapse to verbose trajectories leaving too little room for progress between compactions.
Sam: That makes this a strategy to test, not a universal setting to switch on. And are the advertised financial savings actual API bills?
Alex: The headline savings are estimates, not simply observed reductions in current bills. They extrapolate the measured open-model serving advantage into hypothetical context-sensitive prices for the closed models. At published API prices, some compacting runs cost more, because they generated more output or used more new input.
Sam: The authors also propose semantic state space models. Is that a trained system they demonstrate here?
Alex: It is a proposal: models post-trained to carry an explicit language description of state between turns. The experiments apply context management to existing models. They do not demonstrate the proposed training approach, or establish the authors’ suggested training-efficiency benefit.
Sam: With those boundaries, who should read the full document, and where should they start?
Alex: Researchers building agents or document-analysis systems should read the harness definitions first. Then compare detailed accuracy with nominal retail cost, and read the open-weight results before choosing a context budget. The authors explicitly do not prove that every task can use short context.
Sam: And what should everyone else remember when someone calls a workload inherently long-context?
Alex: Needing access to information is not the same as needing all of it in the current conversation.
Sam: Thanks for listening.