Nossa Iyamu
7 min
This paper addresses the limitation that current LLM agents lack reliable episodic memory, often relying on conversation history rather than actual user behavior. The author proposes a deterministic compilation pipeline that processes raw screen-capture data—including application focus, window titles, URLs, and input events—into structured 'activity frames.' Unlike existing methods that use LLMs to summarize or interpret activity, this approach uses a zero-model, rule-based compiler. This ensures that the output is byte-identical, mechanically auditable, and cacheable, effectively separating raw measured facts from optional, namespaced interpretations.
By moving from model-based summarization to deterministic compilation, this work provides a robust foundation for long-term agent memory. It solves the 'consumption problem' of screen capture, where raw data is too voluminous and noisy for effective use. Furthermore, by providing an open-source reference implementation that integrates with the Model Context Protocol (MCP), the author offers a standardized way for agents to understand and replay complex desktop routines, potentially lowering the cost and increasing the reliability of autonomous computer-use agents.
Computer-use agents pay full frontier inference to re-derive routines their user has already performed, because an agent's memory today records what the user said, not what the user did. We compile passively captured screen activity into agent memory with a deterministic, zero-model pipeline: it segments a local capture stream into typed activity frames, bounded episodes carrying application, site, timing, input volume, and evidence pointers back to the raw rows, with no model in the loop, so the output is byte-identical, cacheable, and mechanically auditable. On one professional's single-user corpus of 128,756 frames over 51 active days, the compiler reduces a day of raw capture to a prompt-ready context block 86x smaller in 68 ms, and an agent reading that block answers questions about the day at 98.4% accuracy (Wilson 95% CI 91.7-99.7%) against an independent oracle, versus 66-80% for an LLM summary of the same capture, a mid-tier model reading the block matching a frontier one. The same compiler doubles as a demand-side cost instrument. Read off passive, pre-delegation human activity rather than agent rollouts, it supplies two parameters that agent-cost models assume but, to our knowledge, have not measured: the Routine Overhead Ratio R and the routine recurrence h. We report first values of R, a modeled upper bound, at 60-343x, and a delegable recurrence of 9.0% in-sample and 7.7% out-of-sample, for a realistic all-fleet token ceiling near 8%; a compiled routine replays deterministically with the model out of the loop, demonstrated live at zero model tokens on a guard-matched hit. Schema, compiler, and evaluation harness are open.
Alex: Think of it as a single entry in a very organized diary. Each frame captures things like which app you were using, the URL if it was a website, and how long you spent there. The system uses what the paper calls a "two-tier schema." The first tier is purely measured facts—things the computer can observe directly, like a timestamp or a URL. The second tier is for interpretations, like labeling something as "a coding project"—but crucially, those interpretations are kept clearly separate from the facts.
Sam: Oh, so the facts tier is the solid foundation, and the interpreted tier is just an optional layer on top. That way you always know what's a direct observation versus what's a guess.
Alex: Exactly. And because every fact is linked directly to the raw data it came from, you can always audit it—trace it back to the source. That's what prevents the AI from making things up. It's not working from memory; it's reading a verified log.
Sam: But if the system is recording constantly, doesn't it just get buried in noise? Every time you glance at a tab for two seconds, does that become an entry?
Alex: The paper addresses this with what it calls a "flicker merge" rule. If you switch between apps rapidly—say, you glance at your email for three seconds before going back to your document—the system collapses those quick blips into a single frame rather than logging each one separately. Think of it like a video editor with an automated script: it cuts out the meaningless flashes and keeps only the meaningful scenes.
Sam: So the log stays clean and focused. But how does the system decide what counts as "meaningful" without an AI making judgment calls? Isn't that the same problem again?
Alex: That's a sharp question, and it's where the fixed rules really matter. The system uses things like dwell-time caps—if you stayed on a page for less than a certain number of seconds, it gets merged away. The threshold is set in advance by the designers, not decided on the fly by a model. That keeps the system predictable. There's no "black box" making hidden decisions.
Sam: It's like the difference between a rule that says "ignore anything under five seconds" versus asking someone to use their judgment about what seemed important. One is consistent; the other varies.
Alex: Precisely. And that consistency is what makes the whole system auditable. Every frame in the log can be traced back to a specific, observable event.
Sam: Okay, so the memory is clean and verifiable. But there's still the question of whether it's actually useful. What does the paper say about efficiency—about whether this actually saves the AI meaningful work?
Alex: The paper introduces a metric called the "Routine Overhead Ratio." It's a way of measuring wasted effort. Imagine you've solved a particular math problem before—say, converting currencies. An AI without this system would re-derive the steps from scratch every single time you ask. With a compiled routine, it just replays the steps it already worked out. The ratio measures how much more expensive that re-derivation is compared to the replay.
Sam: Like the difference between solving a long division problem by hand every time versus just remembering the answer. The second time should be nearly free.
Alex: That's the idea. The paper suggests the cost of re-deriving actions step-by-step is significantly higher than replaying a compiled script. And importantly, the system doesn't need a larger model to get better at this over time. Coverage concentrates on high-frequency, high-signal pages first—things like email or search results. For less common tasks, the system improves incrementally by adding new rules, not by training a bigger, more expensive model.
Sam: So it grows by getting more precise rules, not by getting more powerful. That does sound more sustainable than just scaling up.
Alex: And more transparent. Every improvement is an explicit rule someone wrote down, not a pattern buried inside a neural network that nobody can fully explain.
Sam: So pulling this all together—the paper is making an argument that reliability and efficiency come from the same source: treating screen history like structured code rather than raw footage for an AI to interpret.
Alex: That's the central claim. By moving away from AI-based summarization and toward this deterministic compilation, the memory becomes a verifiable record rather than a best guess. The paper suggests this is a meaningful step toward AI assistants that are not just more capable, but more honest about what they actually know—and what they don't. Thanks for listening to ResearchPod.