Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at why AI coding agents—the tools that help developers write and manage software—often repeat the same mistakes even after you correct them.
Sam: So this paper is asking why these agents struggle to learn from us, even when they seem to "remember" our feedback?
Alex: That's the puzzle. The researchers found the core problem isn't that agents forget your instructions. It's that they treat your feedback as mere advice rather than as binding rules.
Sam: And "advice" is easy to ignore. It's like telling a student to show their work—they hear you, but they don't *have* to do it to finish the assignment.
Alex: Precisely. The study identifies what they call an "access-compliance gap." The agent can successfully retrieve your past correction from its memory, but it doesn't treat that correction as a condition it must satisfy. It's aware of the rule, but not bound by it.
Sam: So if I tell an agent, "Always delete these temporary files before you finish," it remembers that I said it—but it can still hand the task back to me with the files sitting right there?
Alex: Exactly. It treats the instruction as a suggestion. Because there's no enforcement mechanism, the agent can finalize its work while leaving those files behind, even if it "knows" you prefer otherwise. The memory is there. The compliance isn't.
Sam: That sounds like a fundamental design flaw. How does this research propose to bridge that gap?
Alex: They introduce a system called Trace—which stands for Test-time Rule Acquisition and Compiled Enforcement. Think of it like a pre-flight checklist for a pilot. The pilot might forget a step, but the plane won't take off until every box is checked. Trace does the same thing for coding agents by turning your natural language feedback into a strict, code-based rule.
Sam: So instead of just storing my words in a memory bank, it converts them into a gatekeeper?
Alex: That's the right way to think about it. The process is called "compiled runtime enforcement." When you give feedback, Trace doesn't just file it away—it translates it into what the researchers call an "atomic rule." Think of that as a specific, verifiable condition that must be true before the task can be marked complete. It's less like a sticky note and more like a lock on the door.
Sam: So if I tell it to clean up debug files, Trace writes a small piece of code that checks the folder. If those files are still there, the agent literally cannot mark the task as finished?
Alex: That is the mechanism. The key shift is from "advice" that the model can quietly ignore, to a "gate" that the system itself enforces. The agent isn't relying on its own judgment to remember your preference—it's structurally blocked from finishing until the condition is met.
Sam: That sounds significantly more reliable than hoping the AI remembers to be helpful. Did the data show this actually works in practice?
Alex: The evidence is notable. In tests on coding tasks, memory-only systems failed to follow user instructions more than half the time. With Trace, the researchers reduced those repeated violations to just two percent on new, previously unseen tasks.
Sam: So the key wasn't giving the AI more memory. It was giving it a better way to *act* on the information it already has.
Alex: That's the central finding. And the system is also designed to stay organized as your feedback grows over time. It includes what the researchers call a "five-action resolver"—essentially a decision-making process that figures out what to do when you add a new instruction. Should it replace an old rule? Merge two rules together? Keep both, because they apply to different situations? The resolver handles those conflicts so your library of constraints stays coherent rather than becoming a contradictory pile of notes.
Sam: So it treats my feedback like a growing set of laws for the agent—not just a log of things I've said.
Alex: That's a good way to put it. The broader argument the paper makes is that making AI agents genuinely reliable isn't just a question of better memory or smarter models. It's about building systems where user intent is actively enforced, not passively stored. When you move from passive memory to active enforcement, you stop the cycle of repeating the same corrections over and over again.
Sam: It's a meaningful distinction. Remembering a rule and being *bound* by a rule are very different things—for humans and, apparently, for AI agents too.
Alex: And that gap between the two is exactly what this research is trying to close. Thanks for listening to ResearchPod.