Yujun Zhou, Kehan Guo, Haomin Zhuang, Xiangqi Wang, Yue Huang, Zhenwen Liang, Pin-Yu Chen, Tian Gao, Nuno Moniz, Nitesh V. Chawla, Xiangliang Zhang
4 min
Abstract
Interactive LLM agents are becoming part of daily work, but they do not reliably become easier to work with over time: a correction remembered in one session may still be violated in the next. We study this gap between preference access and preference compliance. In tasks derived from anonymized real-user friction cases, Mem0 memory still leaves 57.5% of applicable preference checks violated. We introduce Test-time Rule Acquisition and Compiled Enforcement (TRACE), a drop-in skill-layer pipeline for coding-agent runtimes that mines user corrections, rewrites them as atomic rules, and compiles them into runtime checks that must pass before an agent completes future tasks. Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections. We evaluate TRACE with simulated user-in-the-loop experiments on ClawArena coding-agent tasks and MemoryArena-derived memory-intensive tasks. On ClawArena, TRACE reduces held-out preference violation from 100.0% to 37.6% on in-distribution tasks and from 100.0% to 2.0% on out-of-distribution tasks. On MemoryArena-derived tasks, TRACE reduces in-distribution violation from 100.0% to 60.5% while matching or exceeding the strongest memory baseline on task pass. These results suggest that compiling corrections into runtime enforcement can address a repeated-friction failure mode that memory alone does not reliably solve, reducing the need for users to restate the same correction across future sessions. Experiment code is available at https://github.com/YujunZhou/TRACE_exp, and the deployable skill is available at https://github.com/YujunZhou/tellonce.
Sam: That sounds significantly more reliable than hoping the AI remembers to be helpful. Did the data show this actually works in practice?
Alex: The evidence is notable. In tests on coding tasks, memory-only systems failed to follow user instructions more than half the time. With Trace, the researchers reduced those repeated violations to just two percent on new, previously unseen tasks.
Sam: So the key wasn't giving the AI more memory. It was giving it a better way to *act* on the information it already has.
Alex: That's the central finding. And the system is also designed to stay organized as your feedback grows over time. It includes what the researchers call a "five-action resolver"—essentially a decision-making process that figures out what to do when you add a new instruction. Should it replace an old rule? Merge two rules together? Keep both, because they apply to different situations? The resolver handles those conflicts so your library of constraints stays coherent rather than becoming a contradictory pile of notes.
Sam: So it treats my feedback like a growing set of laws for the agent—not just a log of things I've said.
Alex: That's a good way to put it. The broader argument the paper makes is that making AI agents genuinely reliable isn't just a question of better memory or smarter models. It's about building systems where user intent is actively enforced, not passively stored. When you move from passive memory to active enforcement, you stop the cycle of repeating the same corrections over and over again.
Sam: It's a meaningful distinction. Remembering a rule and being *bound* by a rule are very different things—for humans and, apparently, for AI agents too.
Alex: And that gap between the two is exactly what this research is trying to close. Thanks for listening to ResearchPod.