Multi-agent LLM systems commonly use an orchestrator to decompose a task for a team of workers and then improve through textual reflection. Despite strong empirical results, these systems lack a unified account of coordination, memory improvement, and the role of external verification. We model orchestrator-worker interaction as a bilevel coordination game: under bounded coupling, the workers' local-update game is an approximate potential game whose equilibrium slack is controlled by decomposition quality. We then analyse reflection as stochastic movement over semantic memory states. For free-form reflection, we derive a finite-time upper bound, prove worst-case tightness, and give a positive lower bound under a falsifiable persistent-harm condition. We further prove an information-theoretic impossibility result: no gate that observes only the generated transcript can improve uniformly over text-indistinguishable environments, whereas an environment-grounded gate can. Motivated by this separation, we introduce Stochastic Reflective Memory Ascent (SRMA), which accepts a candidate memory only after a grounded evaluation risk strictly decreases. Under calibration and non-degenerate corrective mass, SRMA converges exactly, geometrically or polynomially; matching constructions show that both rate regimes are order-tight. We also provide confidence gating for stochastic evaluation and re-anchoring guarantees for piecewise-stationary environments. Experiments instantiate these objects with environment-grounded metrics and test the predicted coordination and drift laws. On 500 SWE-bench instances, the complete Kimi-based system resolves 72.2% versus a 70.8% public mini-SWE-agent reference. Code: https://github.com/YihangChen9/Bilevel-Coordinated-Reflection
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper that formalizes multi-agent LLM systems as a bilevel coordination game — and more specifically, asks why reflection loops in these systems tend to plateau rather than converge.
Sam: So the core problem is that current systems treat reflection as just another text-generation step, and that's where things break down?
Alex: Exactly. When you let agents reflect purely in text — no grounding, no external verification — you get hallucination cascades instead of convergence. The system generates plausible-sounding updates that compound errors rather than correcting them. The paper's argument is that this isn't a tuning problem; it's a structural one. Text-only reflection gates are information-theoretically limited in what they can guarantee.
Sam: And the proposed fix is Stochastic Reflective Memory Ascent — SRMA?
Alex: Right. The key mechanism is a grounded verifier — think of it like a test harness — that sits between an agent's proposed memory update and the shared memory store. The system only commits an update if the verifier confirms it actually reduces the error signal. Think of a scientist who only publishes results that survive a controlled experiment, rather than writing down every hypothesis that sounds plausible.
Sam: So the gate isn't semantic — it's not asking "does this sound right?" It's asking "does this measurably improve outcomes?"
Alex: That's the distinction the paper is drawing. And it matters because a purely semantic gate can't distinguish a helpful reflection from a well-phrased hallucination. Without a grounded signal, no history-dependent gate can guarantee convergence — that's one of the paper's central theoretical claims.
Sam: Where does the bilevel structure come in? You mentioned an orchestrator.
Alex: The orchestrator plays the role of the leader in a Stackelberg-style bilevel game. It operates on a slower timescale — decomposing the overall task, setting the strategic frame. The worker agents then optimize their local responses on a faster timescale, within whatever subproblem the orchestrator has handed them. The paper models the worker layer as an approximate potential game, which gives you a handle on when and whether the workers' updates will settle.
Sam: And the "approximate" part is doing real work there — because workers aren't truly independent if they share constraints or resources.
Alex: Exactly. Full independence would give you a clean potential game with guaranteed convergence to a Nash equilibrium. But in practice, workers share context, memory, sometimes even overlapping task boundaries. That coupling introduces a residual — a slack term — and how large that slack gets depends directly on how cleanly the orchestrator decomposed the problem in the first place.
Sam: So a poor decomposition doesn't just make the task harder in an intuitive sense — it formally degrades the convergence guarantee for the workers.
Alex: Right. And SRMA addresses this by making the verifier the bottleneck. Even if the orchestrator's decomposition is imperfect, the grounded gate prevents workers from writing conflicting or error-amplifying updates into shared memory. The system only advances to a new state when the verifier signs off on a genuine reduction in the risk score.
Sam: So you have two timescales — the orchestrator's slow strategic loop and the workers' fast execution loop — and the verifier is what keeps the fast loop from running ahead of itself and accumulating garbage in memory.
Alex: That's a clean way to put it. And it's why the paper emphasizes the "hallucination floor" problem in ungrounded loops — without the verifier, the fast loop has no mechanism to distinguish productive updates from fluent noise. The system can keep iterating indefinitely without actually improving, which is exactly the plateau behavior that motivated the work.
Sam: It's an interesting design choice to make the verifier external to the agents rather than trying to train the agents themselves to be better at self-assessment.
Alex: The paper's position is that self-assessment is precisely what can't be trusted without grounding. An agent evaluating its own output is operating in the same distribution that produced the error. The verifier needs to be grounded in something the agent can't simply rationalize around — execution results, test outcomes, something with a truth value independent of the agent's own generation process.
Sam: Which also means the approach is most natural in domains where you have a reliable verifier — code execution, formal reasoning, anything with a checkable answer.
Alex: That's the binding limitation, and the paper doesn't fully sidestep it. The framework is theoretically general, but the convergence guarantees are only as strong as the verifier. In open-ended domains — creative tasks, strategic planning with no clear ground truth — you'd need a proxy verifier, and the theoretical properties become harder to defend. That's the gap a careful referee would push on.
Sam: So the contribution is really two things: a formal game-theoretic account of why ungrounded reflection fails, and a mechanism — the grounded memory gate — that restores convergence guarantees under the right conditions.
Alex: That's the load-bearing claim. The bilevel framing gives you the vocabulary to talk precisely about orchestrator quality and worker coupling. The SRMA mechanism gives you a constructive answer to the convergence problem. Whether that answer generalizes beyond verifiable domains is the open question the paper leaves on the table — and probably the most important one for anyone thinking about deploying this in practice.
Sam: That's a useful frame. The theory is doing real work here, not just dressing up an engineering heuristic.
Alex: Which is rarer than it should be in this space. Thanks for listening to ResearchPod.