Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems | Peilin Feng et al. | ResearchPod

Transcript: Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study on "Σ-Mem," an online memory system designed to help AI agents work together more reliably.

Sam: So the paper is asking why teams of AI models often fail, even when each individual model is quite capable on its own?

Alex: Exactly. Imagine you're working on a group project, and everyone claims to be an expert. The problem is you have no record of who actually got things right in the past. These AI systems have the same issue—they store transcripts of past conversations, but they don't track how reliable each participant actually was. There's no trust memory.

Sam: So they remember what was said, but not who was worth listening to.

Alex: Right. And that gap matters a lot when you have multiple AI agents giving different answers. The system has no principled way to decide whose input to weight more heavily. Σ-Mem is designed to fix that by building what the researchers call a "reputation scorecard" for each agent.

Sam: What does that scorecard actually track?

Alex: Two things. First, how well an individual agent performs on specific types of tasks. Second—and this is the subtler part—whether two agents tend to succeed or fail together. That second piece matters because if two agents always agree and always get things wrong, they might share the same blind spot or bias. The system needs to know that, not just how each one performs in isolation.

Sam: So it's not enough to know that Agent A is usually right. You also need to know whether Agent A and Agent B are making the same mistakes.

Alex: Precisely. To keep track of all of this, they use a structure called a symmetric matrix—think of it as a grid, where each row and column represents one agent, and each cell in the grid stores evidence of how competent that pair is together. It's a compact way to hold a lot of relationship information at once.

Sam: That makes sense. But how do they stop one bad result from tanking an agent's whole reputation?

Alex: This is where a mathematical principle called Weyl's inequality comes in. The name sounds technical, but the idea is straightforward: it acts like a speed limit on updates. No single piece of feedback—good or bad—can shift the scorecard by more than a small, controlled amount. So a one-off mistake doesn't erase a strong track record, and one lucky correct answer doesn't inflate a poor performer's reputation overnight.

Sam: So the system is designed to be patient. Real, consistent patterns build up gradually, while random noise gets smoothed out.

Alex: Exactly. And because the updates happen continuously—as the system is running, not in a separate training phase—it can adapt in real time. If an agent that was reliable starts making errors, the scorecard shifts. If a new agent proves itself over time, it earns trust. No retraining required.

Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

Peilin Feng, Suorong Yang, Soujanya Poria

Jul 30, 2026·4:36·arxiv·Original Paper

Σ-MemLLM-based Multi-Agent SystemsReliability MemoryOnline AdaptationMulti-Agent Systems (MAS)Historical Competence

Listen

5 min

Briefing

Research Question

In multi-agent systems (MAS), a central model often struggles to determine which peer agent to trust, especially when verification of peer responses is difficult or impossible. Existing memory systems typically focus on storing interaction content rather than modeling the reliability of the agents themselves. This paper asks: can we build a persistent, task-conditioned memory that explicitly tracks peer competence and inter-peer relationships to improve coordination in MAS?

Approach

The authors introduce Sigma-Mem, a memory mechanism that maintains two types of evidence: historical competence (how trustworthy a peer is under specific task conditions) and peer relationship evidence (how peers' correctness patterns correlate). Both are stored as real symmetric matrices.

To ensure stability, the system uses a decayed update rule. By applying Weyl's inequality, the authors prove that the spectral change caused by any single update is bounded, preventing noisy or isolated feedback from destabilizing the memory. This allows the system to accumulate reliable, long-term competence signals. The memory provides a flexible interface, supporting three distinct coordination strategies: residual steering of a central model, response-free peer routing, and reliability-weighted voting.

Main Findings

Sigma-Mem demonstrates strong adaptability and generalization across five Qwen-family models. Key results include:

Adaptability: On counterfactual benchmarks where the "best" peer changes over time, Sigma-Mem significantly improves selection accuracy compared to static baselines.
Generalization: The system successfully generalizes to unseen peers and task domains not present during training.
Versatility: The same frozen memory state effectively supports multiple decision-making mechanisms, including routing and weighted voting, without requiring additional training.
Progressive Improvement: Performance consistently improves as more correctness feedback is provided, confirming that the system successfully accumulates actionable reliability information over time.

Key Terms and Definitions

Sigma-Mem — An online memory system that records peer competence and relationship evidence to guide trust-aware coordination in multi-agent systems.
Historical Competence Evidence — A symmetric matrix representing a peer's past performance, used to estimate the likelihood that a peer will be correct on a future task.
Peer Relationship Evidence — A matrix tracking whether peers tend to succeed or fail together, helping the system distinguish between independent corroboration and correlated errors.
Residual Steering — A technique where the memory readout is injected into the central model's residual stream to influence its attention and utility evaluation without modifying the model's weights.
Weyl's Inequality — A mathematical property used to bound the change in eigenvalues of a symmetric matrix after a perturbation, ensuring the memory remains stable during online updates.

Abstract

Memory is central to long-horizon LLM agents, yet existing memory systems primarily preserve interaction content rather than modeling which agents can be trusted and under what conditions. This limitation is particularly important in multi-agent systems, where a central model may be unable to directly verify plausible or correlated peer responses. We introduce $Σ$-Mem, an online reliability memory that records historical competence evidence for individual peers and peer relationship evidence across the peer set. Both forms of evidence are maintained as real symmetric states and updated from post-decision correctness feedback. By Weyl's inequality, the spectral change caused by each event-level update is bounded, enabling stable online adaptation without retraining the underlying models. $Σ$-Mem provides a general write-and-read interface: the same memory can be used for residual steering of a central model, response-free peer routing, or reliability-weighted voting. Across five Qwen-family models, $Σ$-Mem adapts to counterfactual reliability shifts and generalizes to unseen peers and task domains. Direct memory readouts also outperform majority voting and the best fixed peer over the full OOD evaluation set. Moreover, performance improves consistently as more correctness feedback becomes available, indicating that $Σ$-Mem progressively accumulates actionable reliability information. These results establish reliability memory as a reusable foundation for adaptive coordination in LLM-based multi-agent systems.

Σ-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems: Research Paper Podcast