Chunming Wu, Dafei Qiu, Charles Quan, Jun Wu, Suipeng Li, Mo Wu, Gavin Xie, Hope Chen, Max Yao, Congde Yuan
5 min
The authors propose an Evidence-Grounded Customer-Service Agent Workflow designed to address the limitations of standard RAG systems, which often struggle with policy compliance, hallucination, and the integration of structured business rules. Instead of treating the LLM as an autonomous generator, the system uses a fixed LangGraph DAG to orchestrate a multi-stage pipeline: intent routing, parallel evidence gathering (FAQ retrieval and rule-based context), evidence fusion, and a final decision stage. This architecture ensures that the LLM only selects answers from a pre-validated, auditable set of candidates.
This work shifts the focus of LLM deployment from "model scaling" to "system engineering." By isolating the decision-making process from the evidence-retrieval layer, the authors provide a framework for building reliable, auditable AI agents in high-stakes environments. The results show that even with a fixed backbone model, systematic diagnosis and iterative refinement of evidence sources and decision logic can yield substantial improvements in accuracy and policy compliance, outperforming legacy RAG-only approaches.
Production customer-service bots must improve answer quality across iterative releases, yet large language models must not bypass evidence boundaries, policy rules, or human-handoff safeguards. We present an \textbf{Evidence-Grounded Customer-Service Agent Workflow} deployed in a real-world customer-service setting. BM25 recall, issue-title-vector recall, issue-description-vector recall, weighted RRF fusion, and cross-encoder reranking construct grounded FAQ evidence for controlled LLM decisions. Policy-guided orchestration then combines this RAG evidence with scenario-specific rule evidence, conversation memory, and clarification state inside a fixed LangGraph DAG~\cite{langgraph2024}. The paper contributes three reusable deployment patterns: \textbf{hybrid RAG evidence construction}, where multi-channel retrieval and reranking produce auditable FAQ candidates; \textbf{evidence-grounded issue/action decision}, where an Evidence-Grounded Decision Module selects an issue/action from typed FAQ evidence and scenario-specific rule evidence; and \textbf{trace-driven RAG and reranker improvement}, where traces diagnose whether failures come from recall, ranking, final candidate selection, clarification, rule-derived evidence, or action policy, and where reranker fine-tuning is evaluated not only for in-domain gain but also for forgetting risk.
Alex: That's where the "Decision Module" comes in. Rather than letting the AI pick whichever answer sounds most confident, it lines up all the evidence as candidates and compares them side by side — like a judge weighing documents from two different lawyers — before committing to an action.
Sam: And they use something called "Direct Preference Optimization" to sharpen that comparison process?
Alex: Right. Imagine teaching someone a job by showing them examples of good decisions and bad decisions side by side, and asking them to explain why one is better. That's essentially what this technique does. It trains the AI to prefer actions that align with actual business rules, rather than just picking whatever sounds most plausible.
Sam: How do they stop the AI from getting stuck in a loop — like endlessly asking for more information without ever resolving anything?
Alex: They use something called a Directed Acyclic Graph, or DAG. The name sounds technical, but the idea is simple: it's a one-way flowchart. Every path through it moves forward and eventually reaches an endpoint. There are no cycles, no dead ends where the AI can spin in place. It has to keep moving toward a resolution.
Sam: And there's also a "Unified Candidate Contract" — what does that actually do?
Alex: Think about the different kinds of information the system handles. A live database entry telling you whether an account is locked looks very different from a written FAQ article. If the AI has to figure out how to read each format differently, that's extra room for error. The Unified Candidate Contract standardizes every piece of evidence so it all looks the same to the AI. A database entry and an article are presented as equal candidates, and the AI just has to compare them — not decode them first.
Sam: And for that live data — like actually checking whether an account is locked right now — how does it pull that in?
Alex: It uses what the paper calls a "Model Context Protocol." Think of it as a standardized plug socket. Rather than building a custom connection to every internal system, the agent uses one universal connector to pull live data from whichever system it needs, in real time.
Sam: So the overall picture is: don't just buy a bigger model. Build a better pipeline for delivering evidence, constrain the AI so it can't go off-script, and build proper tools to diagnose exactly where things break down.
Alex: That's the central message. A system built this way is auditable — you can always trace why it made a particular decision. It's reliable, because the constraints prevent it from improvising in dangerous ways. And it's easier to improve over time, because when something goes wrong, you know exactly which part of the pipeline to fix. Thanks for listening to ResearchPod.