Enhancing LLMs through human feedback: a journey towards self-improvement

Large Language Models (LLMs) often struggle with static knowledge bases, leading to outdated or contextually misaligned responses. While Retrieval Augmented Generation (RAG) helps by pulling external data, it frequently fails to incorporate user preferences or correct factual errors over time. This paper introduces FLARE (Feedback-driven LLM Adaptive RAG Enhancement), a framework designed to enable continuous system-level self-improvement through structured human feedback.

FLARE operates through two parallel pipelines: an offline management system and an online inference workflow.

• Offline Pipeline: User feedback is collected and enriched with conversation history. An LLM then classifies this feedback into factual corrections or stylistic guidelines. These are synthesized into structured knowledge, embedded, and indexed in a vector database.
• Online Pipeline: When a user submits a query, the system performs a dual retrieval: one for the primary RAG context and one for relevant feedback from the vector database. Factual corrections are injected directly into the context, while stylistic guidelines are applied as system-level instructions to refine the output.

The authors validated FLARE across three datasets: a synthetic Fictional Persona Dialogs set, a current-events Trivia benchmark, and a proprietary Wireless Technical Documentation dataset.

In the Trivia benchmark, which included queries beyond the model's initial training cutoff, FLARE achieved an average score of 3.91 compared to 3.06 for the baseline RAG system. The results demonstrate that FLARE effectively bridges the gap between static model knowledge and evolving real-world information, allowing the system to learn from its mistakes and adapt to specific user requirements without requiring full model retraining.