KARLA: Knowledge-base Augmented Retrieval for Language Models

As language models (LLMs) grow in popularity, their reliance on internal, parametric memory for factual knowledge presents significant challenges: models hallucinate, struggle with long-tail entities, and become outdated as world knowledge changes. The authors investigate whether an LLM can be trained to offload factual storage to an external, structured knowledge base (KB) by learning to issue inline queries during text generation.

The authors propose KARLA (Knowledge-base Augmented Retrieval for Language Models). Instead of relying on standard retrieval-augmented generation (RAG) or general tool-use, KARLA fine-tunes a base LLM to generate specific, relation-aware tokens that act as triggers for KB lookups.

Key components of the approach include:

• Synthetic Training: The authors generate a training corpus by sampling facts from a KB and prompting a teacher model to write encyclopedic paragraphs that interleave natural language with these specific query tokens.
• Masked Training Loss: During training, the model is not penalized for failing to predict the factual object itself; instead, it is trained to correctly identify the subject and relation, while the factual object is provided by the KB.
• Inference Pipeline: At runtime, the model identifies entities in the prompt, uses a bi-encoder and re-ranker to map them to canonical KB entities, and pauses generation to fetch the required fact when a query token is emitted.

Experiments on the YAGO (general domain) and PrimeKG (biomedical) knowledge bases demonstrate that KARLA effectively separates linguistic competence from factual storage.