Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds | Pierre Jourlin | ResearchPod

Transcript: Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds

Alex: Welcome to another episode of ResearchPod. Sam, I've been thinking about how AI tools often need huge computers or paid online services to handle real tasks—like organizing facts from documents into something useful.

Sam: This paper, titled "Frugal Knowledge Graph Construction with Local LLMs," by Pierre Jourlin, looks at building what's called a knowledge graph using everyday computer hardware. A knowledge graph is like a map of facts where people, places, or ideas are dots connected by labeled lines showing relationships—think of it as a web you can query to check if an answer has real evidence behind it, unlike vague chat responses. The central puzzle is whether smaller AI models, run right on your home computer without any special training, can create these maps and use them well enough for tough questions.

Alex: So this paper is basically asking if we can skip the expensive cloud services or heavy training to build reliable fact maps from documents and answer tricky questions locally?

Sam: Yes, exactly—that's the core challenge. Analysts often need to pull knowledge graphs from hundreds of documents for multi-step questions, like "Who directed the movie scored by this composer?" Cloud AI costs too much and runs too slow for batches, while trained models demand powerful servers and days of setup. This work tests a full pipeline that runs in about five hours on a single consumer GPU, like an RTX 3090, with no training at all.

Alex: Right, so the practical hurdle is making AI handle fact-linking and complex queries without big budgets or tech expertise.

Sam: Precisely. The paper sets up a test across benchmarks like DocRED for linking facts in documents and HotpotQA for questions needing two or more fact jumps. It aims for quality close to top trained systems but frugally—achieving solid relation detection at an F1 around 0.70, which measures how accurately it spots and labels connections without false alarms.

Alex: Okay, so they hit that F1 score of around 0.70 on spotting relations in documents like DocRED. But how do they pull that off without any training—just using these local models?

Sam: The approach starts with a setup called the SYNSYNTH pipeline, which links together four separate steps, each handled by a different specialized language model. It's like an assembly line where one model reads the text and picks out pairs of things—like a person and a place—and figures out how they're connected, such as "works at" or "located in." For the key step of finding relations, they use a model named Gemma with 27 billion parameters, but only 4 billion activate at a time thanks to a design where experts specialize in different tasks—imagine a team where only the right players step up for each play. This keeps it running on a home GPU like the RTX 3090.

Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds

Pierre Jourlin

Apr 13, 2026·6:50·arxiv·Original Paper

Knowledge Graph ConstructionFrugal AISelf-ConsistencyWisdom of Artificial CrowdsConfidence-Routing CascadeLarge Language Models (LLMs)

Listen

7 min

Summary

Overview

This paper by Pierre Jourlin introduces SYNSYNTH, a fully reproducible zero-shot pipeline for building and querying knowledge graphs (KGs) using quantized large language models (LLMs) that run entirely on consumer hardware like a single RTX 3090 GPU. No training is needed, and the entire system completes in ~5 hours with a tiny carbon footprint of 0.09 kg CO₂eq—making it 'frugal AI' that democratizes KG construction beyond cloud-dependent labs.

Traditional KGs provide verifiable structure to ground LLMs' hallucination-prone outputs, but building them usually requires massive supervised training on specialized hardware. SYNSYNTH flips this: it uses local LLMs for document-level relation extraction, text-to-query translation, and multi-hop reasoning, evaluated on real benchmarks like DocRED (F1=0.70 zero-shot vs. 0.80 supervised), WebQuestionsSP (80% accuracy), and HotpotQA (46% exact match, 96% RAGAS faithfulness).

The key insight? Diversity mechanisms like self-consistency and cross-model ensembling unlock performance on hard tasks, revealing counterintuitive behaviors akin to human crowds.

Zero-Shot KG Construction Pipeline

SYNSYNTH extracts relations across full documents (not just sentences), identifying entity links separated by intervening text—a challenging 'document-level relation extraction' task. On DocRED's 500 relations, it hits 70% F1 using prompts with synonyms and semantic constraints, closing 87% of the gap to supervised state-of-the-art like DREEAM (80% F1).

Quantized LLMs (e.g., Phi-4, GPT-OSS via Ollama) enable this on consumer GPUs. The pipeline then converts natural language questions to graph queries (e.g., Cypher/SPARQL) with 80% accuracy on synthetic WebQuestionsSP data, enabling KGQA without semantic parsers.

Diversity for Multi-Hop Reasoning

Multi-hop QA (HotpotQA) requires chaining facts across the KG. Base zero-shot hits 46% exact match (EM). Self-consistency—sampling multiple outputs (k=5, T=0.7 temperature) and voting—recovers 23% of the 181 'unsolvable' zero-temperature questions using a single Mixture-of-Experts model.

Wisdom of Artificial Crowds ensembles across architectures (3 models × 5 samples), reaching 46.4% EM oracle. But here's the twist: an agreement paradox emerges—strong sample consensus often flags collective hallucination, not truth, mirroring human crowd dynamics where overconfidence signals error.

Confidence-Routing Cascade

The best result? A confidence-routing cascade: Start with fast Phi-4 (self-consistency k=5). If low confidence, reroute 45% of questions to stronger GPT-OSS, boosting full-pipeline EM to 55% (+9 pts over zero-shot, +11 over voting). This adaptive routing exemplifies efficient LLM orchestration.

Prompt engineering matters: V3 prompts (synonyms + constraints) drive gains, but model-specific interactions mean tricks don't transfer universally (e.g., Gemma-4 unique).

Why It Matters

Frugal KG-LLM coupling (LLM-enhanced KG) counters LLM hallucinations with verifiable structure, runnable anywhere. The multi-task framework (DocRED + HotpotQA + RAGAS) sets a local-inference baseline, open-sourced under Hypocratic license. Insights on diversity paradoxes inform scalable reasoning, while carbon efficiency highlights sustainable AI paths.

Abstract

This paper presents an empirical study of a multi-model zero-shot pipeline for knowledge graph construction and exploitation, executed entirely through local inference on consumer-grade hardware. We propose a reproducible evaluation framework integrating two external benchmarks (DocRED, HotpotQA), WebQuestionsSP-style synthetic data, and the RAGAS evaluation framework in an automated pipeline. On 500 document-level relations, our system achieves an F1 of 0.70 $\pm$ 0.041 in zero-shot, compared to 0.80 for supervised DREEAM. Text-to-query achieves an accuracy of 0.80 $\pm$ 0.06 on 200 samples. Multi-hop reasoning achieves an Exact Match (EM) of 0.46$\pm$0.04 on 500 HotpotQA questions, with a RAGAS faithfulness of 0.96 $\pm$ 0.04 on 50 samples. Beyond the pipeline, we study diversity mechanisms for difficult multi-hop reasoning. On 181 questions unsolvable at zero temperature, self-consistency (k=5, T =0.7) recovers up to 23% EM with a single Mixture-of-Experts (MoE) model, but the cross-model oracle (3 architectures x 5 samples) reaches 46.4%. We highlight an agreement paradox: strong consensus among samples signals collective hallucination rather than a reliable answer, echoing the work of Moussa{ï}d et al. on the wisdom of crowds. Extending to the full pipeline (500 questions), self-consistency (k=3) raises EM from 0.46 to 0.48 $\pm$ 0.04. A confidence-routing cascade mechanism (Phi-4 $\rightarrow$ GPT-OSS, k=5) achieves an EM of 0.55 $\pm$ 0.04, the best result obtained, with 45.4% of questions rerouted. Finally, we show that V3 prompt engineering applied to other models does not reproduce the gains observed with Gemma-4, confirming the specific prompt/model interaction. The entire system runs in $\sim$5 h on a single RTX 3090, without any training, for an estimated carbon footprint of 0.09 kg CO2 eq.

Frugal Knowledge Graph Construction with Local LLMs: A Zero-Shot Pipeline, Self-Consistency and Wisdom of Artificial Crowds: Research Paper Podcast