LCM team, Loïc Barrault, Paul-Ambroise Duquenne, Maha Elbayad, Artyom Kozhevnikov, Belen Alastruey, Pierre Andrews, Mariano Coria, Guillaume Couairon, Marta R. Costa-jussà, David Dale, Hady Elsahar, Kevin Heffernan, João Maria Janeiro, Tuan Tran, Christophe Ropers, Eduardo Sánchez, Robin San Roman, Alexandre Mourachko, Safiyyah Saleem, Holger Schwenk
9 min
Large Concept Models (LCMs) represent a bold shift from the token-by-token world of traditional Large Language Models (LLMs) to reasoning at a higher, more human-like level of abstraction. While LLMs like Llama, Mistral, and GPT excel at next-token prediction using transformer decoder architectures, they lack the explicit hierarchical planning humans use—outlining high-level ideas before filling in details. LCMs address this by operating entirely in a semantic 'concept' embedding space, where concepts are language- and modality-independent units roughly equivalent to sentences. This enables zero-shot generalization across 200 languages and modalities (text and speech), without retraining.
Current LLMs process everything at the granular token level, requiring massive data (trillions of tokens) and compute to scale. This data-driven approach struggles with explicit reasoning: humans plan a talk by sketching a flow of ideas, not scripting every word. LLMs implicitly learn hierarchies but can't easily edit at abstract levels or handle long contexts efficiently. Knowledge is baked into weights, making multilingual or multimodal extension costly via synthetic data injection.
LCMs define 'concepts' as atomic, abstract ideas—typically a sentence's worth of meaning. They use SONAR, a fixed encoder-decoder model supporting 200 languages (text) and 76 (speech). SONAR maps sentences to a dense embedding space where semantic operations like summarization work regardless of input language or modality. This 'abstract embedding space' is modality-agnostic: encode English speech, reason in concepts, decode to French text seamlessly. No tokenization biases or language-specific quirks.
LCMs are autoregressive models predicting the next concept embedding given prior ones. Trained on ~2.7T tokens (1.6B then 7B parameter models), they explore MSE regression and diffusion-based generation (including quantized SONAR space). Diffusion adds noise to embeddings and denoises step-by-step, improving sample quality. This setup yields coherent sequences of concepts, forming hierarchical structures naturally—ideal for long-form generation where global structure matters.
Explicit hierarchy boosts readability, allows local edits (tweak one concept without regenerating all), and sidesteps transformer context limits. LCMs shine in generative tasks: summarization and 'summary expansion' (elaborating bullet points into full text). They outperform same-size LLMs in zero-shot settings across languages, thanks to universal concept reps. Training code is open-sourced, democratizing this approach. LCMs hint at future AI mimicking human top-down planning, scaling reasoning beyond tokens.
LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to humans who operate at multiple levels of abstraction, well beyond single words, to analyze information and to generate creative content. In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept. Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow. Hence, we build a "Large Concept Model". In this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. We explore multiple approaches, namely MSE regression, variants of diffusion-based generation, and models operating in a quantized SONAR space. These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We then scale one architecture to a model size of 7B parameters and training data of about 2.7T tokens. We perform an experimental evaluation on several generative tasks, namely summarization and a new task of summary expansion. Finally, we show that our model exhibits impressive zero-shot generalization performance to many languages, outperforming existing LLMs of the same size. The training code of our models is freely available.
Alex: So the Two-Tower keeps context processing separate from the cleanup, like having a planner and an artist working together.
Sam: Precisely. This separation helps the denoiser focus purely on denoising. They train both with and without priors, then blend at generation to balance following the sequence and adding variety.
Alex: So with these architectures in place, how do they actually measure if the models are predicting sensible next ideas?
Sam: They test pre-trained models by feeding in real document prefixes and checking how well the predicted next sentence embedding matches the true one. One key measure looks at the straight-line distance in the SONAR space between the prediction and reality—shorter means closer fit. They also decode the prediction to text, re-encode it, and measure again to catch if it strays from natural sentences.
Alex: Like double-checking if the idea holds up after translating back to words.
Sam: Yes. They also check if the prediction picks the right next idea over distractors—that's contrastive accuracy. Across datasets like stories and Wikipedia, diffusion-based models outperform the simpler Base-LCM on coherence.
Alex: So the continuations feel more naturally tied to what came before?
Sam: Exactly—it's a sign of stronger overall text flow. After instruction-tuning on story data, these diffusion models score close to a comparable small Llama model on coherence.
Alex: That makes the abstract planning more reliable... without getting bogged down in words.
Alex: Building on that, how does this hold up computationally when generating long sequences?
Sam: A key advantage shows in inference efficiency—the total computing power needed to run predictions on longer texts. Standard language models ramp up costs quadratically as context grows, like a snowball effect from attention layers comparing every pair of words. Large Concept Models scale more linearly in the sentence space, costing less for typical sentence lengths.
Alex: So for real documents with dozens of sentences, the LCM uses less power overall.
Sam: Yes, including the steps to encode into SONAR space and decode out. They also analyzed fragility in that SONAR space—how a tiny nudge to an embedding vector can scramble the decoded text's meaning, like a small edit ruining a puzzle.
Alex: Filtering fragile samples makes sense for stability... but did scaling the model reveal more?
Sam: They scaled the Two-Tower diffusion LCM to 7B parameters, pre-training on 2.7 trillion tokens across 2.3 billion documents, then instruction-tuning on public data. On abstractive summarization—distilling long articles to key points—they match or exceed baselines like Mistral-7B on ROUGE-L, a measure of content overlap with ideal summaries, while copying less directly from sources.
Alex: A meaningful edge in handling structure without word-for-word mimicry.
Alex: That edge in structure makes me wonder about non-English tasks... since SONAR covers 200 languages, does the LCM generalize zero-shot to summarization in other tongues?
Sam: Yes, they test this on XLSum, a benchmark for abstractive news summaries across 45 languages. The Two-Tower-7B-IT LCM, trained only on English, averages better over the six languages both it and Llama-3.1-8B-IT officially support. It also performs strongly on low-resource ones like Pashto, Burmese, and Hausa.
Alex: So it holds up across languages it never trained on... but what about reversing summaries, like expanding them back to full articles?
Sam: On summary expansion for CNN DailyMail and XSum, LLMs like Llama-3.1-8B-IT get higher ROUGE-L by recreating more original document content, but the LCM generates paraphrased sentences instead. This leads to lower copying and notably less fluent output on DailyMail.
Alex: Paraphrasing fits the abstract space idea... but for long texts, doesn't it need better planning to stay coherent?
Sam: Exactly—LLMs often lack explicit multi-level planning, like outlining macrostructures before details. To address this, they introduce a planning setup where the model predicts concepts until a special break concept signals a topic shift. Then a planning component generates a high-level plan concept summarizing the next chunk, conditioning further generation for better flow. In a simplified single-model version, called LPCM, they train multitask on paragraph-segmented data with synthetic plans from Llama-3.1-8B-IT.
Alex: So inserting these plan summaries mimics human outlining... how much does that help?
Sam: In an ablation on Cosmopedia after instruction-tuning, the LPCM scores higher coherence via LLM-as-judge than the baseline One-Tower LCM. The paper notes this as an initial step toward explicit hierarchical reasoning.
Alex: A clear lift from that added layer... grounds the abstract predictions nicely.
Alex: That planning layer shows promise for longer texts... but what are the main limits holding this back from matching top LLMs right now?
Sam: A central limit is the embedding space itself—like SONAR, which maps sentences to points in a high-dimensional map designed mainly for short translation pairs. It excels at keeping very similar ideas close together locally, but struggles with looser connections across a whole document, and it's fragile on things like links or numbers that don't match its training data.
Alex: So the map works nearby but not for the big picture... and picking sentences as the unit size adds its own issues?
Sam: Yes. Sentences vary wildly in length and content, making the space of possible next ones vast and sparse—most are unique, unlike repeated words in token models. Diffusion helps with that continuous variety, but text's underlying discrete nature means it can't easily use sharp contrastive checks.
Alex: Trade-offs in the foundation and scale... yet it still generalizes zero-shot across languages better than some peers.
Sam: The paper positions LCM as a step toward models that treat all languages and even other inputs like speech as equals, without special tweaks. By reasoning on abstract ideas, it sidesteps word-level biases, opening paths to true multilingual planning at sentence, paragraph, or document scales.
Alex: A meaningful shift from token-by-token guessing, then—prioritizing ideas for broader reach, even if fluency lags for now. The open-sourced code should help others build on it.
Sam: Precisely. This demonstrates feasibility for language-agnostic reasoning, fostering diversity beyond current LLM norms.
Alex: That's our close look at Large Concept Models today. Thanks for joining us on ResearchPod.