Gemma Team, Google DeepMind
7 min
Abstract
We introduce Gemma 4, a new generation of open-weight, natively multimodal language models in the Gemma model family. Designed to advance compute efficiency and reasoning, the Gemma 4 model suite features dense and Mixture-of-Experts architectures, ranging from 2.3B to 31B parameters. Alongside improved vision and audio encoders for all model sizes, we propose a unified, encoder-free architecture for our 12B model, which ingests raw audio and image patches. Furthermore, we integrate a thinking mode, enabling Gemma models to generate reasoning traces prior to responding. We improve inference speed, memory, and compute efficiency, as well as long-context abilities through critical design choices. Gemma 4 establishes a leap in performance across STEM, multimodal, and long-context benchmarks, and rivals larger, frontier open models in human-rated tasks.
Sam: And that helps with things like maths or coding problems?
Alex: Notably so. The performance gains on those kinds of tasks are meaningful. It is a post-training addition—meaning it is layered on top of the base model's existing capabilities rather than baked in from scratch.
Sam: Longer reasoning traces must use more memory, though. How do they manage that?
Alex: They optimised something called a KV cache. Here is the simplest way to think about it: imagine you are reading a long novel. You do not re-read every previous chapter before starting a new page—you just remember what happened. The cache works the same way, storing a running summary of everything the model has already processed so it does not have to recalculate from the beginning each time. By sharing parts of that cache across different layers of the model, they reduced its memory footprint significantly.
Sam: So the model is working smarter about what it keeps in short-term memory.
Alex: Exactly. And there is a related technique for speeding up how quickly the model produces text. A smaller, faster companion model drafts a few words ahead of time, and the main model rapidly checks and corrects that draft. It is a bit like having an assistant write a rough version that you quickly proofread. The report calls this speculative decoding, and the suggestion is that the impact on output quality is minimal—the main model still verifies every word.
Sam: So you get speed without sacrificing accuracy. What about making these models small enough to run on a phone or a laptop?
Alex: That is where quantization comes in. Think of it like compressing a high-resolution photograph into a smaller file. You lose a tiny amount of detail, but the file takes up far less space. Quantization does the same thing to the model's internal settings—it stores them using less precise numbers, which shrinks the overall size. The problem is that compressing a finished model usually degrades its performance noticeably.
Sam: So how do they avoid that?
Alex: They train the model with that compression in mind from the very beginning. Rather than building a full model and then squeezing it down, the model learns to be resilient to lower precision throughout training. The result is a much smaller model that still performs close to its full version—which is a meaningful step toward making these systems genuinely accessible on everyday hardware.
Sam: The report also covers a larger model with a different architecture entirely. How does that one work?
Alex: This one uses a different design philosophy. Rather than one large, unified network that activates fully for every question, it is divided into smaller specialised sub-networks. For any given input, only the most relevant sub-networks switch on. The rest stay idle.
Sam: Like having a team of specialists rather than one generalist doing everything at once.
Alex: That is a good way to put it. The total model is large, but at any given moment, only a fraction of it is active. You get the knowledge base of a large system with the running costs of a much smaller one.
Sam: And safety? These models are being released openly, which means anyone can use them.
Alex: The report describes a layered approach. Training data was filtered to remove sensitive personal information. Post-training techniques were applied to encourage the model to decline requests that could lead to harmful or private disclosures. And the reasoning trace we discussed earlier has a useful side effect here—by thinking through a request before answering, the model has an opportunity to recognise when something crosses a line before it commits to a response.
Sam: So the same mechanism that helps with maths problems also helps with safety judgements.
Alex: That is the suggestion, yes. Though it is worth noting the authors frame this carefully—it is a design feature that supports safety, not a guarantee of it.
Sam: Stepping back, what is the most significant takeaway from this report for someone following AI development?
Alex: The clearest signal is a shift in priorities. For years, the dominant approach was simply to build larger models. This report reflects a different question: can we make models that are architecturally smarter, so that size becomes less important? The encoder-free design, the cache optimisation, the speculative decoding—these are all answers to that question. The direction of travel is away from "how big can we make it" and toward "how efficiently can we make it work on hardware people actually have."
Sam: And if that trend continues, what does it look like?
Alex: The next generation of these models may treat all sensory input—text, images, audio—as a single unified stream, with no specialist translators required at all. That would make building things like on-device voice assistants considerably more straightforward. What this report demonstrates is that careful architectural choices can translate into real, practical improvements—not just on benchmarks, but in where and how these models can actually be used.
Sam: That is a useful frame for thinking about where this field is heading. Thanks for walking me through it, Alex.
Alex: It was a pleasure. And thank you for listening to ResearchPod.