Speculative decoding (SD) addresses the high inference costs of LLMs by having lightweight drafters generate candidates for large verifiers to validate in parallel. Existing draft-verify methods use binary decisions: accept or fully recompute. Yet we find that many rejected tokens can be verified correctly by a slim submodel derived from the full verifier via intra-model routing, instead of the full verifier. This motivates our slim-verifier to handle tokens requiring moderate verification resources, reducing expensive large-model calls. We propose Verification via Intra-Model Routing for Speculative Decoding (VIA-SD), a multi-tier framework using a routed slim-verifier. Draft tokens are processed hierarchically: direct acceptance for high-confidence cases, slim-verifier regeneration for medium-confidence cases, and full-model verification for uncertain cases. Across four representative tasks and multiple model families, VIA-SD reduces rejection rates by 0.10-0.22 and delivers 10-20% speedups over strong SD baselines, while achieving 2.5-3x acceleration over non-drafting decoding. Moreover, VIA-SD is compatible with existing SD frameworks without modifying their training procedures. Our results suggest multi-tier SD as a general paradigm for scalable and efficient LLM inference. Project page: https://zju-xyc.github.io/VIA-SD-Project-Page/
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper called "VIA-SD," which proposes a new way to make large AI models run faster and more efficiently.
Sam: So is this basically asking: how do we stop wasting enormous computing power on simple tasks?
Alex: That's exactly the problem. To understand it, you need to know how these AI systems currently work. When a large language model generates text, it doesn't write everything itself from scratch. There's a technique where a smaller, cheaper model drafts the words first, and then the big model checks whether those drafts are good enough to keep. It's called speculative decoding.
Sam: So the small model does the rough work, and the big model is the quality checker?
Alex: Right. And most of the time, the small model's guesses are fine. But the system uses a blunt rule: if the small model's guess doesn't meet the standard, the big model takes over entirely. There's no middle ground. It's all-or-nothing.
Sam: That does seem wasteful. Like calling in a surgeon every time someone has a headache.
Alex: That's a good way to put it. So the researchers asked: what if we added a middle tier? Not just a small model and a giant model, but something in between — a "slim verifier" that can handle the moderately tricky cases without pulling in the full power of the largest model.
Sam: And how do they build this middle tier? Do they just train a whole new model from scratch?
Alex: No, and that's one of the more practical aspects of the approach. Instead of building something new, they take the big model itself and selectively skip certain internal processing steps. Imagine a large factory assembly line with twenty stations. For a simple product, you might only need stations one, five, ten, and twenty — you skip the rest. The result is faster and cheaper, but it's still the same factory, using the same knowledge.
Sam: So the slim verifier isn't a separate thing — it's the big model running in a stripped-down mode?
Alex: Exactly. Because it shares the same underlying knowledge as the full model, it's much better at catching errors than a completely separate small model would be. They call this approach "intra-model routing" — routing work through different depths of the same model depending on how hard the task is.
Sam: But how does the system know, in the moment, whether a given word or phrase needs the slim verifier or the full model?
Alex: This is where a statistical tool called KL divergence comes in. Here's the plain version: every time the small model suggests a word, you can compare its level of confidence to what the big model would have expected. If those two pictures of the situation look very similar, the task is probably routine. If they look quite different, something unusual is happening and it needs more scrutiny.
Sam: So it's like a disagreement meter. A small disagreement means the slim verifier can sort it out. A large disagreement means escalate to the top.
Alex: That's a precise way to describe it. And because this check happens automatically for every single word the system generates, the routing is continuous and dynamic — not a one-time decision at the start of a task.
Sam: So you end up with three tiers: the small draft model, the slim verifier for moderate cases, and the full model only for the genuinely hard ones.
Alex: Exactly. And the practical consequence is that the full model gets called on far less often. The paper suggests this reduces what they call the "rejection rate" — the frequency with which drafts get thrown out and reprocessed — by up to twenty-two percent compared to existing methods. That translates into meaningful speedups without a measurable drop in the quality of the output.
Sam: It's a bit like how a well-run organisation works. Most decisions are handled at the team level, some go to a manager, and only the genuinely consequential ones reach the top.
Alex: That's a fair comparison. The underlying insight is that not all problems are equally hard, and treating them as if they are wastes resources. By building a system that can distinguish between routine, moderate, and genuinely difficult cases — and route each one appropriately — the researchers show it's possible to maintain accuracy while using significantly less computational energy. It's a practical contribution to a real bottleneck in how these models are deployed.
Sam: And given how much energy large AI systems consume, finding ways to do more with less seems like a worthwhile direction.
Alex: The paper makes a reasonable case that it is. Thanks for listening to ResearchPod.