Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a system called FreeToken — designed to run massive artificial intelligence models on everyday personal computers, without needing expensive cloud servers.
Sam: So the core question is: why is running these big models on a home machine so difficult in the first place?
Alex: It comes down to size and speed. The models people want to run locally have grown enormous — we're talking about systems with hundreds of billions of learned settings that make the model work. A standard gaming PC simply doesn't have enough fast memory to hold all of that at once.
Sam: So the model doesn't fit in the fast memory on the graphics card?
Alex: Right. The graphics card — the GPU — is where the actual number-crunching happens, and it's very fast. But its memory is limited. So the overflow has to sit in the computer's main system memory, which is slower and farther away. Every time the model needs a piece that isn't on the GPU, it has to fetch it across a connection called PCIe — and that connection becomes a bottleneck. Think of it like a highway that's too narrow for the traffic it's being asked to carry.
Sam: And I'd imagine that problem gets worse with certain types of models?
Alex: Exactly right. The models that make this especially sharp are called Mixture of Experts models. Here's the idea: instead of one giant network that's always fully active, you have many smaller specialist networks — the "experts" — and for any given input, only a small handful of them actually switch on. It keeps the math manageable, but the full set of experts is still enormous, so most of them have to wait in system memory.
Sam: So you're constantly fetching whichever experts the model decided to use for this particular input.
Alex: And that fetching is unpredictable. The model chooses different experts for every single token it generates — every word, essentially. So you can't just preload the right ones ahead of time. You're constantly hitting that narrow highway.
Sam: So how does FreeToken get around it?
Alex: The key insight is to stop treating the connection speed as a fixed wall and start treating it as a scheduling signal. FreeToken does two things simultaneously. First, while the GPU is computing the current layer, a background process is already loading the next layer's data across the connection — so by the time the GPU finishes, the next batch is ready and waiting. It's like a kitchen where one chef is cooking while another is already prepping the next course.
Sam: That hides the loading time by overlapping it with useful work.
Alex: Exactly. But the more novel part is what happens when an expert the model needs isn't already on the GPU. FreeToken doesn't just wait for the GPU to fetch it. Instead, it splits the work. Some missing experts get streamed across to the GPU. Others get computed directly on the main processor — the CPU — right where they already live in system memory. You're running two parallel paths at once, rather than waiting in a single queue.
Sam: How does it decide which experts go where?
Alex: It measures the actual transfer speed of the connection in real time, then figures out how much processing capacity the CPU has left over. Because both the data transfer and the CPU computation draw from the same pool of system memory, once you know how much the connection is using, you can calculate exactly how much is left for the CPU to use in parallel. The system assigns experts to each path in that proportion — so neither the connection nor the CPU is sitting idle.
Sam: What happens if the memory situation changes mid-run — like if another app opens and takes some memory?
Alex: FreeToken handles that gracefully because it treats the CPU's copy of the model as the permanent, authoritative version. The GPU cache is treated as a performance layer, not a requirement. So if available GPU memory shrinks, the system can quietly rebuild its GPU cache to fit the new budget — without crashing or restarting the whole engine. Correctness is never at risk. Only performance varies with available memory.
Sam: Are there situations where this approach doesn't work as well?
Alex: Yes, and the paper is candid about it. The whole pipeline depends on a hardware feature that lets the GPU pull data from system memory without the CPU having to manage every transfer manually. It also requires a specific memory configuration that locks certain data in place so it can be transferred reliably at high speed. If a machine doesn't support those features well, the parallel pipeline breaks down, and the system falls back to a slower, simpler approach. So the gains are real on relatively modern consumer hardware, but they're not universal.
Sam: When you step back, what does this kind of work point toward?
Alex: The paper suggests the logical next step is to extend this across multiple devices. If one consumer machine can serve a large model by intelligently splitting work between its CPU and GPU, then several machines on a local network could potentially pool their resources — sharing experts across devices the way FreeToken shares them across components inside one machine. That could eventually make very large models accessible without any centralized cloud infrastructure at all.
Sam: It reframes the question. Instead of asking how to build bigger servers, you ask how to make smarter use of hardware that already exists.
Alex: That's the central idea the research is pointing toward. Thanks for listening to ResearchPod.