Shuo Yang, Xiaoze Fan, Melissa Pan, Haocheng Xi, Zhe Wang, Shanlin Sun, Kurt Keutzer, Song Han, Matei Zaharia, Chenfeng Xu, Ion Stoica
5 min
Abstract
Frontier open-weight models are increasingly available, but serving them still largely assumes datacenter infrastructure. We present FreeToken, an edge-native MoE serving system that treats a personal machine not as a small GPU, but as a unified, elastic inference platform. FreeToken co-designs the full serving stack, including model layout and loading, expert residency, CPU--GPU execution, agentic state reuse, and runtime memory management, around two realities of local AI: agent workloads continuously change their execution pattern, and edge hardware exposes heterogeneous resources whose balance differs from machine to machine. Rather than committing to a fixed offloading strategy, FreeToken continuously maps computation and model state onto the resources actually available. FreeToken supports more than 20 MoE models and real coding and tool-using agents across hardware ranging from an 8GB laptop GPU to a single workstation GPU. More importantly, it changes what these machines can practically serve, from a 35B model on a laptop to a 284B model on a gaming desktop and the 753B GLM-5.2 on a single workstation GPU. FreeToken turns open weights into deployable local software, making the machines users already own a practical platform for frontier-scale intelligence. We release the system at flashml.ai.
Alex: Exactly. But the more novel part is what happens when an expert the model needs isn't already on the GPU. FreeToken doesn't just wait for the GPU to fetch it. Instead, it splits the work. Some missing experts get streamed across to the GPU. Others get computed directly on the main processor — the CPU — right where they already live in system memory. You're running two parallel paths at once, rather than waiting in a single queue.
Sam: How does it decide which experts go where?
Alex: It measures the actual transfer speed of the connection in real time, then figures out how much processing capacity the CPU has left over. Because both the data transfer and the CPU computation draw from the same pool of system memory, once you know how much the connection is using, you can calculate exactly how much is left for the CPU to use in parallel. The system assigns experts to each path in that proportion — so neither the connection nor the CPU is sitting idle.
Sam: What happens if the memory situation changes mid-run — like if another app opens and takes some memory?
Alex: FreeToken handles that gracefully because it treats the CPU's copy of the model as the permanent, authoritative version. The GPU cache is treated as a performance layer, not a requirement. So if available GPU memory shrinks, the system can quietly rebuild its GPU cache to fit the new budget — without crashing or restarting the whole engine. Correctness is never at risk. Only performance varies with available memory.
Sam: Are there situations where this approach doesn't work as well?
Alex: Yes, and the paper is candid about it. The whole pipeline depends on a hardware feature that lets the GPU pull data from system memory without the CPU having to manage every transfer manually. It also requires a specific memory configuration that locks certain data in place so it can be transferred reliably at high speed. If a machine doesn't support those features well, the parallel pipeline breaks down, and the system falls back to a slower, simpler approach. So the gains are real on relatively modern consumer hardware, but they're not universal.
Sam: When you step back, what does this kind of work point toward?
Alex: The paper suggests the logical next step is to extend this across multiple devices. If one consumer machine can serve a large model by intelligently splitting work between its CPU and GPU, then several machines on a local network could potentially pool their resources — sharing experts across devices the way FreeToken shares them across components inside one machine. That could eventually make very large models accessible without any centralized cloud infrastructure at all.
Sam: It reframes the question. Instead of asking how to build bigger servers, you ask how to make smarter use of hardware that already exists.
Alex: That's the central idea the research is pointing toward. Thanks for listening to ResearchPod.