Yuntian Deng, Pengyu Nie, Stuart Shieber
6 min
Abstract
Many recurring text functions are easy to describe but difficult to implement with rules, while calling a large remote model for every input introduces repeated cost, latency, and dependency on a provider. We present compile by training, which turns a natural-language specification into a reusable neural function. At compile time, teacher models generate task-specific examples that are used to train a small adapter for a compact interpreter. The resulting function runs without the teachers and can be stored, versioned, and composed like ordinary software. On FuzzyBench-Hard, a subset on which the Program-as-Weights fast compiler produced no exact matches, compile by training reaches 83.6% semantic accuracy. This higher accuracy comes with a higher compile-time cost: roughly a minute rather than seconds for the fast compiler. We deploy the compiler in a public interactive service and demonstrate compiled functions in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.
Sam: How do they evaluate whether the compiled function is actually doing the right thing? Exact string match seems obviously too strict for natural language outputs.
Alex: They use what they call LLM Exact Match — a stronger model, GPT-4.5 in their setup, acts as a semantic judge. It evaluates whether the compiled function's output matches the intent of the specification, not the literal string. They show that naive string matching misses a meaningful fraction of semantically valid outputs, so this judge-based metric is load-bearing for their accuracy claims.
Sam: So the headline number — over 83% semantic accuracy on tasks where previous fast compilers failed entirely — that's measured against that judge.
Alex: That's the main finding, yes. And to be clear about what "failed entirely" means: the comparison class is methods that try to do fast, amortised compilation without the refinement step. Those collapse on tasks with complex or ambiguous specifications. The compile-by-training approach recovers most of that gap.
Sam: Did they look at how much synthetic data you actually need? There's a real practical question about whether the teacher call costs are worth it.
Alex: They did. Scaling from roughly 1,400 unique pairs to 3,600 improved accuracy, but the gains plateaued beyond that. So you're not looking at an open-ended data collection problem — there's a practical ceiling where additional teacher calls stop paying off, and it's within a range that's economically reasonable for a one-time build step.
Sam: And the composability question — can you chain these functions, or is each adapter strictly siloed?
Alex: They demonstrate composability directly. The paper includes a website helper that routes incoming questions through a tree of specialised PAW programs, mixed with conventional procedural code. Each node in the tree is an independently compiled adapter. So the architecture supports the kind of modular decomposition you'd want for anything beyond a toy task.
Sam: The versioning angle is what I keep coming back to. Right now, prompt engineering is a runtime activity — you're negotiating with a model every time. This turns it into a build-time activity with reproducible outputs.
Alex: That's the core reframing. The authors are explicit that the goal is to make LLM-backed functions behave like ordinary software components — deterministic given fixed inputs, auditable, deployable without cloud dependency. The public playground they've released lets developers test the build pipeline directly, which is a reasonable way to stress-test whether that framing holds in practice.
Sam: The natural next question is whether you'd want a package registry for these adapters. If someone's already compiled a high-quality JSON summariser, why recompile it?
Alex: That's the logical extension, and the authors gesture at it. If the artifact is truly portable — same frozen interpreter, same adapter format — then redistribution is straightforward. Whether the ecosystem develops that way depends on how stable the interpreter layer stays across versions, which is an open engineering question they don't fully resolve.
Sam: And the teacher drift problem becomes more acute in a shared registry, because you're now trusting someone else's compile-time teacher.
Alex: Exactly. That's where the limitation bites hardest. For now, the practical scope is probably internal tooling — teams compiling functions against specifications they control, using teachers they've evaluated. The broader redistribution story needs a solution to provenance and drift that this paper doesn't yet provide.
Sam: So: a concrete, working approach to a real bottleneck, with a clear limitation that honest reviewers would push on.
Alex: That's a fair summary. The mechanism is sound, the accuracy gains over the amortised baseline are the load-bearing result, and the teacher drift issue is the constraint that most limits where you'd trust this in production. Thanks for listening to ResearchPod.