Tanmay Sah
8 min
How can exact, deterministic multi-digit arithmetic and symbolic logic be embedded into the weights of large language models (LLMs) without retraining, gradient fine-tuning, or relying on external code-execution interpreters?
The author develops a declarative compiler and a set of mathematical primitives that allow for direct "circuit-grafting" into the weights of existing Transformers. The approach utilizes three primary mechanisms:
The compiler automatically translates high-level arithmetic and Boolean specifications into constant-depth DAGs without requiring any training data.
The grafted circuits demonstrate robust performance across multiple architectures, including IBM Granite 4.2 3B and Alibaba Qwen 3.5 4B. The resulting models achieve 99.17% to 100% accuracy on 6-digit arithmetic operations. Notably, this is achieved with negligible impact on natural language retention, passing rigorous pre-registered benchmarks. Furthermore, the author implements a factorized block-sparse sidecar that maintains a footprint of less than 0.06% of the base model's parameters, allowing for efficient inference even with INT4-quantized weights.
This work challenges the notion that LLMs must rely on external tools for reliable multi-step computation. By embedding circuits directly into weight-space, the approach enables portable, air-gapped deterministic computation that remains active during batch decoding. It provides a new paradigm for mechanistic interpretability and supply-chain security, revealing that LLMs can be modified to hold reliable "sub-brains" without compromising their general linguistic fluency.
Pretrained language models (LMs) remain unreliable on exact multi-digit arithmetic without gradient fine-tuning or external tool interpreters. We introduce closed-form, zero-gradient circuit-grafting primitives that compile deterministic computation directly into pretrained LM weights using zero training examples and zero runtime hooks, while bounding per-layer normalization drift and preserving natural-language (NL) competence. Our construction combines three analytical mechanisms: (1) two-layer Mellin-GLU quadrature over bias-free SwiGLU/GeGLU layers, turning unknown per-token RMSNorm scaling into translation along a log-spaced gate grid ($>250\times$ dynamic range) with IEEE 754 error bounds; (2) cross-token scale cancellation via signed Chebyshev Indefinite-RoPE self-sink attention and memoryless SiLU-antisymmetric GatedDeltaNet shifts; and (3) axis-aligned width extension ($d \to d+K$) with $\sqrt{d/(d+K)}$ norm compensation, Base-1000 canonical BPE emission, and differential $(+254,-256)$ tied unembeddings. An automated 5-stage compiler synthesizes 4-8 digit signed $\{+,-,\times\}$ circuits in constant $\mathsf{TC}^0$ depth alongside 3-literal Boolean (100%, $N=1{,}728$; plus 42-layer Gemma 4 E4B audits) and 5-bit Majority/Parity (100%) circuits. On 6-digit arithmetic ($N=1{,}200$, BF16), IBM Granite 4.2 3B (1.07M-1.33M active non-zeros, $\le 0.036\%$) and Alibaba Qwen 3.5 4B (2.25M-2.39M non-zeros, $\le 0.057\%$) reach 99.17% and 100.00% ALU-prefix accuracy (100% with an FP32 sidecar) and 98%-100% mid-sentence resumption, outperforming 12.8K-sample LoRA (42.2% / 61.7%, 0% inline continuation). A factorized block-sparse sidecar holds A100 memory at 7.89 / 10.65 GiB (4.60 / 8.14 GiB under INT8/INT4 + FP32 sidecar) while passing all 14/14 pre-registered $>2$K-token Tango splits ($N=4{,}392$; 0/11,134 OOD triggers).
Alex: So a wire meant to represent a fixed value no longer has a fixed magnitude. How does the circuit recover its meaning?
Sam: It carries a reference alongside the value. Normalization multiplies both by the same unknown factor, so their ratio preserves the information. The challenge is recovering that ratio using operations the pretrained architecture actually provides.
Alex: What is the construction that makes that possible?
Sam: It uses Mellin quadrature, a weighted collection of gates at geometrically spaced scales. Think of the unknown rescaling as sliding along that collection. The combined response stays nearly flat over the supported range, instead of collapsing like a gate calibrated at just one scale.
Alex: That explains a value at one position. Arithmetic also needs digits from earlier positions. Does the reference travel with them?
Sam: Yes. Routing moves both the payload and its reference, allowing the receiving circuit to cancel the source’s scale. The paper develops different routing constructions for the Granite and Qwen architectures. It also compensates normalization for the expanded width, limiting disturbance to the language coordinates.
Alex: Then calculation is one interface problem, and returning its answer to the language model is another. What makes the second interface difficult?
Sam: Tokenization. Byte-pair encoding, which can merge several digits into one token, means identical-looking text can have different internal representations. Granite normally uses multi-digit chunks. An early emitter instead wrote each digit separately.
Alex: If the visible answer was still correct, how did they show that those internal chunks mattered?
Sam: They tested the unmodified Granite model with the same text prefix, changing only the final integer’s tokenization. With twelve forty-five split into separate digits, it produced a version-like string with dots. With its canonical chunks, it continued the calculation correctly.
Alex: That is a useful control because the grafted weights are absent. It isolates the representation the original model receives.
Sam: And the failure was consequential. Across the early emitter’s reasoning regressions, nearly all activated calculations were mathematically correct. The model nevertheless miscopied spans, inserted decimal points, or doubted correct results. Emitting native numeric chunks largely repaired that damage.
Alex: Let’s turn to the headline evidence. When the paper says exact arithmetic, what does it actually score?
Sam: The main arithmetic evaluation used twelve hundred unseen signed expressions. It covered bounded addition, subtraction, and multiplication, with a tighter size restriction on products. The paper separately scores circuit-driven answer emission, the leading integer in generated text, and whether prose resumes naturally.
Alex: That separation seems essential. How well did the circuit itself do?
Sam: On Qwen, the circuit-driven arithmetic prefix was correct on every expression in that evaluation, using the model’s native reduced precision. Granite reached about ninety-nine percent. Higher-precision circuit execution removed Granite’s remaining prefix errors in the reported setting.
Alex: And compared with teaching the same models arithmetic through training?
Sam: The paper trained low-rank adapters, a standard parameter-efficient fine-tuning method, on roughly thirteen thousand examples. Those baselines scored substantially below the grafts on arithmetic. They also failed the fluent inline-continuation test, unlike the grafted models.
Alex: But a correct prefix is not necessarily a correct completed answer. What happens after the circuit switches off?
Sam: The original model takes over immediately. On a bare equation, it sometimes appends another digit or echoes a chunk from the prompt. In prose, surrounding words help anchor the continuation. Reported inline resumption was ninety-eight to one hundred percent, but that was a small, separate test.
Alex: And the headline arithmetic setting is not the whole robustness story?
Sam: Broader tests with more surface formats gave lower overall accuracy. Some formats deliberately tested syntax the parser rejects. Automatically compiled circuits also scored below the strongest frozen reference patches. The claim is a bounded construction, not universal arithmetic reliability.
Alex: Preserving language ability is the other major claim. How much evidence supports that?
Sam: Both grafts passed all fourteen preregistered retention evaluations within their specified noninferiority margins. These included reasoning benchmarks and longer-context tests. That supports retention on the evaluated tasks, not identical behavior on every prompt; open-ended generations could still diverge.
Alex: There is also a deployment question. Does appending coordinates make the model much bigger to run?
Sam: A straightforward dense implementation wastes memory storing structural zeros and substantially expands the model. The paper implements a sidecar: factorized execution of only the active circuit operations. Its memory savings are relative to that expanded dense implementation, not the untouched base model.
Alex: So the implementation matters as much as the mathematical recipe?
Sam: Precision and kernels matter too. Reduced precision can shift a carry threshold, and some fast-math behavior breaks the intended off-trigger silence. The recommended mixed setup keeps the circuit at higher precision while allowing the language weights to be quantized.
Alex: A circuit that stays silent except on selected inputs also sounds like a checkpoint-security concern. Does the paper address that?
Sam: It does. The author argues that dormant input-triggered computations can evade ordinary behavioral checks. The proposed response combines behavioral evaluation with structural weight audits and cryptographic provenance. Reliable computation and concealed checkpoint behavior are linked consequences of this construction.
Alex: Who should read the full document, and where should they start?
Sam: Reliability researchers should start with Experimental Evaluation, then the appendix on deterministic prefix exactness versus post-answer continuation. Theory readers should examine the analytical primitives and their numerical assumptions. Deployment researchers should read the sidecar implementation; checkpoint auditors should read the Impact Statement. This is a preprint with substantial internal testing, not a demonstration that arbitrary programs now run reliably inside any language model.
Alex: And what should everyone else carry away?
Sam: Exact calculation is not enough: the answer must also arrive in a form the language model can use.
Alex: Thanks for listening.