Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe | Qian Zhao et al. | ResearchPod

Transcript: Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper that re-examines how we train massive AI models using something called low-precision math.

Sam: Right. To understand what that means, think about how a computer stores numbers. Normally, a computer uses a lot of digits to represent a value very precisely—like writing out 3.14159265 instead of just 3.14. More digits means more memory and more energy. So researchers have been asking: what's the minimum number of digits we can get away with, without the model falling apart?

Alex: And "FP4" is the answer they're testing?

Sam: FP4 stands for "4-bit floating point." It's a format that uses only four binary digits—four ones and zeros—to represent each number in the model. That's extremely compact. The payoff is that training a large language model this way uses far less memory and energy. The problem is that current 4-bit recipes keep running into stability issues. The model's learning process becomes erratic, or the final result is noticeably worse than it should be.

Alex: So the efficiency is there on paper, but something keeps going wrong in practice?

Sam: Exactly. And this paper argues the culprit is something more fundamental than people realised—it's the geometric shape of the number system itself. To store a number in 4 bits, you have to round it to one of a small set of allowed values. Think of it like a ruler that only has marks at certain intervals. The question is: where do you put those marks?

Alex: And the current approach puts them in the wrong places?

Sam: That's what the authors argue. The most common 4-bit format, called E2M1, places its marks unevenly. There are more marks clustered near zero and fewer marks spread out toward larger values. Now, here's the problem with that. When you round a number to the nearest mark, and the marks aren't evenly spaced, the rounding errors aren't random—they're systematically biased. The paper calls this "Shrinkage Bias."

Alex: Can you make that concrete?

Sam: Imagine you're measuring flour with a cup that's secretly ten percent too small. Every single scoop gives you slightly less than you think. One scoop, no big deal. But if a recipe calls for a hundred scoops, you end up with far less flour than intended, and the cake fails. That's Shrinkage Bias. Because the rounding marks are unevenly placed, the rounding process consistently nudges numbers slightly toward zero. Each layer of the neural network passes that error to the next, and it compounds. By the time you reach the end of the calculation, the signal has noticeably decayed.

Alex: So the model is quietly losing information at every single step.

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe

Qian Zhao, Kunlong Chen, Changxin Tian, Zhonghui Jiang, Haitao Zhang, Chaofan Yu, Peijie Jiang, Mingliang Gong, Jia Liu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou

Jun 18, 2026·6:23·arxiv·Original Paper

Shrinkage BiasFP4 TrainingUFP4Random Hadamard Transform (RHT)Uniform vs. Non-uniform GridsStochastic Rounding (SR)

Listen

6 min

Briefing

Research Context and Problem

As Large Language Models (LLMs) continue to scale, the computational and memory costs of pretraining have become significant bottlenecks. While 4-bit floating-point (FP4) training offers a path to reduce these costs, current industry standards—such as those used in NVIDIA Blackwell and AMD MI350 systems—rely on the E2M1 format. This format is non-uniform, meaning its representable values are not evenly spaced. The authors identify that this non-uniformity leads to a systematic, negative rounding error termed Shrinkage Bias, which degrades model performance over long training runs.

The Mechanism of Shrinkage Bias

Shrinkage Bias arises from the geometric asymmetry of the rounding bins in non-uniform formats. When using Round-to-Nearest-Even (RTNE) logic, the asymmetric spacing causes a consistent bias toward zero. The authors demonstrate that this bias is not merely localized noise; it accumulates multiplicatively across the deep layers of a neural network, leading to systematic signal decay. Furthermore, while the Random Hadamard Transform (RHT) is often used to disperse outlier energy and improve quantization, the authors show that it inadvertently pushes data into the most asymmetric bins of the E2M1 grid, thereby exacerbating the shrinkage effect.

The UFP4 Solution

To address these issues, the researchers propose UFP4, a uniform 4-bit training recipe. By switching from the non-uniform E2M1 grid to a uniform E1M2 or INT4-style grid, the authors eliminate the geometric source of Shrinkage Bias. This change allows for the application of RHT across all three primary training matrix multiplications (forward, data-gradient, and weight-gradient) without the instability seen in E2M1-based recipes.

Why It Matters

Experimental results on models ranging from 1.5B to 124B parameters show that UFP4 consistently achieves lower loss degradation compared to strong E2M1 baselines. The findings suggest that the industry's current focus on E2M1 may be suboptimal. The authors argue that future hardware accelerators should support uniform 4-bit grids as first-class primitives, as they provide a more stable and accurate foundation for low-precision training at scale.

Key Terms and Definitions

Shrinkage Bias — A systematic negative rounding error inherent to non-uniform quantization grids where the geometric asymmetry of rounding bins causes values to consistently shift toward zero.
E2M1 — A 4-bit floating-point format with two exponent bits and one mantissa bit, currently the standard for many FP4 training implementations.
Random Hadamard Transform (RHT) — A mathematical technique that rotates tensors to disperse outlier energy across all coordinates, improving the utilization of quantization levels.
Uniform Grid — A quantization format (like E1M2 or INT4) where representable values are spaced at equal intervals, ensuring symmetric rounding bins and eliminating geometric bias.
GEMM — General Matrix Multiply, the fundamental computational operation in neural network training that is targeted for low-precision optimization in this study.

Abstract

FP4 training promises substantial reductions in memory and computation cost for LLM pretraining, yet current FP4 hardware paths and recipes, including NVIDIA Blackwell/Rubin-class systems and AMD MI350-series GPUs, remain centered on E2M1 data elements. In this study, we identify a fundamental limitation of that choice: non-uniform formats such as E2M1 inherently suffer from Shrinkage Bias, a systematic negative rounding error caused by the geometric asymmetry of their representable bins. We show that this bias accumulates multiplicatively across layers and is amplified by the Random Hadamard Transform (RHT), providing a unified explanation for the training instability observed in existing E2M1-based FP4 recipes. In contrast, uniform grids (E1M2/INT4) bypass this grid-geometry error and better convert the improved bucket utilization from RHT into higher quantization quality. Based on this finding, we propose UFP4, a uniform 4-bit training recipe that applies RHT to all three training GEMMs while restricting stochastic rounding to dY alone. On Dense 1.5B, MoE 7.9B, and MoE 124B long-run pretraining, UFP4 consistently achieves lower BF16-relative loss degradation than strong E2M1-based baselines, supported by scaling-law analysis and ablation studies. Our results suggest that future accelerators should support E1M2/INT4-style uniform 4-bit grids as first-class training primitives alongside E2M1.

Rethinking Shrinkage Bias in LLM FP4 Pretraining: Geometric Origin, Systemic Impact, and UFP4 Recipe: Research Paper Podcast