Donghyun Lee, Jitesh Chavan, Duy Nguyen, Sam Huang, Liming Jiang, Priyadarshini Panda, Timo Mertens, Saurabh Shukla
6 min
Abstract
Diffusion transformers (DiTs) achieve state-of-the-art image and video generation, but their multi-step sampling and growing parameter count make inference expensive. Post-training quantization (PTQ) is the natural remedy, yet DiT activations shift across timesteps, prompts, and guidance branches, forcing prior methods to re-fit calibration data for every new checkpoint or modality. We present OrbitQuant, a data-agnostic weight-activation quantizer that bypasses range estimation by quantizing in a normalized, rotated basis. In this basis, a randomized permuted block-Hadamard (RPBH) rotation concentrates each coordinate around one fixed, known marginal regardless of the input, so a single Lloyd-Max codebook serves all timesteps, prompts, and layers of a given input dimension. We extend the same quantizer to weight rows offline, absorbing the rotation into the weights so that it cancels inside each linear layer and only a forward rotation on the activations remains at runtime. The same recipe transfers from image to video with no per-modality tuning. Across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX, it sets the state of the art for PTQ at several low-bit settings. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality.
Sam: Precisely. They pre-calculate a lookup table — called a Lloyd-Max codebook — that's perfectly tuned for that predictable distribution. Because the rotation guarantees the numbers always land in the same range, this one table works for any input, any prompt, any generation step. No re-calibration needed.
Alex: That's why they call it "data-agnostic." It doesn't need to see your data to work. What's the specific transformation they use?
Sam: It's called a Randomized Permuted Block-Hadamard rotation — RPBH for short. The name sounds intimidating, but the core idea is elegant: it shuffles and spreads the information across the model's activations so that no single value is unusually large or small. Think of it like shuffling a deck of cards before dealing — you go from a clumped, uneven distribution to something balanced and predictable. And crucially, they fold this rotation mathematically into the model's existing weights, so at runtime, the overhead is minimal. Just one lightweight operation on the activations.
Alex: So the model doesn't even slow down much from the rotation itself?
Sam: That's one of the practical strengths of the approach. The heavy lifting is done once, offline, when you set up the codebook. At runtime, the cost is low.
Alex: What happens when you really push this — compress the model as aggressively as possible?
Sam: This is where the results become particularly notable. There's a compression level the paper calls W2A4 — meaning the model's weights are stored at 2-bit precision and its activations at 4-bit. That's extremely aggressive. At that level, every other method the researchers tested collapsed into visual noise. OrbitQuant was the only approach that still produced usable images.
Alex: So it's not just marginally better — it's the difference between working and not working.
Sam: At that extreme, yes. And because the approach doesn't depend on any specific training data, the same technique transfers directly from image models to video models without any additional tuning. That generality is significant.
Alex: The paper also mentions something called AdaLN. Why couldn't they compress that part as aggressively?
Sam: Good question. AdaLN — Adaptive Layer Normalization — is a component that adjusts the model's behavior based on which step of the generation process it's currently on. Unlike most of the model's weights, which are fixed after training, these adjustments are dynamic. They change with every step. That means you can't fold a static rotation into them the way you can with the rest of the model.
Alex: So compressing those parts too hard would break the timing of the whole generation process?
Sam: Exactly. The researchers keep those components at a higher precision — INT4 rather than 2-bit — to preserve that dynamic behavior. It's a deliberate trade-off: apply aggressive compression where the math is stable, and hold back where it isn't.
Alex: Choosing your battles. Get the bulk of the model down to 2-bit, but protect the parts that actually need to move.
Sam: That's a precise way to put it. The insight is knowing which parts of the system can tolerate compression and which parts are load-bearing. By isolating the dynamic components, they avoid the failure modes that plagued earlier methods.
Alex: So the paper's contribution isn't just a new compression technique — it's a more principled way of thinking about where instability actually comes from in these models, and how to route around it.
Sam: That's the heart of it. By transforming the problem rather than fighting it directly, OrbitQuant offers a path toward running these powerful models on more modest hardware — and the approach is clean enough that it generalizes well beyond the specific models tested. It's a measured but meaningful step forward.
Alex: Thanks for listening to ResearchPod.