Jongmin Kim, Hyesung Ji, Wonseok Choi, Hyunah Yu, Jung Ho Ahn
9 min
Fully homomorphic encryption (FHE), particularly the CKKS scheme used for real-number arithmetic, is limited by massive computational and memory overheads, especially during the bootstrapping process required for unlimited operations. Most existing accelerators for FHE suffer from memory-bound bottlenecks, as they must maintain large working sets for CtS transformations, which are critical for bootstrapping. The WHET (Welding Homomorphic Encryption to Accelerator Architectures) framework proposes a co-design of cryptographic algorithms and hardware to overcome these limitations.
WHET introduces three core algorithmic optimizations: fine-grained CtS (fg-CtS), which replaces high-memory baby-step giant-step (BSGS) algorithms with a decomposed structure; plaintext compression, which exploits repetitive patterns in CtS data; and intermediate ModRaise, which reduces level consumption during the first CtS stage. These techniques allow for a significantly smaller on-chip memory footprint while maintaining the same 128-bit security level.
To capitalize on these algorithmic gains, WHET introduces two specific architectural modifications to a baseline FHE accelerator:
These enhancements allow WHET to reduce the total chip area by optimizing the memory hierarchy—shrinking the primary scratchpad to 128 MiB—without sacrificing performance. The design balances compute throughput and memory bandwidth, resulting in much higher hardware utilization compared to traditional designs.
WHET achieves significant performance improvements over state-of-the-art accelerators, delivering 1.38–8.74x higher performance per area. A notable achievement is the first-ever sub-millisecond CKKS bootstrapping (for 2^15 complex numbers). The framework also enables practical encrypted inference for deep learning models like ResNet-20 and VGG-16, with latencies reduced to the millisecond range compared to the thousands-of-seconds overhead observed in CPU-based implementations.
Sam: A standard algorithm called baby-step giant-step reduces the number of expensive operations in that transformation. It does this by producing intermediate encrypted results and reusing them. Those savings are real, but several large intermediates must remain available together.
Alex: Which sounds sensible until retaining the intermediates crowds out everything else. Then avoiding arithmetic creates more waiting.
Sam: In the authors’ baseline, that’s the central problem. Its main transform engines are busy only about a quarter of the time during bootstrapping. Meanwhile, the off-chip memory interface is heavily occupied, leaving little room to hide transfers.
Alex: What does WHET change besides giving those engines more memory?
Sam: It breaks the coefficient-to-slot transformation into smaller stages and drops baby-step giant-step. Each stage has little enough work that the simpler method becomes affordable. Fewer temporary encrypted results need to remain on the chip together.
Alex: But dropping an arithmetic-saving algorithm should increase the operation count. Where does the gain come from?
Sam: Here’s the trade. Instead of keeping several large encrypted intermediates ready for reuse, the simpler method keeps far fewer. The freed memory can hold data fetched ahead for the next stage. More arithmetic can finish sooner because the hardware spends less time waiting.
Alex: That’s the central reversal: optimize the live data, not just the operation count. Does splitting the transformation more finely have another cost?
Sam: It consumes more levels, leaving less computation available after each refresh. The authors don’t ignore that trade-off. They add a modification called intermediate modulus raising, which changes where the first transformation stage runs.
Alex: What does changing the modulus buy them in practical terms?
Sam: The modulus governs the numerical range used to represent the encrypted data. WHET runs the first stage at a smaller, intermediate modulus rather than the full one. That saves a level and shrinks that stage’s unencrypted inputs, while requiring cryptographic adjustments that the authors analyze for security.
Alex: You mentioned unencrypted inputs. Are those the private values being exposed to save space?
Sam: No. They’re encoded constants used in the transformation, not the client’s encrypted data. The authors find repeated patterns in many of these constants. Storing the repeated values compactly reduces the amount fetched from off-chip memory.
Alex: So there are complementary savings: fewer live encrypted intermediates, smaller constants, and less repeated constant data. What happens once those transfers stop dominating?
Sam: A different bottleneck appears inside the chip. Frequently updated intermediate results compete for access to the main on-chip memory. WHET gives those results a dedicated buffer and adds compound instructions that share inputs across arithmetic operations.
Alex: That makes the hardware changes a response to the algorithm changes, rather than an independent list of improvements.
Sam: That’s how the authors develop the design. They also shrink the main scratchpad, the chip’s explicitly managed storage. The dedicated buffer doesn’t simply sit on top of an unchanged memory system; the resulting chip is slightly smaller than their baseline.
Alex: Let’s anchor the speed claim. What is the strongest comparison, and what actually ran?
Sam: The controlled comparison uses an enhanced accelerator baseline called SHARP-eight-plus. Across four workloads, WHET achieves about twice the speed on average. Those workloads cover bootstrapping, logistic-regression training, sorting, and encrypted image classification.
Alex: And the sub-millisecond result applies to refreshing, not an entire machine-learning application?
Sam: Correct. Full-slot bootstrapping takes about nine-tenths of a millisecond, versus two and a half milliseconds on the controlled baseline. But the refreshed output has fewer remaining levels. That means the headline times don’t buy identical amounts of subsequent computation.
Alex: If refreshing happens sooner again, some of the gain could disappear. Do they account for that?
Sam: They report a bootstrapping comparison normalized by the remaining levels, which still improves. They also evaluate complete workloads and add the optimizations incrementally. For me, those tests make the argument stronger than the isolated latency record.
Alex: How much of this is measured hardware performance?
Sam: The execution results come from a simulator and scheduler, not a fabricated WHET chip. The authors synthesize core logic and model memory, area, and energy. That supports an architectural comparison, but it isn’t a demonstration of a running device.
Alex: What about the broader claim that encrypted neural networks become practical? Does the smaller memory work across all their models?
Sam: Not uniformly. In the broader neural-network evaluation, several models get faster, but VGG-sixteen initially gets slower with the smaller scratchpad. Its large linear transformations still require many stored intermediates. Expanding the scratchpad restores a speed advantage.
Alex: So WHET removes a major working-set problem, not the need to think about memory capacity.
Sam: And larger image inputs present another limit. The evaluated software can materialize more transformation constants than the discussed off-chip memory systems can hold. The authors point toward alternative implementations that need fewer constants.
Alex: Who should read the full document, and where should they begin?
Sam: Accelerator architects and encryption-algorithm researchers should start with “The Working Set Problem in CtS.” Then read the algorithmic enhancements and the sensitivity study, which separates their contributions. Private-machine-learning researchers should turn to “A wider range of CNN workloads,” especially the VGG memory exception.
Alex: And what should everyone else take away, without carrying all those implementation details?
Sam: This is a simulation-backed case for designing encryption algorithms and accelerators together. Keep the hardware caveat and remaining-computation trade-off in view. The line to remember is: doing less arithmetic isn’t always faster; sometimes the winning algorithm moves less data.
Alex: Thanks for listening.