LLMET: Enabling Cross-Layer Evaluation of Emerging M3D Memories for Energy-Efficient LLM Serving

As Large Language Models (LLMs) grow in size and context length, the energy cost of moving data between the processor and off-chip High Bandwidth Memory (HBM) has become a primary bottleneck. While traditional SRAM caches are limited by area and thermal constraints, emerging Monolithic 3D (M3D) integration allows for the stacking of dense memory directly onto logic chips. This paper introduces LLMET, a cross-layer simulation framework designed to evaluate how these ultra-large on-chip memories impact the energy efficiency of LLM serving across various platforms and workloads.

LLMET bridges the gap between device-level memory technology and system-level LLM performance. Unlike existing simulators that treat memory as a fixed resource, LLMET incorporates:

• Technology-specific PPA modeling: It uses circuit-level data for M3D 2T gain-cell (2T-GC) memories to accurately estimate energy and area.
• Capacity-aware mapping: The framework dynamically selects operator-fusion strategies based on available cache size, capturing how off-chip traffic scales as memory capacity increases.
• Component-level energy accounting: It provides a detailed breakdown of energy consumption across functional units, including DRAM, L2 cache, and compute cores.

The study demonstrates that M3D-enabled ultra-large caches provide substantial energy benefits. For a Llama 3.1-70B model on a dual A100 GPU setup, expanding the L2 cache from 40MB to 1GB reduces chip energy by 44% during the prefill phase. Similarly, on an 8x B200-like platform, increasing the cache to 4GB saves up to 24% of prefill energy. These results confirm that M3D integration is a viable path for mitigating the memory wall in AI accelerators, though the magnitude of these savings is highly dependent on the specific workload's sequence length and batch size.