ResearchPod Summary
As generative recommenders replace traditional retrieval systems, they face two primary serving challenges: ensuring that generated ads meet advertiser-specific audience targeting rules (eligibility) and maintaining low latency for high-volume, real-time requests (compute). This paper introduces GRACE, a system designed to integrate personalized eligibility checks directly into the generative decoding loop while optimizing the Transformer architecture for the unique constraints of ad retrieval.
GRACE addresses the eligibility challenge through Generative Target Matching (GTM). Unlike standard constrained decoding, which only ensures that generated identifiers exist in the catalog, GTM uses bitmask and Bloom filter matchers to filter out tokens that lead to ad sequences ineligible for a specific user's targeting attributes. This happens at every step of the beam search, ensuring that only valid and eligible paths are explored.
To address the compute challenge, the authors redesigned the decoder specifically for the wide-beam, short-sequence regime typical of ad retrieval. By optimizing attention kernels, implementing a paged KV cache for beam search, and using dynamic per-step beam sizes, GRACE minimizes the overhead of the decoding loop. The system also employs a multi-stage pipelined inference architecture to hide the latency of lower-throughput stages by overlapping them with compute-intensive tasks.
GRACE significantly improves both the efficiency and the quality of ad retrieval. The GTM mechanism increases the ad-level target matching pass rate from 23.55% to 40.42% compared to standard constrained decoding. In terms of compute, the specialized attention kernels outperform state-of-the-art baselines (FlashAttention-2 and FlashAttention-3) by 68.0x for cross-attention and up to 25.8x for self-attention. Overall, these optimizations reduce decoder latency by 11.1x, enabling real-time performance on NVIDIA GH200 hardware.
This work bridges the gap between the high-quality potential of generative models and the strict operational requirements of real-time advertising platforms. By moving eligibility checks inside the decoder, GRACE avoids the inefficiency of generating ineligible candidates and provides a blueprint for deploying generative recommenders in production environments where latency and compute costs are critical constraints.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.