Nan Hu, Han Li, Jimeng Sun, Lu Wang, Fangkai Yang, Bo Qiao, Pu Zhao, David Dai, Mengyu Liu, Yuefeng Zhan, Jianjin Zhang, Weihao Han, Allen Sun, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Denvy Deng, Feng Sun, Qi Zhang
5 min
Modern advertising platforms often rely on a fragmented ecosystem of small, task-specific models to handle offline labeling, query generation, and user profiling. This approach is operationally expensive and fails to leverage the shared domain knowledge inherent in these tasks. AdNanny addresses this by consolidating these functions into a single, unified, reasoning-centric LLM based on the 671B-parameter DeepSeek-R1 architecture.
The authors developed a scalable training pipeline that transforms existing, label-only datasets into reasoning-augmented corpora. By prompting a powerful teacher model to generate step-by-step explanations for existing labels, they created a dataset where every decision is justified by natural language. The training process involves two main stages: multi-task supervised fine-tuning (SFT) with adaptive reweighting to balance heterogeneous workloads, and a reinforcement learning (RL) stage that aligns the model with downstream advertising metrics like retrieval and ranking quality.
To train such a massive hybrid dense-MoE model, the team built a custom Megatron-based trainer capable of handling complex expert, tensor, and pipeline parallelism. They implemented a bidirectional checkpoint conversion pipeline to ensure compatibility between the training environment and standard inference formats. To maintain production efficiency, the model is quantized to FP8, which significantly reduces memory and compute requirements while maintaining high accuracy, making it a cost-effective solution compared to general-purpose LLM prompts.
AdNanny demonstrates that large-scale, domain-specific applications can benefit from centralizing intelligence into a single foundation model. By providing both labels and reasoning, AdNanny acts as a high-quality offline assistant that can be consumed directly or used to distill knowledge into smaller, latency-optimized models. This approach reduces the maintenance burden of managing a zoo of small models and provides a scalable path for improving ad relevance, user profiling, and creative optimization.
Large Language Models (LLMs) have shown strong capabilities in Natural Language Understanding and Generation, but deploying them directly in online advertising systems is often impractical due to strict millisecond-level latency constraints. This has motivated the use of LLMs offline to improve retrieval, ranking, and recommendation models. Existing solutions typically fine-tune separate LLMs for individual tasks such as query-ad relevance labeling, keyword-based query generation, and user profiling. This results in redundant models, high maintenance cost, and limited performance gains despite substantial overlap in domain knowledge and reasoning patterns. We introduce AdNanny, a unified reasoning-centric LLM that serves as a shared backbone for offline advertising tasks. AdNanny is obtained by fine-tuning a public 671B-parameter DeepSeek-R1 checkpoint using a scalable training system that supports hybrid dense-MoE parallelism. We construct reasoning-augmented corpora that pair structured supervision with step-by-step natural language explanations. A multi-task supervised fine-tuning stage with adaptive reweighting enables AdNanny to handle diverse labeling and generation tasks in a consistent reasoning format. This is followed by reinforcement learning using downstream advertising metrics to align model behavior with online retrieval and ranking objectives. AdNanny is deployed in production within Bing Ads, where it significantly reduces manual labeling effort and improves accuracy across multiple offline tasks. By consolidating many task-specific models into a single reasoning-centric foundation model, AdNanny provides a scalable and cost-effective solution for large-scale advertising systems.
Sam: That's the interpretation the authors lean on, and it's plausible. The reasoning chain encodes not just the answer but the decision boundary — which features mattered, which edge cases to handle. A 7B model trained on that is getting substantially more information per example than one trained on a label alone.
Alex: So what are the load-bearing empirical claims?
Sam: Two main ones. First, AdNanny outperforms task-specific baselines on the offline ads tasks they evaluate — relevance, query generation, profiling. Second, distillation into 7B models produces systems that match or exceed those baselines, which is the result that makes the economics work. Reduced manual labeling effort is also reported, though that's harder to evaluate rigorously from the outside.
Alex: Where would a careful reviewer push back?
Sam: A few places. The evaluation is entirely internal — proprietary tasks on proprietary data, so external reproducibility is limited. The baselines aren't fully specified in terms of how well-tuned they were, which matters considerably for a "unified model beats specialists" claim. And the student-beats-teacher result, while mechanistically plausible, would benefit from ablations that isolate the reasoning chain's contribution from other factors in the distillation setup.
Alex: So the mechanism is credible, the direction of the result is clear, but the magnitude and generalizability are harder to assess from the outside.
Sam: That's a fair read. What the paper does establish is a coherent design pattern: use a large reasoning model to generate structured supervision, distill that into production-scale models, and maintain a single backbone rather than a fragmented pipeline. Whether those gains transfer to other industrial settings remains an open question.
Alex: It reads as a systems paper as much as a modeling paper — the contribution is the architecture of the whole pipeline, not any single component in isolation.
Sam: Exactly. And for teams managing the kind of model proliferation that large ad platforms accumulate, that systems-level argument may be the more durable takeaway. Thanks for listening to ResearchPod.