Scaling Dense Retrieval with LLM-Annotated Training Data: Structured Mining and Progressive Curriculum for E-Commerce Sponsored Search

How can e-commerce platforms generate high-quality training data for dense retrieval models at production scale without relying on biased click signals or expensive, manual human annotation? The authors address the challenges of position bias, tail-query sparsity, and the prohibitive cost of labeling millions of query-item pairs.

The authors propose an end-to-end pipeline that treats the disagreement between three heterogeneous production retrieval systems (lexical, BM25, and ANN) as a structured training signal. The pipeline consists of four stages:

1. Collection: Mining query-item pairs from three production channels.
2. Annotation: Using a calibrated cascade of three models (a 184M cross-encoder, a 2B LLM, and an 8B LLM) to assign 5-class graded relevance labels. The cascade uses per-class isotonic calibration to maintain high accuracy while reducing compute costs by 50%.
3. Extraction: Classifying pairs into five difficulty levels (Easy Positives, Hard Positives, Hard Negatives, Semi-Hard Negatives, and Random Negatives) based on channel agreement and rank metadata.
4. Curriculum Training: Training a two-tower BERT model using a three-stage progressive curriculum (BCE, MNR, and Triplet loss) that matches loss functions to the difficulty of the training samples.

Findings

The resulting model (Embedding Model V3) was deployed on Walmart’s sponsored search platform. Offline evaluation against 30K human-labeled queries showed a 5.1% increase in NDCG@10, with a 6.8% gain on tail queries. Furthermore, the rate of 'embarrassing' retrievals dropped from 8.7% to 3.5%. A two-week online A/B test confirmed these gains, showing a 2.8% increase in ad spend and a 2.9% increase in click conversion rates.