NOVA: A Verification-Aware Agent Harness for Architecture Evolution in Industrial Recommender Systems

Industrial recommender systems increasingly rely on complex, expressive architectures like Transformers and specialized interaction modules to drive business metrics. However, evolving these architectures is a manual, expert-intensive process. While AutoML can tune hyperparameters, it struggles with the structural, cross-module changes required for significant gains. Conversely, generic coding agents often produce code that is syntactically correct but semantically invalid for recommendation tasks, leading to silent failures that degrade performance without triggering obvious errors.

NOVA (a level-aware agent harness) addresses these challenges by treating architecture evolution as a structured, feedback-driven search. Its core innovation is the architecture gradient, a non-differentiable signal that aggregates previous modifications, verification outcomes, metric feedback, and trajectory memory to inform the next search direction.

To prevent the deployment of flawed models, NOVA employs a verification cascade. This multi-stage process checks structural semantics (e.g., tensor shape consistency, logit-fusion paths) and local executability before any expensive training occurs. Crucially, when a candidate fails, the system records the failure pattern as a 'forbidden direction,' effectively pruning the search space for future iterations.

NOVA organizes tasks into four complexity levels (L1–L4), with high-risk tasks routed to human experts (Copilot). In industrial deployment, NOVA significantly outperformed human-only expert loops and generic coding agents. On L3 'Literature-to-Production' tasks, it achieved a 60% effective pass rate and reduced the development cycle time by over 13x. Online A/B testing confirmed that architectures selected by NOVA improved Gross Merchandise Value (GMV) by up to 2.02% while simultaneously reducing prediction bias.