Ming Chen, Rong-Xi Tan, Ke Xue, Yu-Jie Zhou, Taiye Lu, Zhi-Xuan Gao, Peng Xie, Zijun Shen, Chen Lu, Haopu Shang, Chao Qian
9 min
Abstract
Black-box optimization (BBO) arises in many scientific and engineering problems where objective evaluations are expensive and limited. Recent large language model (LLM) agents offer a new way to approach BBO by combining task semantics, computation, optimization tools, and feedback-driven decision making, showing great potential due to the integration with mathematically rigorous tools. However, existing agentic BBO studies use different task domains and system configurations, making their results difficult to compare and the effects of individual design choices hard to isolate. We therefore introduce AgenticBBO-Bench, a cross-domain benchmark for agentic BBO spanning synthetic functions, hyperparameter optimization, database tuning, chip design, and molecular design under a unified finite-budget evaluation protocol. In our experiments, agentic BBO achieves higher family-averaged scores than direct LLM-based methods in all five domains and outperforms the best numerical optimizers in four. We further study three factors shaping agent performance: optimization tools, task information and prior knowledge, and the role of the LLM during search. Our results show that additional numerical tools do not consistently improve performance, task semantics are broadly useful while more specific priors are less reliable, and numerical optimizers can effectively absorb gains from search trajectories established by the agent. Finally, we introduce a five-task frontier challenge within AgenticBBO-Bench and evaluate seven LLMs under the Codex agent harness, where GPT-6 Astra and DeepSeek-V4.1-Flash lie on the Pareto frontier of performance and cost among the evaluated models. Our code is available at https://github.com/lamda-bbo/agentic-bbo.
Sam: Those sound difficult to compare fairly. What do they hold constant?
Alex: Within each matched task and run, methods get the same initial observations and the same allowance of new objective evaluations. Agents work in an isolated workspace. They can inspect visible history and run code, but the evaluator and hidden task state remain outside it.
Sam: Does every attempt consume that allowance, including malformed candidates or calls to other tools?
Alex: Only accepted candidates evaluated by the benchmark consume it. Invalid submissions can be revised without spending an evaluation. Local code, model reasoning, and optimizer queries also don't count. That distinction will matter when we discuss cost.
Sam: Before the results, make the agent's freedom concrete. What did it do that a fixed proposal rule wouldn't necessarily do?
Alex: In the Bigblue1 chip-layout task, it examined previously evaluated layouts and fitted a simple linear predictor. That predictor related distances between pairs of blocks to wire cost. The agent then used the inferred pattern to construct a placement and refined its compactness.
Sam: It was learning a task-specific approximation from the observed results, rather than just choosing among supplied optimizer suggestions?
Alex: Yes, in that example. The best observed quality improved as it refined the layout. The authors use this case to suggest why custom code can help: the agent can build an analysis suited to the task, rather than relying only on a supplied optimizer's assumptions.
Sam: Now the main comparison. Was the agent better than simply prompting the same model for candidates?
Alex: Across the domain averages, the agentic configuration beat direct generation, with an average relative score improvement of about thirty percent. Both used DeepSeek V4.1 Flash and Codex, the software managing the agent's state and interactions. The difference was the available workspace and interfaces.
Sam: Beating direct generation is useful, but numerical methods are the harder reference. How did those compare?
Alex: The agent achieved the highest family-average score in four of the five domains. Database tuning was the exception, where TuRBO, a numerical optimizer, did best. So this study supports competitiveness across diverse tasks, not a clean sweep over established methods.
Sam: And when you say “score,” is that just the quality of the final answer?
Alex: Mostly, it rewards getting good answers early. Seventy percent comes from normalized best-so-far quality throughout the search; the rest comes from the final best result. The normalization uses shared references, not whichever agent happened to win.
Sam: That fits limited-budget research: an optimizer that improves quickly deserves credit. Where does the more-tools result enter?
Alex: The controlled tool study added Gaussian-process tools, which predict objective values and uncertainty from observed trials. One version offered candidate suggestions. Another added predictions, diagnostics, and search-policy controls. Neither augmentation improved the mean over the base agent, though individual tasks sometimes benefited.
Sam: Could that mean the agent simply ignored the new tools?
Alex: It used them, but custom Python remained common. The result isn't that numerical tools are useless. It's that exposing this particular backend didn't reliably help these agents on this task subset. The authors argue for task-adaptive tool selection rather than just adding interfaces.
Sam: What about information rather than tools? A database parameter's name seems more useful than an anonymous coordinate.
Alex: Restoring task and parameter meanings improved the mean on the real-domain diagnostic tasks. Adding more detailed domain knowledge was less reliable: it helped some tasks and hurt others. A separate controlled study found the strongest gains when information about relevant variables and search geometry was correct.
Sam: Can they cleanly attribute the first gain to meaning alone?
Alex: Not completely. The anonymous condition also changed the candidate representation, hiding original types, ranges, and transformations. The document explicitly acknowledges that. The comparison between semantics and added domain knowledge is cleaner because representation stays fixed.
Sam: Let's return to the cost question. Does the model need to stay involved throughout the search?
Alex: Not always in their tests. After an agent warm-up, handing the same search history to TuRBO essentially matched continued agentic optimization. It also beat TuRBO starting from the shared initial observations. That suggests the agent can contribute a useful opening phase without controlling every later decision.
Sam: How much confidence should we put in that pattern beyond these experiments?
Alex: I'd treat it as a promising design clue, not a universal recipe. The broad benchmark used two runs per task. The diagnostic studies used small subsets and excluded molecules. Most experiments also fixed the default model and harness, so they don't establish that every agent behaves this way.
Sam: And equal evaluation allowances still don't tell us which system is cheaper overall.
Alex: They don't. Extra reasoning and code execution sit outside that allowance. The paper separately compares frontier models' scores with official token costs, finding similar performance at substantially different prices. But that compact challenge uses one run per task and model, so I'd read it as a snapshot.
Sam: There's also a question about what the application tasks actually represent.
Alex: Database tuning uses learned predictors rather than live database workloads. Molecular design uses specified molecular scores, not drug efficacy, toxicity, or synthesizability. These are useful optimization tests, but success on them shouldn't be mistaken for success on every downstream application goal.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building optimization agents should start with Common Optimization Protocol, then the studies of tools and the model's role. Read the ablation settings for the exact interventions. Application researchers should first check their domain's evaluator and constraints. For everyone else: better optimization is not simply more tools; it's choosing useful analysis and knowing when to hand over.
Sam: Thanks for listening.