Optimizing CUDA like a Human: Micro-Profiling Tools as Expert Surrogates for LLM-Based GPU Kernel Optimization

Optimizing GPU kernels for high performance is a complex task that requires deep expertise in hardware architecture, memory hierarchies, and instruction scheduling. While recent LLM-based agents have attempted to automate this, they often struggle because they treat hardware telemetry as raw data for the LLM to interpret implicitly. This paper asks: can we improve kernel optimization by explicitly separating hardware diagnostic reasoning from code generation?

KernelPro introduces a closed-loop, multi-agent architecture that treats hardware analysis as a structured, rule-governed task. The system operates in two stages:

1. Benchmarking Agent: Performs a one-time roofline analysis to classify the kernel's bottleneck (e.g., compute-bound vs. memory-bound).
2. Profiling Agent: Uses a suite of 15 pluggable micro-profiling tools that translate raw hardware metrics (from ncu, nsys, and SASS) into semantic, natural language directives.

Unlike reactive agents that might selectively query data, KernelPro uses proactive tool orchestration, ensuring all relevant diagnostic tools are executed based on the initial bottleneck classification. The system also employs a domain-adapted Monte Carlo Tree Search (MCTS) to explore optimization paths, using progressive widening and search memory to improve performance over multiple iterations. Finally, it generates raw CUDA and CuTe code, allowing it to optimize at a lower, more granular level than systems that merely stitch together high-level library calls.

KernelPro establishes new state-of-the-art performance on the KernelBench benchmark, achieving significant speedups across all difficulty levels. In practical testing on expert-optimized MoE training kernels, it outperformed hand-tuned Triton code by 1.23x. Ablation studies confirmed that each component—specifically the micro-profiling tools and the MCTS search—contributes significantly to the system's success. Notably, KernelPro is the first agent to incorporate energy efficiency as a primary optimization objective, demonstrating an 11.6% reduction in energy consumption at matched performance levels.