InfraRSI Team
7 min
Abstract
While Recursive Self-Improvement (RSI) accelerates isolated algorithm and kernel synthesis, distributed LLM training depends on tightly coupled launch settings, operator kernels, and communication schedules, where single edits often hit out-of-memory (OOM) boundaries, compiler conflicts, measurement jitter, or silent numerical drift. We present InfraRSI, the agentic harness that turns open-ended human commands into verified training optimizations via a five-stage workflow: init locks editable scope and a fixed task contract over workload, loss, and timing invariants to prevent reward hacking, doctor profiles baseline phases and calibrates cluster noise, run co-optimizes launch knobs and code patches before re-verifying top candidates against the baseline and logging findings to a cross-hardware experience bank, and report and apply audit source paths, track failed-trial costs, and commit promoted changes. Rather than stepping greedily from a single incumbent, InfraRSI initializes from diverse experience-seeded configurations and fits global execution-time and censored peak-memory models across all trials, including OOMs. Using these models, it prices per-setting memory costs, revises miscalibrated LLM priors from live runs, and pairs memory-hungry optimizations with memory savers and verified code patches. Across four single- and multi-node dense and Mixture-of-Experts (MoE) LLM workloads, InfraRSI achieves a speedup of up to $2.75\times$ and lifts mean Model FLOPs Utilization (MFU) from $17.4\%$ to $30.0\%$, outperforming existing tuners and coding agents while transferring experience across GPUs. As it improves training infrastructure efficiency, InfraRSI forms a two-level Recursive Self-Improvement loop---revising priors online within runs and distilling experience across clusters---to push foundation model scaling.
Alex: Give me the concrete case. What did the system find that a one-change-at-a-time search would struggle to reach?
Sam: Take their Gemma mixture-of-experts workload, a model that routes tokens through selected expert components. It used activation recomputation: recalculating intermediate results to save memory. Turning that off could save time, but the attempted configuration ran out of device memory.
Alex: That failure alone doesn’t tell you whether the idea is bad. It tells you the idea needs somewhere else to recover memory.
Sam: That’s how InfraRSI treats it. It keeps the promising failed configuration and looks for a memory-saving partner. In this case, disabling an existing fused-kernel package called Liger freed memory. Graph compilation, which compiles groups of operations for efficient execution, also needed compatible attention code.
Alex: So the winning change was not simply “turn off recomputation.” What did the completed package achieve?
Sam: The combination included compilation, disabling Liger, disabling recomputation, and a compatible attention patch. On that Gemma workload, it reached a speedup of two point seven five times the reference. That was the largest gain, not the typical result across the evaluation.
Alex: How does the search know which failed configurations deserve rescuing? Memory failures don’t give you a completed benchmark.
Sam: It maintains separate models of execution time and memory demand. A completed run gives a measured memory peak. An out-of-memory failure still supplies a lower bound on what the configuration needed. The system uses that incomplete evidence rather than discarding the attempt.
Alex: That also seems to put the language model’s confidence in its place. Are its initial performance guesses useful?
Sam: Not reliably in these experiments. The authors report negative rank correlations between initial speed predictions and measured results. So InfraRSI sometimes deliberately tests high-impact settings that the language model rates poorly. Measurements then update the predictions rather than letting an initial guess permanently veto a useful change.
Alex: That explains finding candidates. Let’s turn to accepting them, because a fast candidate can still be wrong.
Sam: Before searching, the system locks a task contract. Architecture, data, work per optimizer update, allowed precision, and update protocol must stay fixed. It also freezes files such as loss definitions and timing code. Candidates must satisfy checks covering execution, workload, update schedules, numerical behavior, and timing.
Alex: Fixed files help, but editable code can still hide a shortcut. How do they address behavior that a receipt fails to reveal?
Sam: They also audit the source paths before promotion. That review caught deferred updates and unauthorized precision changes. So the evidence supports a combination of automated checks and code auditing. It does not show that a valid-looking execution receipt alone catches every shortcut.
Alex: Cluster noise is another way to manufacture progress without intending to. How do they separate a better configuration from a lucky run?
Sam: They measure baseline timing variation on the actual cluster and trim warmup steps. At the end, top candidates face fresh, back-to-back runs against the original configuration on the same processor allocation. Promotion requires a gain beyond calibrated noise. If nothing qualifies, the reference configuration stays.
Alex: Now the main comparison. How much better was the whole system than the alternatives?
Sam: Across the evaluated workloads, InfraRSI achieved a geometric-mean speedup of one point eight three times the reference. The strongest comparison method, the RE-Bench agent scaffold, reached one point three seven times. Auto-Research and the knob-only tuner also finished below InfraRSI under identical task contracts.
Alex: Those are gains after searching. Does the accounting hide the failed attempts needed to discover them?
Sam: The authors retain failed and rejected attempts in computing-time and language-model usage accounts. The verification audit also matters: it rechecked about twelve hundred stored executions. Roughly a third were rejected, including some that required numerical checks to catch. That’s evidence that filtering changes which results count.
Alex: What should make us hesitate before assuming this will improve every training setup?
Sam: The evaluation covers selected dense and expert-routed models, on single-node and multi-node Hopper-generation hardware. Cross-hardware experience transfer helped in a separate comparison, but that test used one target workload. I’d treat the breadth as promising, not evidence of universal transfer.
Alex: And keeping loss close during a benchmark is not the same as preserving an entire training trajectory.
Sam: That’s the most consequential boundary. The numerical checks test short-horizon loss consistency under declared tolerances. They do not establish equivalent final model quality after a complete training run. Reproducing a speedup and preserving long-run learning behavior remain different claims.
Alex: Who should read the full document, and where should they begin?
Sam: Researchers building training systems or coding agents should start with the end-to-end evaluation and its workload comparisons. Then read Semantic, Numerical, and Source-Path Verification, followed by Budgeted Compositional Search. For everyone else, remember this: a speedup only counts if the work stays the same.
Alex: Thanks for listening.