RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving

As frontier large language models (LLMs) become increasingly expensive, developers face a trade-off between high-quality, high-cost models and lower-cost, lower-quality alternatives. This paper addresses how to achieve the performance of frontier models like Claude Opus at a fraction of the cost, specifically within agentic coding environments where workloads interleave simple text generation with critical tool-selection tasks.

RLM-Cascade introduces a proxy-layer system that operates at the response level, bypassing the need for internal model access or shared vocabularies. It employs a three-path strategy:

1. SKIPPED Path: A lightweight, rule-based router identifies simple agentic turns and routes them exclusively to a cheaper draft model (DeepSeek-V4-Pro), avoiding the expensive verifier entirely.
2. Draft-and-Verify Path: For complex requests, the draft model generates a candidate response, which the verifier (Claude Opus) then accepts or enhances.
3. Direct Path: Tool-selection turns are routed directly to the verifier to ensure strict adherence to JSON schemas, which draft models often fail to maintain.

In production testing on Claude Code, RLM-Cascade achieved a 45.8% reduction in API costs compared to a direct Opus baseline. Counter-intuitively, the system also improved median end-to-end latency by 1.83x. This occurs because the majority of requests (64–70%) are handled by the faster, cheaper draft model, significantly reducing the overall load on the expensive verifier. Furthermore, the system maintained or exceeded the quality of the baseline, achieving a 100% pass rate on a 20-task benchmark compared to 95% for the native Opus model.

This work demonstrates that speculative decoding can be successfully applied at the API level, even across heterogeneous models and providers. By treating the entire response as the unit of speculation rather than individual tokens, RLM-Cascade provides a practical, cost-effective infrastructure for enterprise AI applications that cannot modify model internals but must optimize for both latency and budget.