CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

This paper investigates the trade-offs of linguistic compression in Large Language Models (LLMs). While "shortening" prompts or responses is often proposed to reduce inference costs, the authors argue that the impact depends heavily on which channel is compressed. They introduce CAVEWOMAN, a two-channel evaluation protocol that measures task accuracy, realized per-item cost, and reference-text agreement against an unconstrained baseline across eight models and five datasets.

The study reveals a stark asymmetry between input and output compression. Output compression—instructing the model to generate shorter responses—effectively reduces realized costs for most API and open-weight models. In contrast, input compression—pruning the user prompt—is a "lose-lose" strategy. It fails to lower net costs because models tend to compensate for the missing information by generating longer, more verbose responses, which are more expensive due to the higher price of output tokens compared to input tokens.

A critical finding is that output compression often leads to "surface-text divergence." For non-reasoning models, a significant portion of generations remain factually correct yet differ substantially from the model's unconstrained baseline output. This suggests that while models can maintain accuracy under strict output constraints, the internal reasoning or linguistic path taken to reach those answers changes significantly, even when the final answer remains correct.

For developers and researchers looking to optimize LLM inference, this work demonstrates that cost-saving strategies must be applied to the output channel rather than the input. Furthermore, it warns that relying solely on accuracy metrics can mask significant shifts in how a model arrives at its conclusions, which may have implications for consistency and reliability in production environments.