What Else Needs Fixing? Exploring Cost-Effective Test-Time Compute for Revision Propagation in Artifacts Generated Through Conversation

Large Language Models (LLMs) are increasingly used to generate structured artifacts like JSON through iterative conversations. A critical challenge arises when a user requests a local change: the model must identify and update all dependent elements across the artifact to maintain consistency. This paper investigates how LLMs perform this revision propagation in settings where dependencies are often implicit and embedded within the conversation history, rather than explicitly defined in a codebase or knowledge graph.

The author introduces RevPropBench, a new human-annotated benchmark consisting of 150 samples across nine domains (e.g., travel planning, invoices, software configuration). The benchmark evaluates the ability of six LLMs (gpt-oss-20b/120b, gpt-5.4-mini, and qwen3.5-9b/27b/122b) to generate correct JSON patches (RFC 6902) in response to local revision requests. The study evaluates nine different revision methods, ranging from single-pass baselines to test-time compute strategies like sequential reflection and parallel sampling with various selection rules.

The study finds that providing both the conversation history and the final artifact as context significantly improves performance, confirming that conversation history contains vital dependency information. Among test-time compute methods, parallel sampling with selection is highly effective. Specifically, generating three parallel samples and selecting the best one using either LLM-based selection or medoid-based selection (which chooses the candidate with the smallest mean disagreement with others) provides the most consistent accuracy gains. These methods improve completion rates by 2.2–9.7% compared to single-inference baselines. While sequential reflection offers consistent gains, it is generally outperformed by parallel sampling methods in both accuracy and cost-effectiveness.