Zekai Liu, Zhilin Wang, Xuzheng He, Yu Cheng, Yang Yang
10 min
Text-to-music systems often struggle to meet specific user constraints, particularly when prompts are underspecified or imply complex musical traits like structure, rhythm, or mood progression. Current evaluation methods frequently rely on broad, opaque scores that fail to diagnose where a generation fails to meet user intent. This paper addresses this by formulating intent alignment as a per-request rubric of independently verifiable requirements. The authors introduce MuRA-Bench, a benchmark of 100 expert-revised requests that decouple explicit constraints from implied musical traits, providing a framework for diagnostic evaluation.
To improve alignment, the authors propose MIRA (Musical Intent Refinement Agent), an inference-time system that uses a verifier-guided tree search to iteratively update generation prompts. MIRA functions without modifying the parameters of the black-box backend generator, instead grounding intent in objective rubrics, observing requirement-level feedback, and retaining memory of successful and unsuccessful revision trajectories to optimize prompt selection within a fixed generation budget.
Experimental results across three different open-source backends demonstrate that MIRA consistently improves intent alignment compared to direct prompting. When operating under a budget of nine generations, MIRA achieved substantial gains in overall alignment scores (e.g., +12.9% for ACE-Step and +19.0% for SongGeneration2). Importantly, the framework allows open-source models to achieve intent-alignment scores comparable to, or in some cases better than, representative commercial systems like Suno and Mureka. The authors also show that the tree-structured feedback memory and adaptive search strategies utilized by MIRA make more effective use of the generation budget than simple best-of-N sampling.
As text-to-music models become more capable, the primary barrier to adoption shifts from raw audio quality to the ability of the system to correctly interpret and execute precise, often nuanced user intent. By providing a fine-grained, rubric-based evaluation benchmark and an agentic refinement framework that can be applied to any black-box model, this research establishes a scalable method for improving the steerability and reliability of generative music technology without the need for expensive model retraining.
Text-to-music systems produce increasingly convincing audio, yet evaluation reveals little about whether the result matches user intent. A global text-audio relevance score can overlook the implicit intent in underspecified prompts and mask failures in specific requirements, such as instrumentation, structure, rhythm, or mood progression. To bridge this gap, we formulate text-to-music intent alignment as satisfying a per-request rubric of independently verifiable items covering both a request's explicit requirements and its implied musical intent. Scoring items individually makes evaluation diagnostic by intent source and musical dimension, rather than a single opaque score. We instantiate this as MuRA-Bench, a benchmark of real-world platform requests curated by music experts. We further propose MIRA (Musical Intent Refinement Agent), a test-time agent that first grounds a request's intent into rubrics, then searches over prompt revisions for a black-box generator under a bounded budget, iteratively generating music, verifying it against the rubrics, and using this feedback to guide a trajectory-aware tree search. Experiments across open-source and commercial backends show that MIRA improves intent alignment, enabling an open-source generator to achieve performance comparable to representative commercial systems (e.g. Suno and Mureka). Project page: https://mirareview.github.io/.
Sam: A different expert might hear that scene differently. How do they constrain the interpretation?
Alex: Reviewers must remove personal preferences, unsupported additions, and overly specific guesses. Requirements must be compatible with the request and assessable by listening. Named artists or songs are translated into general musical traits, rather than direct imitation targets. The authors still acknowledge that musical conventions can influence these judgments.
Sam: How much material does the benchmark contain, and where did the requests come from?
Alex: MuRA-Bench contains one hundred controlled requests derived from anonymized intent seeds collected on Mureka. The authors exclude sensitive requests, unavailable context, and cases without stable musical criteria. A language model drafts the requirements, and music experts revise them.
Sam: Now take us from evaluation to improvement. Does MIRA get those expert checklists when it generates music?
Alex: No, and that separation is central. The expert checklists are reserved for evaluation. MIRA builds its own online rubric from the request and admitted tool evidence. That rubric stays fixed throughout the search, so the agent cannot keep redefining success around whatever it generated.
Sam: What kind of evidence can it bring in? The danger would be filling an underspecified request with unsupported assumptions.
Alex: Its tools retrieve musical knowledge, interpret reference audio, or connect a scenario with musical attributes. Added evidence must pass checks for supporting text, usability, and confidence. That constrains what enters the prompt, although it does not establish that every admitted interpretation is correct.
Sam: Once that checklist is fixed, how does the system decide what actually happened in the audio?
Alex: MusicFlamingo, a music-specialized model that answers questions about audio, checks each requirement separately. It scores the relative probability of a yes rather than a no answer. Those soft scores retain confidence instead of reducing every judgment to pass or fail.
Sam: A verifier can identify a missing flute, but there might be several ways to repair it. How does MIRA choose?
Alex: It proposes alternative complete prompts and generates a track for each. This forms a search tree: different revisions become different branches. It retains the strongest candidate overall, but also a candidate doing better on its weakest requirements. That keeps a different repair direction available.
Sam: So it avoids pursuing only the highest average. But another revision could restore the flute and damage the battle-ready mood.
Alex: That trade-off motivates its memory. Each branch records what improved, what regressed, and what remains unresolved. It compares a revision with its parent and with other candidates at that search layer. Later prompts receive that history, with instructions to preserve satisfied requirements.
Sam: What is the practical budget here? Is this an open-ended loop until the verifier likes something?
Alex: No. The broader setting allows nine generated candidates, including the initial track. MIRA returns the highest-scoring candidate it visited. But that budget counts generations, not retrieval calls, planner calls, verifier questions, or elapsed time.
Sam: Let’s separate improvement from simply buying more attempts. What did the main experiment find, and what controls for repeated sampling?
Alex: On the full benchmark, MIRA improved overall intent alignment across all the tested open-source generators. For ACE-Step, the score rose from about sixty-eight to seventy-six on the hundred-point reporting scale. That is average requirement satisfaction, not a percentage of songs judged completely successful.
Sam: That comparison gives MIRA extra generations. Did the advantage remain when both methods had the same generation allowance?
Alex: Yes, in a subset of twenty requests. They compared MIRA with independent repeated sampling, using the same verifier for final selection. On ACE-Step, MIRA gained about three-and-a-half score points over that equal-budget baseline. The other tested backend also improved.
Sam: That supports guided revision over just trying again, within that subset. Can they tell which parts of the method contribute?
Alex: Their component analysis adds tool grounding, then adaptive search, then structured memory. Grounding improves inferred-intent scores. Search adds further gains, and memory improves overall and dimension-balanced scores on both tested backends. That supports the combination, though it is still a limited subset.
Sam: The same verifier guides the search and supplies the headline evaluation. Could the agent mainly be learning to please that judge?
Alex: The authors address that concern in several ways. MusicFlamingo agrees with expert ratings better than the alternative evaluators they tested. Other audio evaluators also find alignment gains on the selected tracks. And blinded expert listening gives refined ACE-Step higher mean ratings than direct ACE-Step.
Sam: What about the headline comparison with commercial generators? Does the evidence establish that an open-source system is now better?
Alex: Only in a tightly scoped sense. Under the primary automatic evaluation, refined YuE achieves the highest overall score, narrowly above the strongest commercial baseline. Alternative evaluators still show refinement gains, but do not reproduce that commercial lead. In human listening, refined ACE-Step remains below Suno on mean ratings.
Sam: So improvement is more consistent than the ranking. And none of these scores tells us whether listeners prefer the music.
Alex: Correct. The studies assess fulfillment of musical intent, not general perceptual quality or preference. Each human target receives an assigned expert’s rating, rather than overlapping ratings from every expert. And the benchmark measures expert-revised interpretations, not a universal standard of musical value.
Sam: Given those boundaries, who should read the full document, and where should they start?
Alex: Researchers evaluating music generation should start with the benchmark construction and expert revision guidelines. Researchers building refinement agents should read the equal-budget comparison and component analysis, then the search and reproducibility appendix. Look closely at what the budget counts and how online requirements differ from evaluation requirements.
Sam: And what should everyone else remember when they encounter another impressive generation score?
Alex: This study offers credible evidence that requirement-level feedback can improve music generation without changing the generator. Its broader lesson is simple: sounding plausible is not the same as satisfying the request. Check the requirements that the overall score can hide.
Sam: That’s a useful distinction to carry into the next evaluation.