Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes. However, flow matching audio editing models manipulate only synchronized music samples of the same duration, limiting their understanding of expressive timing. We introduce PianoKontext, a flow matching rendering model for classical piano music that generates variable-length performances in the latent space of a pretrained Music2Latent model. We synthesize MIDI scores into deadpan audio and employ Dynamic Time Warping (DTW) in the latent space to construct paired data for training. The aligned embeddings are concatenated in DiT blocks, allowing for a simple and effective learning of the dependencies between the score and performances. Audio samples are available at our demo page: https://realfolkcode.github.io/pianokontext_demo/.
Alex: Welcome to another episode of ResearchPod. Today, we're looking into a challenge that anyone who's ever listened to computer-generated music might recognize: why does it often sound so stiff and robotic?
Sam: It's a classic problem in music technology. We're trying to take a rigid, mechanical sequence of notes—like a MIDI file that a computer plays without any feeling—and transform it into something that sounds like a human pianist, complete with natural timing and dynamics. Researchers call this "expressive performance rendering."
Alex: So this paper, PianoKontext, is asking how we can teach an AI to add that human touch to a robotic score—without needing to manually label every single note?
Sam: Exactly. And the core difficulty is subtle. Most AI models for music are designed to work on pairs of audio that are perfectly synchronized in time. If you want to change the style of a piece, the model expects the input and the output to be the same length. But human expression is fundamentally about stretching and squeezing time—playing some notes slightly early, holding others a little longer.
Alex: Right, because if the model forces the output to match the input's duration exactly, it can't actually learn the expressive timing that makes music sound alive.
Sam: That's the hurdle. To get around it, the researchers treat the whole process as a translation task—not between languages, but between two different mathematical representations of music. Think of it like a translator who doesn't just swap words, but understands the rhythm and flow of a language. To make that translation possible, they use a technique called Dynamic Time Warping, or DTW.
Alex: I've heard of "warping" in video editing. How does it apply here?
Sam: Imagine you have two recordings of the same piece: one is a slow, methodical practice session, and the other is a fast, expressive concert performance. They play the same notes, but the timing is completely different. DTW is an algorithm that creates a map between these two recordings, lining up the notes so the computer knows exactly which part of the slow version corresponds to which part of the fast one. It tells the model: this specific note in the rigid score belongs to this specific moment in the human performance.
Alex: So it's essentially a temporal bridge between the two versions.
Sam: Precisely. And by computing that map in advance, the model can learn how to bridge the gap between a mechanical input and a human-like output, even when the durations don't match at all.
Alex: Once it has that map, how does it actually generate the expressive version?
Sam: It uses a framework called flow matching. Picture a sculptor slowly carving a statue from a block of stone. The model starts with something like random noise and gradually shapes it into a structured, musical signal. It learns a kind of directional guide—a mathematical description of the path that noise should take to become a specific, expressive piece of music. The DTW map acts as the blueprint the sculptor follows.
Alex: And this all happens in what the paper calls a "latent space." Why not just work with the raw audio directly?
Sam: Working with raw audio is computationally very heavy—it's like trying to edit a film by examining every single pixel of every single frame. Instead, they use a compressed mathematical representation of the audio that keeps the essential features—melody, harmony, rhythm—while discarding the computational clutter. Think of it as working from a detailed storyboard rather than the full-resolution film. That's what "latent space" means in practice.
Alex: So the model is learning to translate from a mechanical latent sequence to an expressive one, using the DTW map as its guide.
Sam: That's the mechanism. They also use an architecture called a Diffusion Transformer, which can look at the entire musical sequence at once rather than processing it note by note. That matters because the timing of a note early in a phrase has to make sense in the context of what comes later. The model needs that broader view to produce something that feels coherent.
Alex: The paper also mentions something called 2D Rotary Position Embeddings. What's that doing?
Sam: In any system like this, the model needs to track the order of things—knowing that the first note comes before the second, and also knowing whether it's looking at the mechanical input or the expressive output. These embeddings give the model a kind of coordinate system with two axes: one for time, one for which version it's examining. Without that, the model could easily lose track of which note belongs where.
Alex: How does it actually perform compared to earlier approaches?
Sam: The results show a meaningful improvement. The key comparison is against a method that tries to steer the model without any pre-computed alignment map. PianoKontext produced better audio fidelity and, more importantly, was far less likely to drop notes or invent ones that weren't there. It preserved the structure and harmony of the original score much more faithfully.
Alex: Are there limitations? It can't be perfect at everything.
Sam: The authors are candid about that. The model currently struggles with subtle articulation differences—for instance, the distinction between playing notes in a smooth, connected way, which musicians call legato, versus playing them with clear separation. Those nuances are genuinely difficult to capture. It's also currently limited to the piano, so the expressive qualities of a violin or a flute are outside its scope for now.
Alex: That makes sense. Articulation is one of the most personal parts of a human performance. Still, even within those limits, the practical implications seem significant—a tool that could take a composer's rough MIDI sketch and produce something that actually sounds played, rather than programmed.
Sam: That's the direction this points toward. A plugin where a composer inputs a basic, mechanical MIDI file and the software humanizes it—adapting to the tempo and style they choose—would meaningfully close the gap between a raw musical idea and a professional-sounding result.
Alex: So the real contribution here isn't just that it sounds better. It's that it solves the timing mismatch problem by treating expressive performance as a translation task in latent space.
Sam: That's it. By using DTW to build the temporal map and flow matching to generate the audio, they move past the constraints of fixed-duration models. It's a meaningful step toward AI tools that understand the nuance of human musicality—not just copying the notes, but grasping something of how and why a human performer would shape them.
Alex: A thoughtful piece of work. Thanks for walking us through it, Sam, and thanks to everyone for listening to ResearchPod.