Dmitrii Gavrilev
6 min
Abstract
Expressive performance rendering (EPR) aims to generate realistic performances constrained on sequences of notes. However, flow matching audio editing models manipulate only synchronized music samples of the same duration, limiting their understanding of expressive timing. We introduce PianoKontext, a flow matching rendering model for classical piano music that generates variable-length performances in the latent space of a pretrained Music2Latent model. We synthesize MIDI scores into deadpan audio and employ Dynamic Time Warping (DTW) in the latent space to construct paired data for training. The aligned embeddings are concatenated in DiT blocks, allowing for a simple and effective learning of the dependencies between the score and performances. Audio samples are available at our demo page: https://realfolkcode.github.io/pianokontext_demo/.
Alex: And this all happens in what the paper calls a "latent space." Why not just work with the raw audio directly?
Sam: Working with raw audio is computationally very heavy—it's like trying to edit a film by examining every single pixel of every single frame. Instead, they use a compressed mathematical representation of the audio that keeps the essential features—melody, harmony, rhythm—while discarding the computational clutter. Think of it as working from a detailed storyboard rather than the full-resolution film. That's what "latent space" means in practice.
Alex: So the model is learning to translate from a mechanical latent sequence to an expressive one, using the DTW map as its guide.
Sam: That's the mechanism. They also use an architecture called a Diffusion Transformer, which can look at the entire musical sequence at once rather than processing it note by note. That matters because the timing of a note early in a phrase has to make sense in the context of what comes later. The model needs that broader view to produce something that feels coherent.
Alex: The paper also mentions something called 2D Rotary Position Embeddings. What's that doing?
Sam: In any system like this, the model needs to track the order of things—knowing that the first note comes before the second, and also knowing whether it's looking at the mechanical input or the expressive output. These embeddings give the model a kind of coordinate system with two axes: one for time, one for which version it's examining. Without that, the model could easily lose track of which note belongs where.
Alex: How does it actually perform compared to earlier approaches?
Sam: The results show a meaningful improvement. The key comparison is against a method that tries to steer the model without any pre-computed alignment map. PianoKontext produced better audio fidelity and, more importantly, was far less likely to drop notes or invent ones that weren't there. It preserved the structure and harmony of the original score much more faithfully.
Alex: Are there limitations? It can't be perfect at everything.
Sam: The authors are candid about that. The model currently struggles with subtle articulation differences—for instance, the distinction between playing notes in a smooth, connected way, which musicians call legato, versus playing them with clear separation. Those nuances are genuinely difficult to capture. It's also currently limited to the piano, so the expressive qualities of a violin or a flute are outside its scope for now.
Alex: That makes sense. Articulation is one of the most personal parts of a human performance. Still, even within those limits, the practical implications seem significant—a tool that could take a composer's rough MIDI sketch and produce something that actually sounds played, rather than programmed.
Sam: That's the direction this points toward. A plugin where a composer inputs a basic, mechanical MIDI file and the software humanizes it—adapting to the tempo and style they choose—would meaningfully close the gap between a raw musical idea and a professional-sounding result.
Alex: So the real contribution here isn't just that it sounds better. It's that it solves the timing mismatch problem by treating expressive performance as a translation task in latent space.
Sam: That's it. By using DTW to build the temporal map and flow matching to generate the audio, they move past the constraints of fixed-duration models. It's a meaningful step toward AI tools that understand the nuance of human musicality—not just copying the notes, but grasping something of how and why a human performer would shape them.
Alex: A thoughtful piece of work. Thanks for walking us through it, Sam, and thanks to everyone for listening to ResearchPod.