ResearchPod Summary
Existing audio-based piano transcription systems often fail to distinguish between physical key release and the sustain pedal's effect on sound duration. While Visual Piano Transcription (VPT) can observe the keyboard directly, prior methods have been limited by short temporal windows (typically 0.2 seconds), poor offset accuracy, and a lack of note-level velocity prediction. This paper introduces V2N (Video to Notes) to address these limitations by treating piano transcription as a complete multi-task visual problem.
V2N processes 1-second video clips through a shared temporal backbone consisting of Conformer-style convolutional blocks. Unlike previous models that supervise only the center frame of a window, V2N applies per-frame supervision across four dedicated task-specific heads: onset, offset, key hold, and velocity. The system uses a bidirectional LSTM to aggregate temporal information, and it employs an offset-guided decoding strategy that prioritizes physical key-release peaks over simple frame-activity thresholds.
V2N achieves state-of-the-art performance on the PianoVAM and R3 datasets. By incorporating dedicated offset and velocity heads, the model significantly outperforms prior VPT systems in predicting physical key release and dynamic note intensity. Ablation studies demonstrate that the multi-frame loss and the inclusion of auxiliary tasks (like velocity and offset prediction) are essential for improving overall onset accuracy, confirming that these tasks provide complementary signals that strengthen the shared temporal backbone.
This work demonstrates that visual cues are sufficient to recover full MIDI information—including velocity and precise physical release—without relying on audio. By decoupling transcription from the sustain pedal's acoustic artifacts, V2N provides a more accurate representation of the pianist's physical performance, which is particularly valuable for analyzing recordings where audio is noisy, ambiguous, or absent.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study on Visual Piano Transcription—the process of converting video of a piano performance into digital music data.
Sam: So the paper is asking whether we can use cameras to "see" music instead of just listening to it? And the core problem is that microphones can't reliably tell when a note actually stops?
Alex: Exactly. When a pianist uses the sustain pedal, the sound lingers long after their finger leaves the key. Current audio-based systems get confused by this, often marking a note as still active until the sound finally fades on its own.
Sam: That makes sense. It's like trying to time a runner by listening for their footsteps instead of watching them cross the finish line. If the track is echoey, you'll never know exactly when they finished.
Alex: That's a precise analogy. The team behind this system—called V2N—realized that if you watch the keyboard directly, you can see the physical state of the key. Whether it's pressed down or back up, regardless of how long the sound is still ringing.
Sam: So they're sidestepping the audio entirely and focusing on the mechanics. But why is that hard to do with video? I'd imagine a camera has a pretty clear view of the keys.
Alex: It's harder than it looks. Earlier attempts only analyzed about one-fifth of a second of footage at a time. That's simply not enough context to reliably tell whether a key is being pressed down, held steady, or just starting to rise back up.
Sam: And that's why those systems struggled most with note endings—the moments when a key is released?
Alex: Precisely. V2N addresses this by looking at a full second of video at once. To do that processing efficiently, it uses a structure called a Conformer. Think of it as a processor with two modes of attention running in parallel—one that notices fine details in a single frame, and one that tracks how those details change across the whole sequence.
Sam: Like the difference between looking at one page of a flipbook versus flipping through the whole thing. One page tells you a position; the sequence tells you the motion.
Alex: That's exactly it. But the deeper innovation is in how the system is trained. Rather than teaching it to spot just one thing—say, when a note starts—they train it to predict four things simultaneously: when a note begins, when it ends, whether it's currently being held, and how hard the key was struck.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: Why tackle all four at once? Wouldn't it be simpler to focus on one thing at a time?
Alex: You might think so, but the tasks actually reinforce each other. If the system has already learned that a key is in a sustained hold, it becomes much easier to correctly identify the precise moment that hold ends. The four signals together give the model a richer, more complete picture of what's physically happening, which makes each individual prediction more reliable.
Sam: So it's like learning a sport by studying the whole game rather than drilling one position in isolation. Understanding the full flow makes each individual moment easier to read.
Alex: That's the logic. The researchers call it Multi-Task Temporal Modeling. By training the system to account for every frame in that one-second window, it learns to connect the physical movement of a finger and key to the underlying musical intent.
Sam: And estimating the force of the strike—the velocity—is new territory for a visual system?
Alex: It is. Force is subtle; you can't directly see it the way you can see a key moving. The system uses a dedicated output layer trained specifically on the moment of impact, and it's carefully designed to ignore frames where no strike is happening, so it doesn't learn from irrelevant data.
Sam: So in a sense, the system is doing what a conductor does—watching the pianist's hands rather than listening to the hall. It's a cleaner signal, as long as the camera has a clear view.
Alex: That's the practical upshot. The paper is candid that limitations remain—an obstructed camera angle, for instance, is a real problem. But as a step away from the fundamental constraints of audio-only transcription, it's a meaningful one.
Sam: It's a careful, incremental improvement, then. They're not claiming to have solved music transcription—just fixed a specific, longstanding error in how we track when a note actually ends.
Alex: That's a fair reading. And it's a useful reminder that sometimes the best way to solve a problem isn't to build a better ear—it's to build a better eye. Thanks for listening to ResearchPod.