MULTI-TASK MULTI-FRAME VISUAL PIANO TRANSCRIPTION | Yonghyun Kim et al. | ResearchPod

Transcript: MULTI-TASK MULTI-FRAME VISUAL PIANO TRANSCRIPTION

Alex: Welcome to another episode of ResearchPod. Today we're looking at a study on Visual Piano Transcription—the process of converting video of a piano performance into digital music data.

Sam: So the paper is asking whether we can use cameras to "see" music instead of just listening to it? And the core problem is that microphones can't reliably tell when a note actually stops?

Alex: Exactly. When a pianist uses the sustain pedal, the sound lingers long after their finger leaves the key. Current audio-based systems get confused by this, often marking a note as still active until the sound finally fades on its own.

Sam: That makes sense. It's like trying to time a runner by listening for their footsteps instead of watching them cross the finish line. If the track is echoey, you'll never know exactly when they finished.

Alex: That's a precise analogy. The team behind this system—called V2N—realized that if you watch the keyboard directly, you can see the physical state of the key. Whether it's pressed down or back up, regardless of how long the sound is still ringing.

Sam: So they're sidestepping the audio entirely and focusing on the mechanics. But why is that hard to do with video? I'd imagine a camera has a pretty clear view of the keys.

Alex: It's harder than it looks. Earlier attempts only analyzed about one-fifth of a second of footage at a time. That's simply not enough context to reliably tell whether a key is being pressed down, held steady, or just starting to rise back up.

Sam: And that's why those systems struggled most with note endings—the moments when a key is released?

Alex: Precisely. V2N addresses this by looking at a full second of video at once. To do that processing efficiently, it uses a structure called a Conformer. Think of it as a processor with two modes of attention running in parallel—one that notices fine details in a single frame, and one that tracks how those details change across the whole sequence.

Sam: Like the difference between looking at one page of a flipbook versus flipping through the whole thing. One page tells you a position; the sequence tells you the motion.

Alex: That's exactly it. But the deeper innovation is in how the system is trained. Rather than teaching it to spot just one thing—say, when a note starts—they train it to predict four things simultaneously: when a note begins, when it ends, whether it's currently being held, and how hard the key was struck.

Sam: Why tackle all four at once? Wouldn't it be simpler to focus on one thing at a time?

Alex: You might think so, but the tasks actually reinforce each other. If the system has already learned that a key is in a sustained hold, it becomes much easier to correctly identify the precise moment that hold ends. The four signals together give the model a richer, more complete picture of what's physically happening, which makes each individual prediction more reliable.

MULTI-TASK MULTI-FRAME VISUAL PIANO TRANSCRIPTION

Yonghyun Kim, Hoyeol Sohn, Juhan Nam, Alexander Lerch

Aug 4, 2026·4:14·arxiv·Original Paper

Visual Piano Transcription (VPT)V2N (Video to Notes)Multi-task LearningAutomatic Music Transcription (AMT)Offset PredictionVisual Piano Transcription

Listen

4 min

Briefing

Research Question

Existing audio-based piano transcription systems often fail to distinguish between physical key release and the sustain pedal's effect on sound duration. While Visual Piano Transcription (VPT) can observe the keyboard directly, prior methods have been limited by short temporal windows (typically 0.2 seconds), poor offset accuracy, and a lack of note-level velocity prediction. This paper introduces V2N (Video to Notes) to address these limitations by treating piano transcription as a complete multi-task visual problem.

Approach

V2N processes 1-second video clips through a shared temporal backbone consisting of Conformer-style convolutional blocks. Unlike previous models that supervise only the center frame of a window, V2N applies per-frame supervision across four dedicated task-specific heads: onset, offset, key hold, and velocity. The system uses a bidirectional LSTM to aggregate temporal information, and it employs an offset-guided decoding strategy that prioritizes physical key-release peaks over simple frame-activity thresholds.

Main Findings

V2N achieves state-of-the-art performance on the PianoVAM and R3 datasets. By incorporating dedicated offset and velocity heads, the model significantly outperforms prior VPT systems in predicting physical key release and dynamic note intensity. Ablation studies demonstrate that the multi-frame loss and the inclusion of auxiliary tasks (like velocity and offset prediction) are essential for improving overall onset accuracy, confirming that these tasks provide complementary signals that strengthen the shared temporal backbone.

Why It Matters

This work demonstrates that visual cues are sufficient to recover full MIDI information—including velocity and precise physical release—without relying on audio. By decoupling transcription from the sustain pedal's acoustic artifacts, V2N provides a more accurate representation of the pianist's physical performance, which is particularly valuable for analyzing recordings where audio is noisy, ambiguous, or absent.

Key Terms and Definitions

VPT (Visual Piano Transcription) — The process of recovering symbolic MIDI note attributes (pitch, onset, offset, velocity) directly from video footage of a piano keyboard.
Physical Key Release — The moment a piano key returns to its rest position, which is distinct from the acoustic offset that may be extended by the sustain pedal.
Multi-Task Head — A neural network architecture component where a shared backbone feeds into multiple parallel output layers, each responsible for a specific prediction target (e.g., onset vs. velocity).
Conformer ConvModule — A neural network block that uses depth-wise 1D convolutions to capture local and global temporal dependencies, providing a large receptive field without the high computational cost of self-attention.
PianoVAM/R3 — Benchmark datasets consisting of synchronized video and MIDI recordings of piano performances used to evaluate transcription accuracy.

Abstract

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.

MULTI-TASK MULTI-FRAME VISUAL PIANO TRANSCRIPTION: Research Paper Podcast