Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a preprint called "Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners," by Umberto Cappellazzo and colleagues. The central question is this: why is training AI on audio so much harder than training it on text or images?
Alex: That's a fair question. Language models just predict the next word. Why can't audio work the same way?
Sam: It can—and that's exactly the argument this paper makes. Text and image models have largely converged on one elegant idea: predict the next element in a sequence, and the model learns to understand structure on its own. Audio research, by contrast, has relied on elaborate multi-component training setups. This paper proposes a much simpler path.
Alex: What does that simpler path actually look like?
Sam: Think about reading a book with your hand covering the right half of every page. Your brain is constantly guessing what word comes next, and in doing so, it builds a deep understanding of language. The authors apply that same logic to sound. They call their method NAPE—Next Audio Patch Embedding prediction. The model listens to the beginning of a sound recording and tries to predict what comes next, one small chunk at a time.
Alex: And it learns just from that guessing game? No human labels, no teacher model telling it what's correct?
Sam: None. That's what makes it notable. Most competitive audio models need either human-annotated data or a separate pre-trained model to guide them. NAPE needs neither. It learns purely by predicting forward through the audio.
Alex: Walk me through the mechanics. How does a sound recording become something a model can actually process?
Sam: The first step is converting raw audio into a visual representation. Imagine plotting a sound wave so that time runs left to right, and pitch—from low to high—runs bottom to top. The brightness at each point shows how loud that frequency is at that moment. That picture of the sound is called a log-mel spectrogram, and it turns audio into something that looks like a photograph.
Alex: A picture of the sound. Then what?
Sam: That picture gets divided into small square tiles—patches. Each patch is converted into a compact list of numbers, a kind of fingerprint for that small slice of sound. Now you have a sequence of these fingerprints, and the model's job is to predict the next fingerprint from all the previous ones.
Alex: But a spectrogram has two dimensions—time and frequency. How do you turn a grid into a sequence?
Sam: That's one of the more interesting design choices in the paper. You have to pick a reading order—a path through the grid. The paper tests four options: raster order, which reads left to right and top to bottom like a page; diagonal sweeps; zigzag patterns; and time-major order, which reads all the frequencies at one moment before moving to the next.
Alex: Does the reading order actually matter?
Sam: It does, and the reason is intuitive. Audio is fundamentally about things changing over time. The raster, diagonal, and zigzag orders all move forward in time as the sequence progresses, so the model is always predicting something that comes later. Time-major order, by contrast, exhausts every frequency at a single instant before advancing—and that turns out to work noticeably worse. The model needs to be reading forward in time to learn useful audio structure.
Alex: What about what the model is actually trying to predict? You said it predicts a patch fingerprint—are there other options?
Sam: The paper tests three targets. One is the raw spectrogram values inside each patch. Another is the patch fingerprint itself. The third is the output of the model's own first processing layer—and that last option causes the training to collapse entirely. The target keeps shifting as the model learns, so the system can never stabilize. The first two work comparably, with the patch fingerprint performing slightly better on average.
Alex: And how does the model measure whether its prediction was close?
Sam: This is a subtle point. The obvious approach—measuring the straight-line distance between the predicted fingerprint and the real one—turns out to be unstable. The numbers can drift to extreme values without bound. Instead, the paper uses something called negative cosine similarity. Rather than measuring the distance between two points, it measures the angle between them. Two fingerprints pointing in the same direction in mathematical space are considered a good match, regardless of how large the numbers are. That stability is what makes the whole training process reliable.
Alex: So the direction of the prediction matters more than its magnitude.
Sam: Exactly. And that choice, combined with a mechanism that prevents the model from peeking at future patches, forms the complete training recipe. It's genuinely minimal.
Alex: When you test what the model has actually learned—without any fine-tuning—where do the most useful representations live?
Sam: The paper uses a method called linear probing. You freeze the entire trained model and attach only a simple classifier to its output. If the features are rich, even that simple classifier should work well. What they find is that the strongest features consistently come from the middle layers of the network, not the top.
Alex: Why the middle and not the top?
Sam: The top layers are highly specialized for the immediate prediction task—guessing the very next patch. That narrow focus makes them less useful for broader tasks like recognizing speech or environmental sounds. The middle layers retain more general information about the audio's overall structure. It's a bit like how a chess player's mid-game intuition is broadly useful, while their end-game calculations are highly specific to the position in front of them.
Alex: And when you look at what the model is actually attending to—what patterns emerge?
Sam: The attention maps show the model focusing strongly on the current moment in time and tracking how specific frequencies evolved earlier in the clip. What's notable is that the model organizes audio into acoustically coherent regions—grouping similar sounds together—without ever being told what those regions should be. It discovers structure purely through the act of predicting forward.
Alex: So the model builds a map of sound just by asking, what comes next?
Sam: That's the core claim. And the paper argues this establishes causal next-embedding prediction as a simple, scalable foundation for audio understanding—one that could eventually be extended to systems that process audio, text, and images together in a unified way.
Alex: A cleaner approach to a genuinely difficult problem. Thanks for walking us through it, Sam.
Sam: Thanks for listening to ResearchPod.