Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic
6 min
Abstract
Self-supervised learning (SSL) has driven substantial progress in audio representation learning, though existing methods have increasingly relied on elaborate pre-training recipes to reach competitive performance. A markedly different pre-training philosophy underpins the most influential progress in language modeling and, more recently, in visual representation learning: rather than train encoders as static feature extractors, models are trained to predict the next element, a discrete token or a continuous embedding, from the preceding context. Autoregressive prediction thereby provides a unified pre-training interface that transfers across modalities, compelling the model to learn the underlying data distribution. We ask whether such a simple causal paradigm can yield strong audio learners, given that audio's temporal structure makes autoregressive prediction of patch embeddings a natural fit. We introduce NAPE (Next-Audio-Patch-Embedding prediction), a self-supervised framework in which a causal Transformer predicts each next patch embedding of a log-mel spectrogram from the previous ones, using causal masking and stop-gradient as its sole training signal. The design is intentionally minimalist, avoiding reconstruction decoders, acoustic tokenizers, student-teacher setups, and auxiliary regularization losses. Across six audio and speech benchmarks, NAPE achieves state-of-the-art fine-tuning performance on several tasks, scales consistently across encoder sizes, and yields strong linear-probing results. NAPE also produces structured attention patterns without explicit supervision.
Sam: That's one of the more interesting design choices in the paper. You have to pick a reading order—a path through the grid. The paper tests four options: raster order, which reads left to right and top to bottom like a page; diagonal sweeps; zigzag patterns; and time-major order, which reads all the frequencies at one moment before moving to the next.
Alex: Does the reading order actually matter?
Sam: It does, and the reason is intuitive. Audio is fundamentally about things changing over time. The raster, diagonal, and zigzag orders all move forward in time as the sequence progresses, so the model is always predicting something that comes later. Time-major order, by contrast, exhausts every frequency at a single instant before advancing—and that turns out to work noticeably worse. The model needs to be reading forward in time to learn useful audio structure.
Alex: What about what the model is actually trying to predict? You said it predicts a patch fingerprint—are there other options?
Sam: The paper tests three targets. One is the raw spectrogram values inside each patch. Another is the patch fingerprint itself. The third is the output of the model's own first processing layer—and that last option causes the training to collapse entirely. The target keeps shifting as the model learns, so the system can never stabilize. The first two work comparably, with the patch fingerprint performing slightly better on average.
Alex: And how does the model measure whether its prediction was close?
Sam: This is a subtle point. The obvious approach—measuring the straight-line distance between the predicted fingerprint and the real one—turns out to be unstable. The numbers can drift to extreme values without bound. Instead, the paper uses something called negative cosine similarity. Rather than measuring the distance between two points, it measures the angle between them. Two fingerprints pointing in the same direction in mathematical space are considered a good match, regardless of how large the numbers are. That stability is what makes the whole training process reliable.
Alex: So the direction of the prediction matters more than its magnitude.
Sam: Exactly. And that choice, combined with a mechanism that prevents the model from peeking at future patches, forms the complete training recipe. It's genuinely minimal.
Alex: When you test what the model has actually learned—without any fine-tuning—where do the most useful representations live?
Sam: The paper uses a method called linear probing. You freeze the entire trained model and attach only a simple classifier to its output. If the features are rich, even that simple classifier should work well. What they find is that the strongest features consistently come from the middle layers of the network, not the top.
Alex: Why the middle and not the top?
Sam: The top layers are highly specialized for the immediate prediction task—guessing the very next patch. That narrow focus makes them less useful for broader tasks like recognizing speech or environmental sounds. The middle layers retain more general information about the audio's overall structure. It's a bit like how a chess player's mid-game intuition is broadly useful, while their end-game calculations are highly specific to the position in front of them.
Alex: And when you look at what the model is actually attending to—what patterns emerge?
Sam: The attention maps show the model focusing strongly on the current moment in time and tracking how specific frequencies evolved earlier in the clip. What's notable is that the model organizes audio into acoustically coherent regions—grouping similar sounds together—without ever being told what those regions should be. It discovers structure purely through the act of predicting forward.
Alex: So the model builds a map of sound just by asking, what comes next?
Sam: That's the core claim. And the paper argues this establishes causal next-embedding prediction as a simple, scalable foundation for audio understanding—one that could eventually be extended to systems that process audio, text, and images together in a unified way.
Alex: A cleaner approach to a genuinely difficult problem. Thanks for walking us through it, Sam.
Sam: Thanks for listening to ResearchPod.