MIDI-RAE-JEPA: HIERARCHICAL REPRESENTATION LEARNING AND GENERATION FOR SYMBOLIC MUSIC

How can we learn semantically rich, hierarchical representations of symbolic music (piano rolls) that support both discriminative tasks and high-quality generative modeling without relying on large-scale, supervised pretraining?

The author introduces MIDI-RAE-JEPA, a system that treats symbolic music as 2D images. The architecture uses a Swin Transformer V2 encoder trained via self-supervised objectives, specifically focusing on equivariance to pitch and time shifts. Unlike standard image models that use crop-and-resize, this model enforces a target embedding distance proportional to the magnitude of pitch and time shifts. The training pipeline includes:

• LeJEPA: A Joint-Embedding Predictive Architecture that uses Sketched Isotropic Gaussian Regularization (SIGReg) to prevent representation collapse.
• Soft Factorization: A loss function that encourages pitch and time dimensions to be geometrically orthogonal in the latent space.
• RAE Pipeline: A frozen encoder paired with a separate decoder for reconstruction, and a flow-matching generative model for conditional generation.

The model demonstrates that equivariance-based objectives effectively capture musical structure. Key results include:

• Measurable Equivariance: Embedding distances increase monotonically with pitch and time shift magnitude, confirming the model internalizes these musical relationships.