Learning compatible representations aims to learn feature representations that can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate that stationary representations learned by d-Simplex fixed classifiers imply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility using $d$-Simplex fixed classifiers when the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with the cross-entropy loss aligns feature distributions at the first-order statistics. Consequently, it may not fully capture higher-order dependencies in the representation between model updates. To address this issue, we demonstrate that training the model using a $d$-Simplex fixed classifier through a convex combination of the cross-entropy loss and a contrastive loss not only captures higher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show that stationary representations enable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a paper that tackles a surprisingly stubborn problem in artificial intelligence: how do you upgrade a model without breaking everything it already knows?
Sam: So, imagine a facial recognition system used at an airport. It has millions of faces stored in a database. If you want to improve the AI—make it more accurate, say—do you really have to re-process every single one of those records from scratch?
Alex: That's exactly the problem. When you update an AI model, the way it "sees" and describes data tends to shift. Think of it like two people using different languages to describe the same photograph. The new model speaks a slightly different language than the old one, so it can no longer make sense of the old database. And re-processing millions of records is slow and expensive.
Sam: So the researchers are looking for a way to make sure the new model speaks the same language as the old one, even after it's been improved?
Alex: Exactly. And their solution is geometric. Here's the core idea: imagine every category the AI needs to recognize—every person's face, every type of object—as a point on a map. Normally, when you retrain a model, those points drift around. The researchers' fix is to nail those points down permanently, at the corners of a specific mathematical shape called a regular simplex.
Sam: A regular simplex—what does that actually look like?
Alex: Think of the simplest version: a triangle, where every corner is exactly the same distance from every other corner. Now extend that idea into many more dimensions. The key property is that every category gets its own corner, and every corner is equally spaced from all the others. No category is closer to or farther from any other.
Sam: So it's a perfectly fair, perfectly fixed map. And the AI is forced to always place "class A" at the same corner, no matter how many times it gets updated?
Alex: That's it. Because the map never moves, the relative positions of all the categories stay stable across updates. The researchers call this "stationary representation." The practical result is that a newer, smarter model and an older model are still describing the world in compatible terms.
Sam: And the paper actually proves this works mathematically—it's not just a hope?
Alex: That's the central claim. The authors show that when you use these fixed, stationary targets during training, the models automatically satisfy the mathematical conditions required for compatibility. You don't have to add a separate compatibility step; it's baked in by the geometry.
Sam: But I'd imagine just fixing the map isn't enough on its own. How do you actually train the model to use it correctly?
Alex: Good question. They found that a single training objective isn't sufficient to capture all the relationships in complex data. So they combine two. One part of the training pushes the model to be accurate—to correctly identify what it's looking at. The second part is about contrast: it actively pushes different categories apart from each other in that geometric space, while pulling similar things together.
Sam: So without that second part, two different categories might end up sitting too close together on the map, even if the corners are fixed?
Alex: Precisely. The fixed corners define where things should end up, but the contrastive piece is what teaches the model to respect the full structure of the space—not just the endpoints, but the relationships between everything in between. Together, they give you both accuracy and long-term consistency.
Sam: And the practical payoff is that you can deploy an upgraded model without taking the whole system offline to re-index the database?
Alex: That's the key practical benefit. The underlying map is the same, so the new model can search through records that were indexed by the old model without any translation step. You swap in the upgrade, and the service continues without interruption.
Sam: How do you actually measure whether this is working? How do you know the old and new models are truly compatible?
Alex: The researchers use what they call a Compatibility Matrix—think of it as a scoreboard. For every pair of model versions, it records how well the newer model performs when searching through a gallery that was indexed by an older one. A high score means the new model can still find the right face in the old gallery. A low score means the two models have drifted apart.
Sam: And this approach keeps those scores high even after many rounds of updates?
Alex: That's where it becomes particularly notable. Most methods degrade over time—each update introduces a little more drift, and after several rounds, compatibility scores drop significantly. This approach maintains stable scores across a long sequence of updates, because the fixed geometric map prevents that cumulative drift from building up.
Sam: It's like each update is working from the same master blueprint, so errors don't compound.
Alex: That's a good way to put it. And it connects back to why the higher-order contrastive training matters. By preserving the full structure of the space—not just the category corners, but the relationships between them—the model doesn't quietly forget earlier structure when it learns something new.
Sam: Are there limitations? It sounds almost too clean.
Alex: There is a meaningful one. The method assumes you know in advance how many categories you'll ever need to represent. The regular simplex has a fixed number of corners. It's like designing a library with a set number of shelves—if the collection grows far beyond what you planned for, you run out of space. For systems with a stable, well-defined set of categories, this works well. For open-ended problems where new categories keep appearing, it's a genuine constraint.
Sam: So it's well-suited to something like a corporate security system with a known set of employees, but less so for a system that needs to keep learning entirely new kinds of things indefinitely.
Alex: That's a fair characterization. Within its intended scope, though, the approach addresses a real and costly problem. The ability to upgrade a model continuously—without re-indexing a database or taking a service offline—has significant practical value for any large-scale recognition system.
Sam: It's interesting that the solution isn't about making the model more powerful in the traditional sense. It's about giving it a stable foundation so that power doesn't come at the cost of continuity.
Alex: That's well put. A lot of AI research focuses on peak performance. This paper asks a different question: how do you build a system that stays reliable over time, through many rounds of improvement? The answer, it turns out, may be as much about geometry as it is about algorithms. Thanks for listening to ResearchPod.