Niccolò Biondi, Federico Pernici, Simone Ricci, Alberto Del Bimbo
6 min
Abstract
Learning compatible representations aims to learn feature representations that can be used interchangeably over time whenever a model undergoes updates. In this paper, we demonstrate that stationary representations learned by d-Simplex fixed classifiers imply compatibility as in its formal definition. This result establishes a foundation for future works and can be directly exploited in practical learning scenarios. We address the challenge of learning compatibility using $d$-Simplex fixed classifiers when the model is sequentially fine-tuned. Learning according to a d-Simplex fixed classifier with the cross-entropy loss aligns feature distributions at the first-order statistics. Consequently, it may not fully capture higher-order dependencies in the representation between model updates. To address this issue, we demonstrate that training the model using a $d$-Simplex fixed classifier through a convex combination of the cross-entropy loss and a contrastive loss not only captures higher-order dependencies, but is also equivalent to learning with the cross-entropy under the compatibility constraints. We confirm our findings with extensive experiments also considering a new scenario where a pre-trained model is sequentially fine-tuned and occasionally replaced with an improved model. We show that stationary representations enable uninterrupted retrieval services (without reprocessing gallery images) while improving performance during model updates and replacements, achieving state-of-the-art. Code at https://github.com/miccunifi/iamcl2r.
Alex: Good question. They found that a single training objective isn't sufficient to capture all the relationships in complex data. So they combine two. One part of the training pushes the model to be accurate—to correctly identify what it's looking at. The second part is about contrast: it actively pushes different categories apart from each other in that geometric space, while pulling similar things together.
Sam: So without that second part, two different categories might end up sitting too close together on the map, even if the corners are fixed?
Alex: Precisely. The fixed corners define where things should end up, but the contrastive piece is what teaches the model to respect the full structure of the space—not just the endpoints, but the relationships between everything in between. Together, they give you both accuracy and long-term consistency.
Sam: And the practical payoff is that you can deploy an upgraded model without taking the whole system offline to re-index the database?
Alex: That's the key practical benefit. The underlying map is the same, so the new model can search through records that were indexed by the old model without any translation step. You swap in the upgrade, and the service continues without interruption.
Sam: How do you actually measure whether this is working? How do you know the old and new models are truly compatible?
Alex: The researchers use what they call a Compatibility Matrix—think of it as a scoreboard. For every pair of model versions, it records how well the newer model performs when searching through a gallery that was indexed by an older one. A high score means the new model can still find the right face in the old gallery. A low score means the two models have drifted apart.
Sam: And this approach keeps those scores high even after many rounds of updates?
Alex: That's where it becomes particularly notable. Most methods degrade over time—each update introduces a little more drift, and after several rounds, compatibility scores drop significantly. This approach maintains stable scores across a long sequence of updates, because the fixed geometric map prevents that cumulative drift from building up.
Sam: It's like each update is working from the same master blueprint, so errors don't compound.
Alex: That's a good way to put it. And it connects back to why the higher-order contrastive training matters. By preserving the full structure of the space—not just the category corners, but the relationships between them—the model doesn't quietly forget earlier structure when it learns something new.
Sam: Are there limitations? It sounds almost too clean.
Alex: There is a meaningful one. The method assumes you know in advance how many categories you'll ever need to represent. The regular simplex has a fixed number of corners. It's like designing a library with a set number of shelves—if the collection grows far beyond what you planned for, you run out of space. For systems with a stable, well-defined set of categories, this works well. For open-ended problems where new categories keep appearing, it's a genuine constraint.
Sam: So it's well-suited to something like a corporate security system with a known set of employees, but less so for a system that needs to keep learning entirely new kinds of things indefinitely.
Alex: That's a fair characterization. Within its intended scope, though, the approach addresses a real and costly problem. The ability to upgrade a model continuously—without re-indexing a database or taking a service offline—has significant practical value for any large-scale recognition system.
Sam: It's interesting that the solution isn't about making the model more powerful in the traditional sense. It's about giving it a stable foundation so that power doesn't come at the cost of continuity.
Alex: That's well put. A lot of AI research focuses on peak performance. This paper asks a different question: how do you build a system that stays reliable over time, through many rounds of improvement? The answer, it turns out, may be as much about geometry as it is about algorithms. Thanks for listening to ResearchPod.