Ninad Daithankar, Alexi Gladstone, Yann LeCun, Heng Ji
5 min
Abstract
Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.
Alex: But if the model isn't using those human-designed rules, how does it know what to focus on? Does it struggle with the difference between "what something is" versus "where it is"?
Sam: That's a sharp question, and the paper addresses it directly. TDV turns out to be highly effective at "where"—tasks like tracking motion or understanding depth—because it preserves fine-grained spatial details across every pixel in a frame. It's slightly less focused on semantic labels, meaning it's better at understanding the geometry of a scene than at naming the objects in it. It lacks the specific rules that would force it to group pixels together and say "that's a car" or "that's a tree."
Alex: So it's learning the physics of the scene rather than the vocabulary.
Sam: That's a good way to put it. It's a different kind of seeing, built on the structure of time rather than on human-defined categories.
Alex: How do we know which parts of the model are actually doing the heavy lifting?
Sam: The authors ran what are called "ablation studies." The name sounds technical, but the idea is simple: you take the system apart piece by piece and see what breaks. They found two components were absolutely critical.
Alex: Which ones?
Sam: The first is the motion encoder. Remove it, and the model loses its ability to track how things change over time—and the whole system collapses. It stops learning anything useful at all. The second is the error-correction step. After the model makes a prediction about the next frame, this step measures how far off that prediction was from reality. It's like a student checking their own homework. Without it, the model's predictions drift further and further from what's actually happening on screen.
Alex: So those two pieces are what stop it from just making things up?
Sam: Exactly. The motion encoder keeps it anchored to real change over time, and the error-correction keeps its predictions honest. Together with the stable "teacher" network, they prevent the system from taking shortcuts or producing blank, uninformative outputs. It's a self-contained learning loop.
Alex: It does sound like a more natural way to learn. Instead of being told what to look at, the AI is just figuring out how the world moves on its own.
Sam: That's the paper's central point. The authors aren't claiming TDV beats every other model on every task—they're careful about that. But they do argue it demonstrates something meaningful: that as AI vision systems grow larger, we can scale them up by leaning on the simple, universal structure of time, rather than piling on more and more human-designed rules. Whether that holds across all the domains where we might want AI to see clearly is still an open question.
Alex: A question worth watching. Thanks for walking us through it, and thanks to everyone listening to ResearchPod.