Progress in AI has largely been driven by methods that assume less. As compute and data increase, approaches with weaker inductive biases generally outperform those with stronger assumptions. This is particularly characteristic of the field of Visual Representation Learning, where approaches have gone from being dominated by Supervised Learning, to Weakly Supervised Learning, to the now widespread success of Self-Supervised Learning without human labels. Yet, even modern Self-Supervised Learning approaches still depend on strong inductive biases such as augmentations, masking, or cropping. If this trend holds, even these remaining biases should become bottlenecks at scale -- and our experiments confirm this: the optimal strength of inductive biases decreases as data grows. This motivates the search for approaches that rely on fewer assumptions. To this end, we introduce Temporal Difference in Vision (TDV), a new paradigm for self-supervised learning from video that avoids existing inductive biases, relying instead on a causal assumption that the past causes the future. TDV functions by jointly training an image encoder and a motion encoder so that the current frame's representation plus the encoded motion equals the next frame's representation. Despite not leveraging any strong inductive biases, TDV matches state-of-the-art recipes on dense spatial tasks, laying the foundation for representation learning without strong assumptions.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how artificial intelligence learns to see the world.
Sam: We're discussing a new approach called Temporal Difference in Vision, or TDV. The central claim is that as AI systems get larger and process more data, they actually perform better when we stop forcing them to follow rigid, human-designed rules.
Alex: So this paper is basically arguing that our current way of teaching AI is holding it back?
Sam: That's the argument. Right now, to train a computer to "see," we feed it static photos and use tricks like cropping or masking out parts of the image to force the model to pay attention. These are what researchers call "inductive biases." Think of them like training wheels on a bicycle—they help the model learn at first, but eventually, they limit how fast and freely you can ride.
Alex: That makes sense. But if you take the training wheels off, doesn't the model just fall over?
Sam: That's the core challenge. If you simply remove those rules, the system often fails to learn anything useful—a problem called "representation collapse." Imagine asking someone to describe every photo they see, but they just respond with the same blank stare every time. The model essentially gives up and maps everything to the same empty output. The authors wanted to find a way to remove those training wheels without the system crashing.
Alex: So they needed some other, gentler assumption to keep the model on track?
Sam: Precisely. They turned to a very simple idea from everyday life: the past causes the future. In a video, what you see right now directly shapes what comes next. By training the AI to predict the next frame based on the current one, they give it a constant, natural learning signal—no artificial tricks required.
Alex: Oh—so instead of staring at a single frozen photograph, the AI is watching a continuous stream of time?
Sam: Exactly. The system has two parts working together. There's a "frame encoder" that understands what the current picture looks like, and a "motion encoder" that calculates what has changed between this frame and the next. Think of it like video compression software—it stores one clear image and then just tracks the movement from there. By combining those two signals, the model learns to predict what comes next.
Alex: And this works without any of those extra human-designed rules?
Sam: It does. They also use a technique called self-distillation to keep things stable. A "teacher" network—which is just a slower-moving copy of the model itself—provides a reliable target for the "student" version to aim for. This stops the model from collapsing while it learns. The key insight is that as we scale up, we can rely on the simple, universal flow of time rather than increasingly complex human assumptions.
Alex: But if the model isn't using those human-designed rules, how does it know what to focus on? Does it struggle with the difference between "what something is" versus "where it is"?
Sam: That's a sharp question, and the paper addresses it directly. TDV turns out to be highly effective at "where"—tasks like tracking motion or understanding depth—because it preserves fine-grained spatial details across every pixel in a frame. It's slightly less focused on semantic labels, meaning it's better at understanding the geometry of a scene than at naming the objects in it. It lacks the specific rules that would force it to group pixels together and say "that's a car" or "that's a tree."
Alex: So it's learning the physics of the scene rather than the vocabulary.
Sam: That's a good way to put it. It's a different kind of seeing, built on the structure of time rather than on human-defined categories.
Alex: How do we know which parts of the model are actually doing the heavy lifting?
Sam: The authors ran what are called "ablation studies." The name sounds technical, but the idea is simple: you take the system apart piece by piece and see what breaks. They found two components were absolutely critical.
Alex: Which ones?
Sam: The first is the motion encoder. Remove it, and the model loses its ability to track how things change over time—and the whole system collapses. It stops learning anything useful at all. The second is the error-correction step. After the model makes a prediction about the next frame, this step measures how far off that prediction was from reality. It's like a student checking their own homework. Without it, the model's predictions drift further and further from what's actually happening on screen.
Alex: So those two pieces are what stop it from just making things up?
Sam: Exactly. The motion encoder keeps it anchored to real change over time, and the error-correction keeps its predictions honest. Together with the stable "teacher" network, they prevent the system from taking shortcuts or producing blank, uninformative outputs. It's a self-contained learning loop.
Alex: It does sound like a more natural way to learn. Instead of being told what to look at, the AI is just figuring out how the world moves on its own.
Sam: That's the paper's central point. The authors aren't claiming TDV beats every other model on every task—they're careful about that. But they do argue it demonstrates something meaningful: that as AI vision systems grow larger, we can scale them up by leaning on the simple, universal structure of time, rather than piling on more and more human-designed rules. Whether that holds across all the domains where we might want AI to see clearly is still an open question.
Alex: A question worth watching. Thanks for walking us through it, and thanks to everyone listening to ResearchPod.