Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, Yoshua Bengio
9 min
Machine learning has achieved remarkable success, yet it remains largely confined to finding statistical associations in independent and identically distributed (i.i.d.) data. While this approach works well under fixed experimental conditions, it fails significantly when the world changes—such as in transfer learning, generalization to novel environments, or reasoning about the effects of interventions. Animals excel at these tasks precisely because they possess internal models that distinguish between mere correlation and the underlying causal mechanisms that drive the data-generating process.
This paper advocates for a transition from purely statistical models to causal representation learning. Unlike traditional machine learning, which treats data points as inputs to be mapped to labels, causal learning treats the world as a system of autonomous modules. The central challenge is moving beyond hand-engineered causal variables to discovering high-level causal variables directly from raw, low-level sensory input like pixels. By learning these variables, agents can form structural models of the environment that are more robust, transferable, and capable of counterfactual reasoning.
Central to this vision is the Independent Causal Mechanism (ICM) principle. It posits that the world is composed of autonomous modules—mechanisms that do not influence one another. This principle leads to the Sparse Mechanism Shift (SMS) hypothesis: when a distribution changes (e.g., due to an intervention), only a small subset of the causal factors is usually affected. By designing models that favor these sparse, local changes over dense, global entanglement, researchers can build artificial intelligence that reuses knowledge across tasks and generalizes more reliably under distribution shifts.
Integrating causality offers a fresh perspective on common machine learning practices. Techniques like semi-supervised learning, adversarial defense, and data augmentation can be reinterpreted as specific strategies for dealing with non-i.i.d. scenarios or for approximating causal robustness. The authors suggest that by formalizing these techniques through a causal lens, we can build more versatile AI systems capable of acting in the imagined spaces that characterize true intelligence.
The two fields of machine learning and graphical causality arose and are developed separately. However, there is, now, cross-pollination and increasing interest in both fields to benefit from the advances of the other. In this article, we review fundamental concepts of causal inference and relate them to crucial open problems of machine learning, including transfer and generalization, thereby assaying how causality can contribute to modern machine learning research. This also applies in the opposite direction: we note that most work in causality starts from the premise that the causal variables are given. A central problem for AI and causality is, thus, causal representation learning, that is, the discovery of high-level causal variables from low-level observations. Finally, we delineate some implications of causality for machine learning and propose key research areas at the intersection of both communities.
Alex: No, and the authors explicitly warn against that reading. Variables can be strongly related even when their generating mechanisms are independent. Think of altitude and average annual temperature at weather stations.
Sam: How does that example show what can transfer?
Alex: Austria and Switzerland can have different distributions of station altitudes. But the relationship describing temperature given altitude may remain approximately the same. Separate that relationship from the distribution of altitudes, and part of your model can travel across countries.
Sam: Whereas a model of altitude given temperature mixes the temperature-producing mechanism with where the weather stations happen to be.
Alex: That reversed description need not share the same stability. The authors use this contrast to motivate a causal decomposition: divide the overall distribution into the mechanisms that produce it. The hope is to preserve those pieces when only some conditions change.
Sam: But real environments can change in many ways at once. What makes you expect only a few pieces to need updating?
Alex: That’s the sparse mechanism shift hypothesis. Small distribution changes tend to affect only a few causal mechanisms, rather than all of them. It is a hypothesis about useful structure, not a guarantee that every change in the world is local.
Sam: If that hypothesis holds, adaptation becomes a repair job instead of a complete rebuild. But someone still has to decide what the modules describe.
Alex: And that’s the central gap this review highlights. Traditional causal discovery usually starts with meaningful variables already supplied. Causal representation learning asks how to learn those variables and their relationships from low-level observations, such as images.
Sam: So this goes beyond learning which way arrows point between columns in a dataset. You may first need to discover the things that deserve columns.
Alex: Consider the review’s finger-and-cube example. Moving the finger changes the positions of the finger and cube, but it changes many image pixels. A useful representation captures the small change at the object level, instead of treating every changed pixel as a separate event.
Sam: Could an image-reconstruction system discover those objects just by compressing images well?
Alex: Not reliably from that objective alone. The review explains that independent observations admit infinitely many equivalent solutions for recovering the underlying noise variables. Additional assumptions or learning signals are needed to favor a causally useful representation.
Sam: What signals could distinguish those equivalent explanations?
Alex: Observed interventions, changes across environments, or supervision that reveals particular factors. The authors discuss networks with a causal model embedded inside them. The goal is not necessarily to recover all of reality, but variables supporting causal statements relevant to particular tasks.
Sam: Which part is new here, rather than established causality translated into machine-learning language?
Alex: The review builds on established causal models and the independent-mechanism principle. Its contribution is the synthesis: connecting those ideas to learned representations, transfer, and common training practices. It also states the sparse mechanism shift hypothesis and makes learning the variables a central research problem.
Sam: That’s a coherent agenda. What evidence should stop us treating it as an attractive story?
Alex: The review draws on theoretical results under specific assumptions, plus existing applications. It reports that sparse mechanism shifts can be sufficient to identify representations with suitable training data. But which factors become identifiable depends on which interventions are observed.
Sam: Does it also offer an application where causal modeling helped with real scientific observations?
Alex: It describes astronomy work that removed shared measurement noise across stars to reduce the instrument’s influence. That application led to 36 planet candidates. Of those, 21 were subsequently validated as exoplanets.
Sam: That supports the usefulness of causal modeling in a particular application. It doesn’t show that learning causal variables generally beats a conventional neural network.
Alex: That’s the boundary. The example is not a broad comparison of representation-learning methods. The review itself calls for compelling real-world evidence of advantages in generalization, transfer, and reuse.
Sam: Why not just train a bigger model on more diverse data, including many kinds of changes?
Alex: The authors take that objection seriously. Broader training data can improve generalization if it contains information about the relevant changes. But without enough diversity, errors on unseen shifts may still be arbitrarily large.
Sam: Then the causal approach makes the expected changes more explicit, rather than eliminating assumptions.
Alex: Yes. It specifies which mechanisms may change and which should remain stable. That also offers a way to interpret data augmentation: image translations or rotations expose a model to selected changes. They encourage robustness to those changes, not every possible shift.
Sam: So how much should a reader trust the overall promise?
Alex: Treat this as a framework for asking better questions, not a universal recipe. The authors acknowledge unresolved identification problems, insufficient evaluation protocols, and a shortage of scalable, numerically sound algorithms. Even the right variables depend on the tasks and interventions available.
Sam: Who should read the full review, and where should they start?
Alex: Researchers working on transfer, robustness, representation learning, or reinforcement learning should read it. Start with Independent Causal Mechanisms for the organizing principle, then Learning Causal Variables for the harder representation problem. Read those sections with one question: what changes can my data actually reveal?
Sam: And what should everyone else carry away?
Alex: Predicting what you will observe is not the same as predicting what will happen when you act.
Sam: Keep that distinction close.