Ensemble Controlled-Flow Filtering for Implicit Data Assimilation

Data assimilation aims to estimate the state of a dynamical system by combining model forecasts with incoming observations. Traditional methods like the Ensemble Kalman Filter (EnKF) rely on Gaussian assumptions, while particle filters often struggle with high-dimensional degeneracy. This paper addresses the challenge of performing data assimilation when observation mechanisms are many-to-one, non-smooth, implicit, or only accessible via simulators, where standard likelihood-based guidance is unavailable.

The authors introduce Implicit Data Assimilation, which defines the analysis update as an energy-based tilt of the forecast distribution. To realize this update, they propose the Ensemble Controlled-flow Filter (EnCF). This method uses a stochastic controlled flow to transport the forecast ensemble toward the analysis distribution. Unlike existing flow-based filters that require likelihood gradients at every transport step, EnCF approximates the optimal control by using adjoint matching, which propagates terminal energy gradients backward through the dynamics. For cases where no analytic likelihood exists, the authors provide a likelihood-free variant, EnCF-LF, which learns a surrogate energy function from simulator samples.

The study demonstrates that EnCF is exact in the ideal limit and provides a robust framework for complex observation models. Numerical experiments show that while Kalman-type filters remain optimal for smooth, additive-Gaussian observations, EnCF and EnCF-LF significantly outperform them in regimes involving non-Gaussian, multimodal, many-to-one, or implicit observation models. The authors also provide a theoretical error decomposition, showing that local errors do not accumulate under filter stability, ensuring the reliability of the multi-step filtering process.