Nicola Farenga, Andrea Manzoni
8 min
Abstract
Learning the continuous-time evolution of nonlinear dynamical systems relies on trajectory-based objectives, which require simulating the learned dynamics during training and backpropagating through the numerical scheme, at a cost that grows with the rollout horizon. We show that trajectory-based objectives degenerate as the rollout horizon increases, hindering gradient-based optimization, and that the numerical scheme biases the learned dynamics by constraining the spectrum of the learned Jacobian to its absolute stability region. To overcome these limitations, we introduce Velocity Matching (VM), a simulation-free objective that relies on piecewise-linear interpolation and stochastic collocation, and approximates the velocity population risk to second order in the sampling step. We develop a theoretical framework for velocity-based learning, introducing stability estimates that control trajectory errors in terms of velocity errors, which enable us to establish universal and continuous-time approximation results when the learned velocity field is parameterized by deep neural networks. Building on a bias-variance decomposition of the VM objective, we show that its higher-order approximation of the velocity population risk yields a consistent minimizer; that it relates to the residual of an A-stable continuous-stage Runge-Kutta scheme, thus connecting VM to implicit integration without requiring any implicit solve; and that it implicitly matches the action of the learned Jacobian to that of the true one along the observed trajectories, without requiring access to the underlying velocity field or its Jacobian. Numerical experiments on nonlinear ordinary differential equations and semi-discretized time-dependent partial differential equations show that VM improves accuracy over trajectory-based objectives, while avoiding both the cost of simulation and the instabilities of long rollouts.
Sam: Not in that result. At finite horizons, the minimizing parameters remain tied to the numerical update. Increasing the horizon flattens the landscape around them. The authors also establish a related restriction on learned dynamics near a stable equilibrium, under specific assumptions.
Alex: Let’s turn to the alternative. What does the model learn if it never generates a training trajectory?
Sam: It learns a velocity field: a rule assigning a rate of change to each system state. “Velocity” here doesn’t have to mean physical motion. The target is how the observed state changes.
Alex: But observations usually give you states, not their instantaneous rates. Where does that supervision come from?
Sam: Take consecutive observations and connect them with a straight segment. Its slope is their change in state divided by the elapsed time. Sample a point along that segment, ask the network for its rate there, and compare it with the slope.
Alex: So the training input sits between observed states. You aren’t asking the model to generate that intermediate state itself.
Sam: Both the sampled state and the target slope come from the data. This random sampling is called stochastic collocation: evaluating the learning condition at randomly chosen points. The authors call the resulting objective Velocity Matching.
Alex: Estimating rates from observations isn’t new. What makes this more than another version of that approach?
Sam: The paper explicitly builds on earlier velocity supervision. It also contrasts with teacher forcing, which trains each prediction from an observed state. With a simple forward update, teacher forcing can be read as matching a finite-difference slope—but only at the observed starting point.
Alex: The distinction is where the network gets evaluated: at the endpoint, or throughout the segment. Earlier interpolation approaches already used points between observations, though.
Sam: Yes. The authors discuss prior work using higher-order splines, which are smooth fitted curves. Their claim is that straight segments suffice, provided you sample along them. They connect that construction to stronger approximation guarantees and a particular kind of implicit regularization.
Alex: Before the regularization, what justifies replacing trajectory errors with rate errors? Those aren’t automatically interchangeable.
Sam: The authors prove stability estimates that bound trajectory error using velocity error along the true trajectories. That last detail makes simulation-free training possible. You evaluate the learned rule on reference data, not on its own evolving predictions.
Alex: There must be a condition preventing a nearly correct rule from becoming wildly unstable just outside those observations.
Sam: The bound depends on the learned rule’s Lipschitz constant: how rapidly its output can change with its input. To justify minimizing velocity error across candidate models, those constants need uniform control. The result also identifies the true rule only in regions visited by the observed flow.
Alex: So this is neither unrestricted extrapolation nor a guarantee that any low training loss gives accurate long-term predictions.
Sam: And the authors acknowledge that the bounds can become uninformative. Their constants grow exponentially with the time horizon and that sensitivity measure. Still, the estimates establish a conditional bridge between learning local changes and recovering trajectories.
Alex: What does sampling along a straight segment buy mathematically? The slope itself is still a rough estimate.
Sam: Under their assumptions, the full objective approximates the ideal velocity loss with second-order accuracy in the observation spacing. Teacher forcing gives first-order accuracy. That concerns the approximation of the loss, not a claim that every individual slope is equally accurate.
Alex: And what is the regularization doing? That could mean simply pushing the model toward slower, smoother dynamics.
Sam: Their decomposition separates an average mismatch from variation along each segment. They show an implicit matching of the Jacobian’s action along observed trajectories. The Jacobian describes sensitivity to small state changes; this does not mean recovering its entire matrix everywhere.
Alex: That distinction matters. You’re constraining how the rule changes along the observed motion, without supplying sensitivity measurements.
Sam: The average mismatch also corresponds to the residual of an implicit integration method. Such methods normally require solving for an unknown future state. Here, the future observation is already available, so training checks the residual without performing that solve.
Alex: Let’s separate the proofs from performance. How much better did the method do in the tests?
Sam: Under a common iteration budget, the authors report errors more than tenfold lower than teacher-forcing and rollout baselines. They report training cost comparable to teacher forcing. The tests cover nonlinear ordinary differential equations and time-dependent partial differential equations represented after spatial discretization.
Alex: Do we have enough detail here to judge how broad that advantage is?
Sam: Not fully. This excerpt doesn’t provide benchmark counts, dataset sizes, or repeated-run details. It reports successful changes of integration scheme and step size, plus preservation of invariants and chaotic attractors on tested systems. Those are encouraging results, not a universal guarantee.
Alex: And real observations can be noisy. Differencing nearby measurements could make that especially relevant.
Sam: The authors explicitly leave noisy observations largely unexplored. They also leave open which classes of dynamics benefit most from the random sampling. I’d treat this as a theoretically supported method with promising numerical evidence, rather than a settled recipe for every dataset.
Alex: Who should read the full document, and where should they start?
Sam: Researchers learning continuous-time dynamics should start with Section 4.1 for the training recipe. Read Section 3.2 for the stability assumptions, then Section 5 for the reported comparisons and experimental detail. For everyone else: training a simulator need not require running it—but accurate rates still need stable dynamics.
Alex: That’s a useful distinction to keep.