ResearchPod Summary
Predictive State Representations (PSRs) offer a powerful way to model dynamical systems by tracking the probabilities of future observable events (tests) based on past experiences (histories). Unlike latent-variable models like Hidden Markov Models, which rely on unobserved states, PSRs are defined entirely by observable data, making them statistically consistent and easier to learn. However, traditional PSR algorithms struggle with continuous actions and observations because they often require data-heavy density estimation or discrete binnings that fail for complex, high-dimensional systems.
This paper generalizes PSRs to handle continuous systems by leveraging Hilbert space embeddings. Instead of tracking discrete probabilities, the authors embed distributions of tests and histories into Reproducing Kernel Hilbert Spaces (RKHSs). By representing the state as a nonparametric conditional embedding operator, the model can capture near-arbitrary continuous stochastic processes.
To update the model as new actions and observations arrive, the authors apply a kernelized version of Bayes' rule. This allows for recursive state updates directly within the Hilbert space. By using the "kernel trick," the authors demonstrate how to perform this filtering efficiently using Gram matrices, effectively bypassing the need to store infinite-dimensional operators explicitly. The result is a highly flexible model that can learn from continuous sensory data without needing labels for latent variables.
Experiments on synthetic nonlinear systems, real-world robotic slot car telemetry, and robot arm end-effector tracking demonstrate that HSE-PSRs consistently outperform competing methods like Kernelized Linear Dynamical Systems (K-LDS) and Input-Output HMMs. This approach provides a robust, principled framework for learning control-dependent dynamics in scenarios where traditional parametric assumptions are invalid.
Alex: You can predict a robot’s motion without first inventing its hidden states. Byron Boots, Arthur Gretton, and Geoffrey Gordon propose learning directly from actions and observations, including continuously varying measurements. Their method paper, “Hilbert Space Embeddings of Predictive State Representations,” reports competitive or better predictions across several benchmarks.
Sam: Predicting from observations sounds useful, but the practical question is what this changes. Who would reach for this approach?
Alex: Researchers learning models of controlled systems from sensory data, especially when the underlying state is not directly visible. Think of a robot arm observed through a depth camera. The promise is to learn how future observations depend on actions, without requiring labels for hidden states. We’ll unpack that idea, compare it with earlier approaches, and ask how much the experiments establish.
Sam: Let’s start with the state itself. If it isn’t a hidden physical description, what does the model actually remember?
Alex: It remembers predictions. A predictive state representation defines state through the expected outcomes of future observable experiments. Each experiment specifies actions to take and observations that might follow. The paper calls these experiments “tests,” and calls the past sequence of actions and observations a “history.”
Sam: So state means what the system might do next, given what we’ve already seen. But surely you cannot store every possible future experiment?
Alex: You need a sufficient collection, called a core set. Its predictions contain enough information to derive predictions for all other tests. That is the defining assumption. Instead of retaining the entire history, you retain this predictive summary.
Sam: That explains the representation. Why did continuous measurements make earlier versions difficult to learn?
Alex: Many learning algorithms assumed a modest number of discrete actions and observations. With continuous measurements, exact values rarely repeat. You cannot reliably estimate a separate probability for every action and observation combination. Earlier continuous approaches used kernel density estimation or particular families of distributions, with drawbacks including slow statistical convergence and difficult numerical integration.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: Hidden-state models already address partially observed systems. What advantage do the authors claim over learning one of those?
Alex: Predictive states are defined entirely through observable quantities. Earlier spectral learning methods recovered discrete predictive models by manipulating observed moments, avoiding local search. The authors contrast that with expectation maximization, a common hidden-state learning procedure that can converge slowly or reach bad local optima. This paper extends the predictive approach beyond small discrete spaces.
Sam: Then take us through the extension. What replaces a long list of probabilities when the observations are continuous?
Alex: A distribution becomes an average feature vector. A kernel is a function that compares samples through features we need not explicitly construct. The associated function space is called a reproducing kernel Hilbert space. Averaging features over a distribution gives its “mean embedding.”
Sam: Averaging sounds like throwing information away. Wouldn’t different distributions have the same average?
Alex: They can with ordinary features. But the authors use characteristic kernels, where each distribution has a unique embedding. So this is not merely replacing a distribution with its ordinary mean. Under the paper’s mathematical assumptions, the representation can retain the full distributional information.
Sam: The model still needs to connect actions to outcomes, rather than just describe a distribution. Where does that connection live?
Alex: The predictive state is an operator: a mapping from future action features to an embedding of future observations. Given the history, it answers what observations to expect under a proposed action sequence. The representation is expressive, but it assumes that the required conditional operators exist and that a core collection of futures is sufficient.
Sam: Could you make the update concrete using one of their actual systems?
Alex: Consider their slot car. The input is its velocity, and the observations include acceleration and orientation measurements. The model first predicts a longer future that includes the next observation. It then conditions on the selected input, and incorporates the observation that actually arrives.
Sam: That sounds like the familiar predict-and-correct cycle of a filter. What makes this version different?
Alex: The correction happens directly in the distribution embeddings, using kernel Bayes’ rule: Bayesian conditioning in that feature space. There is no need to reconstruct a probability density and numerically integrate it. In computation, the state is represented by weights on training samples, and kernel comparison matrices implement the updates.
Sam: Kernel methods had already been used for dynamical models. What is genuinely new relative to those?
Alex: The earlier embedded hidden Markov model discussed here could not reason about actions, and its update relied on density estimation. This method adds actions and an embedded Bayesian update. Another related approach handled partially observed controlled systems, but required training labels for the true latent states. This approach learns without those labels.
Sam: That is a substantial methodological distinction. What do the experiments show beyond the construction working on paper?
Alex: The authors test a synthetic nonlinear system, the slot car, and a simulated robot arm. Comparisons include a linear dynamical model, a kernelized nonlinear alternative, and a discrete input-output hidden Markov model. They also use simple baselines where appropriate. On the synthetic task, their method meets or exceeds the competing models’ performance.
Sam: The slot car brings in real sensor measurements. How much data did they use, and what happened there?
Alex: They collected 2,500 time steps. Only 500 were used for training; the remainder were held out for testing. They filtered through observed data, then predicted future sensor readings using the supplied control signal. Their model consistently had the lowest reported prediction error.
Sam: And the robot example involves images rather than just a few sensor readings. How demanding was that comparison?
Alex: The observations included depth images, with joint angles treated as control inputs. The goal was predicting the arm’s end-effector position. The dataset contained 1,000 successive observations, split equally between training and testing. Predictions extended as far as 50 steps ahead using the inputs.
Sam: Did the competing models remain credible on that task, or did the comparison mostly expose failures?
Alex: Some struggled substantially. The kernelized alternative produced an unstable model, while the stabilized linear model had poor predictive accuracy. The authors report substantially lower mean prediction error for their method. That supports its usefulness on this difficult benchmark, but we should not translate the plotted results into an invented percentage improvement.
Sam: Repeated predictions along one trajectory can look like a lot of evidence. How far would you trust the broader claim?
Alex: I’d call this promising benchmark evidence, not broad validation. The experiments use limited training sequences and repeated prediction starts within held-out trajectories. Those starts are not independent systems. Also, the learning setup assumes a blind policy, meaning actions do not depend on observations; the slot-car policy violates that assumption, despite good empirical performance.
Sam: There is also an implementation question. Does avoiding hidden-state estimation make the model compact?
Alex: Not necessarily. Its nonparametric state has length proportional to the training dataset. More data therefore means a larger state representation. The method avoids explicitly manipulating infinite-dimensional features, but that does not make storage and computation independent of sample size.
Sam: Given those trade-offs, who should read the full document, and where should they start?
Alex: Researchers in system identification, filtering, or prediction from sensory data should read it closely. Start with “Predictive State Embeddings” and the section on updating state with kernel Bayes’ rule. Then read the robot experiment for the demanding comparison, and the learning section for the sample-based implementation. For everyone else, remember this: a system’s state can be what you predict, not a hidden thing you name.
Sam: Keep that distinction in mind when choosing your next model.