Implementation of reinforcement learning in chemical reaction networks: application to phototaxis as curiosity-driven exploration

How can simple, non-neural organisms like unicellular algae exhibit complex, adaptive navigation in uncertain environments? The authors investigate whether the "run-and-tumble" behavior observed in Chlamydomonas can be reframed not as a mechanistic stimulus-response rule, but as an active, information-seeking strategy that resolves sensory ambiguity through biochemical computation.

The researchers model algal phototaxis as a Partially Observable Markov Decision Process (POMDP). In this framework, the cell maintains a minimal internal belief state about its orientation relative to a light source, which is updated via Bayesian inference based on noisy, localized sensory inputs. The authors define an action-selection policy that balances two goals: exploiting known light gradients and exploring to reduce uncertainty (curiosity). Crucially, they translate this abstract decision-making logic into a set of Chemical-Reaction-Network Ordinary Differential Equations (CRN-ODEs). By using Inverse Reinforcement Learning (IRL) on 30 experimental trajectories, they validate that their model reproduces the empirical alignment patterns observed in real algae.

The study reveals that tumbling is an effective information-acquisition strategy. When the cell's sensory geometry creates ambiguity (e.g., when the light source is directly behind the cell), the curiosity-driven policy triggers a tumble to reorient the cell and gain new sensory information. The authors demonstrate that this entire decision-making pipeline—from belief updates to action evaluation—can be computed by mass-action chemical reactions. Simulations show that the CRN-ODE implementation matches the numerical policy with high fidelity, suggesting that intracellular chemistry is sufficient to support sophisticated, goal-oriented navigation.