SPyCE: Skill-Policy Co-evolution for Multimodal Agents

Multimodal agents often struggle to accumulate stable, reusable tool-use capabilities across diverse tasks. Existing reinforcement learning (RL) methods typically treat completed trajectories as scalar rewards, failing to extract the underlying structured knowledge. This paper asks: can we improve multimodal agent performance by distilling reasoning trajectories into reusable skills that co-evolve with the policy during training?

The authors propose SPyCE (Skill-Policy Co-evolution), a framework that organizes knowledge into a two-tier hierarchical library:

1. Execution Skills: These capture local visual operations (e.g., cropping, rotating) as condition-action-effect triples.
2. Workflow Skills: These encode high-level procedural priors for orchestrating multi-step reasoning and tool use.

During training, the agent performs skill-conditioned rollouts. Successful trajectories are then distilled back into the skill library using a merge-or-add rule that suppresses redundancy. This creates a closed loop: the policy uses retrieved skills to guide its actions, and the resulting high-quality rollouts are used to refine and expand the skill library, which in turn provides stronger priors for future policy updates.

Experiments across eight benchmarks—covering agentic reasoning, general multimodal tasks, and visual search—demonstrate that SPyCE consistently outperforms both standard RL-based and memory-based baselines. The authors show that the hierarchical design is essential, as workflow skills provide global planning guidance while execution skills handle fine-grained visual bottlenecks. Furthermore, the online co-evolution mechanism is critical for sustaining performance gains, as it allows the agent to continuously refine its skill library based on the improved policy.

This work shifts the paradigm of multimodal agent training from simple reward-based optimization to a more structured, knowledge-distillation approach. By treating interaction trajectories as a source of reusable, structured knowledge rather than just scalar signals, SPyCE offers a scalable way to build agents that become more capable and efficient over time.