Connect the Dots: Training LLMs for Long-Lifecycle Agents with Cross-Domain Generalization Via Reinforcement Learning

Research Overview

This paper addresses a critical limitation in current LLM-based agents: their tendency to treat each task as an isolated event, failing to leverage knowledge gained from previous interactions within the same environment. The authors propose the "Connect the Dots" (CoD) framework, a meta-capability that allows agents to interleave task-solving episodes with context-updating episodes. By doing so, the agent proactively builds a persistent understanding of its environment, which it then uses to improve performance on subsequent tasks.

The authors implement CoD through a specialized reinforcement learning (RL) post-training process called CoD-Train. Unlike standard RL, which optimizes for individual task success, CoD-Train uses a GRPO-style algorithm that performs credit assignment across long sequences of tasks. The agent is incentivized to maximize both immediate task rewards and future performance by effectively managing a "hint"—a piece of context that persists across the lifecycle of the deployment. The framework was tested on environments designed to require information discovery, such as FrozenLake-Obscure, where the mapping of actions is randomized and must be learned through trial and error.

Key Findings

Empirical results demonstrate that agents trained with the CoD framework significantly outperform those trained via standard task-by-task RL. For instance, in the FrozenLake-Obscure environment, the agent's success rate on the fourth task in a sequence increased from 28% to 76% after training. The authors also observed that the elicited meta-capability generalizes across different domains and to other settings, such as Ralph-loop scenarios, suggesting that the model learns a transferable skill for environment exploration and context management rather than just memorizing specific task solutions.

As LLMs move toward long-term autonomous deployment, the ability to "learn while doing" is essential. This framework provides a concrete, scalable path for training agents that can adapt to new, underspecified environments without requiring human-crafted scaffolds or constant retraining. It bridges the gap between static reasoning models and truly autonomous, lifelong learning agents.