Language models trained on large-scale vision-language data have demonstrated strong potential for embodied agents. Harnessing models through embodied tools use offers a promising alternative to end-to-end vision-language-action systems by combining high-level reasoning with external modules for perception, planning, and control. However, it remains unclear what makes an effective harness for embodied manipulation, and to what extent such a harness can unlock embodied capabilities in a wide range of reasoning models. In this work, we present Guava, a harness framework for embodied tool use developed through systematic exploration of the design space of agent workflows, action spaces, and observation spaces. Our study identifies three key ingredients for effective embodied agents: iterative perception-reasoning-action loops, semantic action abstractions, and multimodal observations. To understand whether these design principles are universal even to small models, we develop an end-to-end training pipeline that distills embodied manipulation capabilities into a 4B open-source model using fewer than 2K trajectories collected entirely in simulation. Experimental results in both simulation and real-world environments show performance comparable to frontier proprietary models while exhibiting strong generalization to unseen objects, novel instructions, and long-horizon tasks. Results suggest that a well-designed harness can serve as a scalable, model-agnostic interface for embodied manipulation, enabling strong emergent embodied capabilities in compact open-source models with minimal training data.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a framework called Guava. It addresses a persistent problem in robotics: why do massive, expensive AI models often fail at simple physical tasks—like picking up a dropped object? The paper argues the answer isn't just more data. It's about better "harnessing"—how we structure the interaction between the AI and the robot's physical environment.
Alex: So the question is why robots struggle to recover when things go wrong, and whether there's a smarter way to connect the AI's "brain" to the robot's "hands"?
Sam: Exactly. Most current systems try to force the AI to control every tiny motor movement directly. If the robot drops a fork, it often just keeps moving blindly, because it was never trained to notice the mistake. Guava acts as a bridge. It lets the AI think in high-level steps—like a chef who doesn't need to know how to manufacture a knife, but knows exactly how to use one.
Alex: That's a useful distinction. It's like giving the AI a set of tools rather than forcing it to learn physics from scratch. So how does this "harness" actually work?
Sam: It relies on three design choices working together. The first is an iterative loop—the robot constantly perceives the scene, reasons about its next move, and then acts. If a move fails, the loop catches it. The system pauses, registers the error, and tries again. Researchers call this a "look, think, act" cycle, and it repeats until the job is done.
Alex: So the robot isn't just following a pre-set script. It's actively checking its own progress as it goes.
Sam: Precisely. The second choice is what the paper calls "semantic action abstraction." Rather than calculating the exact angle for every joint in the arm, the AI issues simple, high-level commands—things like "grasp" or "move to." This reduces the burden on the model considerably, letting it focus on the *logic* of the task rather than the complex geometry of every movement.
Alex: And the third?
Sam: The third is using what they call multimodal observations. Think of it as giving the robot both eyes and a notepad. It takes in visual images of the scene, but it also receives a plain-text description of the task. By combining what it *sees* with what it *knows*, the model gets a much clearer picture—which helps it avoid confusion when two objects look similar, for instance.
Alex: I follow the logic. But if this approach is so effective, why isn't everyone already doing it? Running these large AI models must be expensive.
Sam: That's a critical point. Using a massive, proprietary model for every single step of a task is slow and costly—not practical for a real robot in a real kitchen. So the researchers asked whether they could "distill" these capabilities. The idea is to teach a smaller, more efficient model to mimic the reasoning patterns of the large one, rather than relying on the large one permanently.
Alex: Squeezing that intelligence into a smaller container. How much data did they need to pull that off?
Sam: Fewer than two thousand practice runs, all conducted inside a computer simulation. That's a notably small dataset for this kind of task. The resulting model—which is meaningfully smaller than the large proprietary systems—performed at a comparable level, and it even handled objects it had never encountered during training.
Alex: So the "harness" itself is doing a lot of the heavy lifting, not just the raw size of the model.
Sam: That is the central argument. By structuring the interaction properly, you can make a smaller, cheaper system behave with the same robustness as a much larger one. It suggests that the *interface* between AI and the physical world deserves as much attention as the AI model itself.
Alex: You mentioned a two-stage training process. How does that help the model learn to recover from mistakes?
Sam: The first stage is supervised fine-tuning—essentially, the model reads through a large set of worked examples, the way a student studies a textbook before an exam. That gives it a solid foundation. The second stage is where recovery behaviour gets reinforced.
Alex: What happens in that second stage?
Sam: They use a technique called Group Relative Policy Optimization—GRPO for short. Here's a useful way to picture it: imagine a student who's given the same maths problem five times and tries a different approach each time. The teacher doesn't just mark them right or wrong—they compare all five attempts and give the highest grade to the one that reached the correct answer most efficiently. GRPO works the same way. The model generates several different sequences of actions for a given task, compares them, and reinforces whichever path actually led to success.
Alex: So it's learning from its own trial and error, rather than just copying examples.
Sam: Exactly. And crucially, it learns *what to do when things go wrong*. If a grasp fails, the model learns that the right response is to reset and try again—not to continue the failed motion. That's the recovery behaviour the earlier approach was missing.
Alex: That explains how it gets so capable with so little training data. It's not just memorising steps—it's learning a strategy.
Sam: That's a good way to put it. And it points to something worth sitting with: the most significant gains here didn't come from scaling up the model or collecting more data. They came from rethinking how the AI is connected to the physical world in the first place. That's a meaningful shift in how researchers are approaching this problem.
Alex: Thanks for walking us through it. And thanks to our listeners for joining us on ResearchPod.