Achyuthan Unni Krishnan, Karthik Rengarajan, Bharath Narasimhan, Chris von Csefalvay
7 min
Abstract
Robots in homes, warehouses and farms must pick up objects they have never seen, usually without a 3D model or a category label. A grasping policy that works directly on the point cloud from a depth sensor needs only the geometry the robot observes, so it applies to any new object. Offline grasp generators such as Contact-GraspNet work this way, but they rank grasps by learned plausibility rather than by whether the grasp actually lifts the object. Reinforcement learning (RL) can optimize for that directly and make a point-cloud grasping policy much better on average, yet it leaves specific grasps persistently unsolved. We present a human-in-the-loop failure loop that targets those grasps: situations an RL policy keeps failing are mined automatically, sent to a virtual-reality application where a person picks the object up with their own hands, and returned to training as two grasp points. The person contributes only where to grasp; the robot chooses its own approach. In simulation, RL raises success on unseen objects from 55% with Contact-GraspNet to 83%. Adding 58 human demonstrations nearly doubles success on the hard cases RL kept failing, from 24% to 45%, and recovers performance that RL alone loses on similar cases that were never demonstrated, without reducing success on unseen objects.
Alex: Failed attempts enter a queue with the exact object and pose. Training periodically retries them, removing cases that succeed. Repeated failures are exported, then checked again using the finished policies so that occasional unlucky attempts don’t automatically consume human attention.
Sam: And once a case reaches the person, are they steering a robot hand, or demonstrating with their own?
Alex: Their own tracked hands, inside virtual reality. The object appears in its recorded pose on a virtual table. The person grasps and lifts it, and the application records the opposing fingertip contacts.
Sam: Human fingers can reach places a robot gripper cannot. Why should those contacts be useful to a different hand?
Alex: The authors deliberately avoid copying the whole hand motion or orientation. They transfer only where to grasp. A simulated robot gripper then searches for an approach that makes those contacts work, and checks the lift under execution noise.
Sam: So a successful human lift is only a proposal. The robot’s own hand has to verify it before training uses it.
Alex: That verification is central. The session produced 58 robust demonstrations. During retraining, the policy imitates their contact points only when an executed demonstration actually lifts the object.
Sam: Does that leave the final policy tied to human choices, or can reinforcement learning move away from them?
Alex: The imitation weight fades to zero early in training. Demonstrated cases keep appearing afterward, but reinforcement learning decides what to do with them. Also, this is a fresh training run from the same warm start, not a direct patch to the finished policy.
Sam: Let’s separate learning those exact examples from learning something transferable. What did their evaluation show?
Alex: On demonstrated hard cases, lifting success rose from about 24 percent with reinforcement learning alone to about 45 percent with demonstrations. That’s the near-doubling in the headline. But exact-case improvement could include memorization.
Sam: If the same object appears in another orientation, the contact locations have to make sense again. Did they hold out situations like that?
Alex: They kept a hidden set that the demonstrator never saw. Most involved either a demonstrated object at another pose, or another instance from a demonstrated family. Success there rose from about 23 percent to about 34 percent.
Sam: That supports transfer across related situations, not necessarily to completely new kinds of object. How does the starting policy change that interpretation?
Alex: It changes it substantially. On hidden cases, the demonstration-trained policy essentially returned to the warm start’s performance. Meanwhile, it retained the broader improvements from reinforcement learning on unfamiliar objects.
Sam: So the strongest interpretation is that demonstrations helped preserve something ordinary training gave up. They did not establish better hidden-case performance than the starting policy already had.
Alex: That’s the authors’ interpretation too. On held-out objects, reinforcement learning alone lifted roughly four in five attempts, versus about half for Contact-GraspNet’s top proposal in the flat condition. Adding demonstrations produced no measurable change to that broader performance.
Sam: The hard cases were chosen because reinforcement learning failed them. Doesn’t that selection make the comparison unusually favourable to anything else?
Alex: It does, and the authors acknowledge it. They tested an additional reinforcement-learning run that had not selected the cases. The demonstrated-case gain remained clear, while the hidden-case advantage became smaller.
Sam: There’s another issue: the demonstration-trained policy sees targeted cases more often and gets extra supervision. Can we attribute the improvement specifically to human judgment?
Alex: Not from this design. Targeted replay, contact supervision, and imitation change together. The authors call for a matched comparison using machine-found grasps under the same training recipe and budget.
Sam: Did automated search already solve many of these failures? That would matter when deciding whether a person belongs in the loop.
Alex: Yes. Given enough attempts, random search solved many of the same cases. The human supplied verified contacts with relatively few attempts, including some missed by the automated alternatives. That supports efficient collection, but doesn’t isolate its downstream training value.
Sam: How far beyond this controlled setting should a robotics researcher trust the result?
Alex: The main comparison used three training runs per condition, with substantial variation between demonstration-trained runs. All testing was simulated, and the sole demonstrator was an author. A simulated robot-arm sorting test did not establish an advantage over reinforcement learning alone.
Sam: And the observations were fairly generous, weren’t they?
Alex: The policy expects points covering most of the object, not a sparse view of one side. The contact physics were not calibrated against a real gripper. Those are major gaps between this result and deployment.
Sam: Given those gaps, who should read the full document, and where should they start?
Alex: Researchers working on grasp learning or targeted human supervision should read “Training with Demonstrations.” Then read “Comparison With the Warm Start” and the limitations on confounded components. Together, those sections explain both the useful training mechanism and what the experiment cannot yet establish.
Sam: And what should everyone else remember when an improved average is presented as progress?
Alex: A better average can hide lost skills. Here, targeted help recovered some of them without surrendering the broader gain—but only within the tested simulation.
Sam: Keep the failures in view.