Saurav Singh, Rodney Sanchez, Alexander Ororbia, Jamison Heard
8 min
Abstract
Reinforcement learning (RL) has the potential to transform real-world decision-making systems by enabling autonomous agents to learn from experience. Deploying RL in real-world settings, especially in the context of human-robot interaction, requires defining state representations and reward functions, which are critical for learning efficiency and policy performance. Traditional RL approaches often rely on domain expertise and trial-and-error, necessitating extensive human involvement as well as direct interaction with the environment, which can be costly and impractical, especially in complex and safety-critical applications. This work proposes a novel RL framework that leverages off-policy evaluation (OPE) for state space and reward function selection, using only logged interaction data. This approach eliminates the need for real-time access to the environment or human-in-the-loop feedback, greatly reducing the dependency on costly real-time interactions. The proposed approach systematically evaluates multiple candidate state representations and reward functions by training offline RL agents and applying OPE to estimate policy performance. The optimal state space and reward function are selected based on their ability to produce high-performing policies under OPE metrics. Our method is validated on two environments: the Lunar Lander environment by OpenAI Gym, which provides a controlled setting for assessing state space and reward function selection, and a NASA-MATB-II human subjects study environment, which evaluates the approach's real-world applicability to human-robot teaming scenarios. This work enhances the feasibility and scalability of offline RL for real-world environments by automating critical RL design decisions through a data-driven OPE-based evaluation, enabling more reliable, effective, and sustainable RL formulation for complex human-robot interaction settings.
Alex: So it's about how much gap there is between winners and losers under each scoring system?
Sam: Precisely. They measured that gap by looking at how spread out the score patterns were for good versus bad strategies—like checking if two teams' game stats cluster far apart or overlap confusingly. The reward with the biggest separation won. In Lunar Lander, the full baseline reward—which mixed position, action, and landing scores—showed the clearest separation and led to the best-performing strategies.
Alex: That data-driven pick avoids guessing. And it carried over to the NASA setup?
Sam: Yes, where multitasking humans got precise help from the selected RL agents, outperforming rigid rule-based bots in adapting to overload.
Alex: How did they apply something similar for choosing what info to feed the robot in the NASA multitasking setup?
Sam: In the NASA-MATB-II environment—where astronauts juggle monitoring gauges and lights, tracking objects, managing fuel pumps, and handling radio calls—they tested different bundles of input data for the robot. One used just task details like gauge readings and fuel levels; others added human body signals such as heart rate and breathing, or estimates of mental workload from those signals. They trained sample strategies on each bundle, then used off-policy estimates to rank them. Task-plus-body-signals gave strong predicted performance without the extra computing demands of adding workload estimates.
Alex: So it's not just piling on data—it's picking what truly shapes good actions. And for rewards there?
Sam: They defined three reward styles: one docking points mainly for errors; another crediting fixes and steady performance; the third blending both. Off-policy checks showed the blend had the widest score gaps between random and strong strategies. That made it the choice for reliable guidance in overload scenarios.
Alex: Practical for space teams, then—adapting without overload guesses. Did they test the RL agent with real people under stress?
Sam: Yes, they ran an IRB-approved study with 19 participants using the NASA MATB-II simulator. People juggled four tasks—tracking a moving target with a joystick, monitoring system lights for alerts, managing fuel pumps over time, and handling radio calls—while cycling through underload, normal, and overload conditions. Participants teamed with one of three agents: a rule-based one that follows fixed if-then rules like a strict recipe, the offline RL agent using the selected inputs and blended reward, or an RL fine-tuning agent that tweaks itself live using fresh data.
Alex: So rule-based is rigid, offline RL adapts from picks, and fine-tuning personalizes on the fly. Which held up best across tasks?
Sam: The offline RL agent showed the most consistent performance overall, especially adapting to overload. In system monitoring, it had the fastest response times and low failure rates, outperforming rule-based under normal and overload. This came from selective automation: it intervened precisely when humans were swamped, thanks to the data-driven choices guiding timely help without overdoing it. Tracking also favored RL for steady low errors, while rule-based did better on resource management.
Alex: Huh, so the offline picks enabled smarter timing, not just more actions.
Sam: Fine-tuning shone in communications for quicker responses but varied more elsewhere, trading broad reliability for personalization risks.
Alex: Makes sense—the offline RL struck a balance for multi-task overload without live tweaks gone wrong. But people aren't just tracking numbers—they have feelings about the robot teammate. Did the study capture that?
Sam: Yes, they gathered in-situ ratings right after each condition on trust, workload, and fluency—how smoothly the team worked. The RL agent led in trust across loads, with about 15 percent higher ratings on average. It also showed the lowest perceived workload and highest fluency, suggesting its targeted help eased strain and felt more in sync.
Alex: Real-time gut checks. How about post-task surveys?
Sam: Post-trial questionnaires with tools like the NASA-TLX showed fine-tuning edging out on some trust items and fluency, while RL kept the lowest overall workload. The paper notes a tradeoff—more adaptive agents build trust but can raise effort if unpredictable.
Alex: That lines up with the objective side: RL balances help and steadiness. Any broader limits?
Sam: A key one is the need for high-quality, diverse past data—if logs lack variety, estimates skew and picks falter. Fine-tuning can amplify issues under stress. The study suggests refining those mechanisms next.
Alex: Pulling it together: this off-policy method lets robots tune themselves offline for human teams, boosting help in overload without live risks. Strong in trust and workload relief across space-like multitasking.
Sam: That's the takeaway—the framework scales RL design data-driven, outperforming rigid rules in adaptive spots. For fields like surgery or healthcare, it means testing aids on stress without patient peril.
Alex: A solid step for safer teaming. Thanks, Sam—that's it for this look at optimizing robots for human collaboration through off-policy evaluation. Thanks for listening to ResearchPod.