Visual interaction models that support users during ongoing physical tasks require evaluation in closed loop, where assistant responses can change subsequent user behavior and task outcomes. Existing benchmarks largely rely on offline replay and therefore cannot capture these effects. We introduce Interactive Visual Gym, a framework for closed-loop evaluation and training of visual interaction models through joint simulation of executable task environments and human users. We contribute three simulators that recreate everyday tasks and human-aligned action spaces across cooking, household manipulation, and mobile-interface domains, together with a persona-conditioned user simulation harness that collaborates with the assistant while incorporating controlled execution errors and other out-of-plan behaviors. We instantiate the framework as IVG-Bench, a benchmark with 315 interaction scenarios. Evaluating state-of-the-art VLMs as visual interaction models reveals substantial limitations: the best-performing model, Gemini 4 Argon, completes only 53.4% of tasks within the time limit, exhibits significant weaknesses in factual grounding and actionable guidance, and is often too slow to prevent errors when response latency is considered. We verify our evaluation protocol with a human study which finds 78.5% agreement between our interaction-quality metric and participant preferences. We further show that the environments provide useful learning signals: in a proof-of-concept GRPO experiment, closed-loop training improves both task success and interaction quality for Qwen3.5-4B model in our cooking environment.
Alex: A correct warning can still be useless if it arrives after the mistake. A new benchmark tests visual assistants with simulated users who keep acting while the assistant thinks. Its central finding: advice that sounds helpful does not necessarily produce successful assistance.
Sam: That sounds like a different test from judging an answer. Whose work is this, and what did they build?
Alex: It’s a preprint from Chenhao Zheng and colleagues at the University of Washington and Salesforce AI Research. Called Interactive Visual Gym, it introduces a framework and benchmark for assistants that watch ongoing activities and offer guidance.
Sam: So if I build an assistant for cooking or household tasks, this tests whether its advice actually helps someone finish?
Alex: That’s the aim. The strongest model completed only about half the tasks under the main evaluation settings. We’ll unpack how the test works, why response speed changes the results, and where simulated users limit what we can conclude.
Sam: Start with the gap. What does this capture that existing visual-assistance benchmarks miss?
Alex: Existing benchmarks largely replay recorded activity and score an assistant against reference responses. But the person in that recording cannot react to new advice. If the assistant says “stop,” the video carries on regardless.
Sam: So replay can assess whether a warning fits the recording, but not whether it prevents anything. How do the authors make that consequence measurable?
Alex: They use closed-loop interaction: advice changes actions, which change what the assistant sees next. They simulate both the environment and the person doing the task. The assistant communicates with that person rather than directly controlling the world.
Sam: What does the simulated world contain? It needs more than pictures if you want to know whether dinner was prepared correctly.
Alex: There are environments for cooking, household activities, and phone settings. Each keeps a structured record of objects, their properties, and relationships. That record lets the engine check whether a dish has cooked ingredients in the required order, for example.
Sam: And it can recognize different ways of completing the same task, rather than demanding one prescribed sequence?
Alex: Yes. Progress depends on conditions in the world, not matching a reference route. A cooking user can prepare ingredients in different valid sequences. In the household simulator, success can depend on dishes ending up inside the dishwasher, regardless of the route taken.
Sam: The person is also a model. How do they keep it from becoming another perfectly obedient task-solving agent?
Alex: A separate vision-language model, which processes images and text, plays the user. Instructions give it a persona: skill, trust in artificial intelligence, execution speed, and interruption preferences. A harness also inserts scheduled mistakes, questions, or plan changes when specified task conditions occur.
Sam: Does the assistant know a mistake is scheduled? Otherwise it might just be responding to a test label.
Alex: The schedule and private action plan are hidden from the assistant. Consider the cheeseburger example: the user is about to add another bun when the bun layer is already complete. Guidance can persuade the user to stop before adding it, or arrive afterward and require recovery.
Sam: That distinguishes spotting an error from preventing it. But how do they score advice that is correct yet hard to follow?
Alex: They measure task completion, error detection, and interaction quality separately. Quality covers factual grounding, relevance, actionable guidance, respect for user intent, and conciseness. Two language-model judges assess responses against the recorded world state, not a preferred wording.
Sam: A fluent answer could satisfy several of those categories while inventing what happened. Does factual accuracy carry special weight?
Alex: It does. A factual-grounding failure zeros that judge’s overall response score. Also, user agreement does not establish correctness. The judges check the engine evidence, so a simulated person saying “you’re right” cannot make a false claim true.
Sam: With those distinctions in place, what is the scale of the test, and what counts as success?
Alex: The benchmark contains 315 interaction scenarios across the environments. Success means satisfying the engine-checked goal within a completion budget derived from an error-free reference execution. The main comparisons average independent repeated runs.
Sam: You said the best assistant finished about half. Was that with its actual response time included?
Alex: Not in the main comparison. Under standardized response timing, Gemini 4 Argon led task success at roughly 53 percent, averaged equally across environments. It still trailed some other frontier assistants on interaction quality, including factual grounding and actionable guidance.
Sam: So completion alone can flatter an assistant. A capable simulated user might finish despite poor advice.
Alex: The authors show that higher completion does not necessarily mean better interaction quality. One open-weight assistant completed more cooking tasks than a frontier model while scoring much worse on guidance. These measures capture different aspects of assistance.
Sam: Now let the clock run while the assistant thinks. How much does that change the result?
Alex: In cooking, Argon’s success fell from about 61 percent with fixed delay to about 33 percent with measured delay. The user kept executing queued actions during response generation. Warnings could arrive after an irreversible mistake, adding recovery work or causing failure.
Sam: That could change which model you would choose, not just lower everyone’s score. Did the rankings move?
Alex: They did. Muse Glimmer led cooking success when measured latency was included, while Argon led under fixed timing. But the latency test is still a model of time: inference delays become simulation ticks, with caps. User-model computation itself does not advance that clock.
Sam: The larger concern is whether these simulated people behave like people. What validation do the authors offer?
Alex: They studied 45 participants using the browser-based cooking simulator. Participants ranked anonymously labeled assistants after interacting with them. The interaction-quality metric agreed with their preferences in roughly four out of five eligible assistant-pair comparisons.
Sam: That supports the scoring. Does it also validate the behavior of the simulated users?
Alex: Only partly. Some differences associated with trust and interruption preferences resembled human differences. But simulated users spoke and replied much more often. Humans largely performed the tasks silently, while simulated users frequently described their next actions.
Sam: That matters because the conversation itself helps determine what the assistant notices and how much guidance it gives.
Alex: It limits how representative these interactions are. Results also changed when the authors changed the user model. And although scheduled errors covered many observed human mistakes, the study found behaviors outside that taxonomy. The schedules test challenging mistakes, not their natural frequencies.
Sam: The environments also support training. Did learning inside them improve assistance beyond the training scenarios?
Alex: A small cooking experiment used reinforcement learning: rewards tied to task success, prevention, false alarms, and excess turns. Held-out validation success rose by about nine percentage points. But those scenarios shared recipe and layout combinations with training, and there was no separate frozen test set.
Sam: So that is evidence of a learning signal, rather than broad transfer to unfamiliar tasks?
Alex: That is the safer reading. Error-detection recall also declined after training. Additional phone-settings experiments did not establish reliable frozen-test improvement, and interaction quality fell. The training evidence is preliminary, not a demonstrated general solution.
Sam: Given those limits, who should read the full document, and where should they start?
Alex: Researchers building visual assistants or evaluating interactive systems should read it. Start with the user–assistant interaction method, then the comparison between fixed and measured latency. Read the human-study appendix for the communication mismatch. For training, inspect the reinforcement-learning appendix before interpreting the headline gains.
Sam: And what should everyone else remember, especially before applying these results outside a simulator?
Alex: The authors explicitly warn that simulator performance is not evidence of real-world safety. What this framework adds is a way to test consequences rather than just responses. Useful assistance is not only saying the right thing; it is changing what happens in time.
Sam: Thanks for listening.