ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs

As multimodal large language models (MLLMs) are increasingly positioned as the cognitive cores of household robots, they must be able to react to sudden physical hazards—such as a falling knife or a slipping plate—with safety and precision. The authors ask: can current MLLMs translate their visual understanding of physics into immediate, safety-critical actions, and how do they perform when evaluated in a physically grounded, embodied setting rather than through passive video question answering?

The authors introduce ReactHuman, a benchmark that evaluates MLLMs by placing them as the 'brain' of a simulated humanoid in a 3D physics engine. The benchmark features 17 event families and over 1,000 reproducible scenes. Using a 'freeze-and-predict' protocol, the simulation pauses at a critical moment, requiring the model to output a structured reaction plan (walking commands and hand keyframes). This plan is then executed by a pre-trained whole-body controller, allowing the researchers to measure the actual physical outcome of the model's decision. The benchmark includes adversarial objects—such as a foam anvil or a steel apple—to test whether models rely on visual appearance or observed physical dynamics.

Testing seven representative MLLMs, the authors find that reactive safety is far from solved. Models mishandle roughly one in three hazards, often by freezing in place rather than attempting to dodge. The study reveals that models possess fixed behavioral dispositions (e.g., a tendency to always retreat) that override the specific requirements of the scene. Furthermore, models treat speed as a binary (moving vs. not moving) and fail to adjust their reactions based on the velocity of hazards. Notably, performance does not improve with model scale, and models consistently fail to revise their physical assumptions when faced with adversarial objects that look like one material but behave like another.