Mingyuan Lei, Yoonchang Sung, Tat-Jen Cham
9 min
Abstract
Generating realistic human-object interactions (HOI) in complex 3D scenes requires two complementary capabilities: reasoning about interaction feasibility in the environment and synthesizing realistic human-object motion. However, supervision for these capabilities is rarely available jointly at scale. Human-scene datasets provide rich information about environment-aware motion, while human-object datasets capture detailed interaction dynamics, yet paired human-object-scene data remain scarce. We present MAMHOI, an affordance-mediated factorization for scene-aware human-object interaction generation. MAMHOI factorizes scene-aware HOI generation through an explicit motion-affordance interface between scene understanding and motion synthesis: a scene-conditioned model first predicts where and how an interaction can be feasibly executed, and an affordance-conditioned HOI model then generates the corresponding human-object motion. This factorization allows scene understanding and interaction dynamics to be learned from complementary sources of supervision without requiring paired human-object-scene data. Experiments in complex indoor environments show that MAMHOI reduces object--scene penetration while better preserving human--object interaction quality, yielding more realistic and physically feasible scene-aware interactions. Project page: https://leimingyuan.github.io/MAMHOI-project-page/
Sam: Make the shared region concrete. What happens for the lamp moving through chairs?
Alex: That’s an example in the document: lift the floor lamp, walk through the chairs, and put it down. The scene model receives scene geometry, a planned object path, and the instruction. It predicts a region covering where the person and lamp can interact.
Sam: If there’s already a planned path, how much movement is left for the generator to decide?
Alex: The path supplies coarse object-level structure, not the final human movement. The predicted region also doesn’t prescribe one complete trajectory. The generator still synthesizes coordinated poses, movement, and contact, conditioned on the task and those spatial constraints.
Sam: A whole-room map sounds unwieldy. How does that become information the movement model can actually use?
Alex: The method lifts the predicted region into the scene’s three-dimensional space. It then queries local grids around the person’s pelvis, the current object, and the object’s goal. Each grid consists of voxels, small cells representing local spatial constraints.
Sam: Those local views separate where the person stands, what surrounds the object, and where it needs to end. Do they update as the interaction progresses?
Alex: They do. Motion is generated in short, overlapping segments. Each segment updates the human and object states, which determine the next local queries. The grids are also aligned with the person’s orientation, keeping nearby geometry consistent with the motion representation.
Sam: Where does the detailed movement come from, if that generator never trains on these furnished rooms?
Alex: It learns from OMOMO, the human–object dataset. During training, the authors construct spatial regions around observed movement. The scene model learns corresponding regions from LINGO’s indoor human–scene data. That common representation is what connects the independently trained components.
Sam: The two datasets supply the same kind of interface, even though their recordings differ. What kind of model produces the motion?
Alex: A diffusion model, which repeatedly removes noise to generate coordinated human and object movement. The local region features condition that process. Training also rewards motion reconstruction, contact prediction, and consistency between joint rotations and resulting body positions.
Sam: A generator could still ignore the region and rely on its familiar movement patterns. How do the authors discourage that?
Alex: They penalize joints outside or too close to the region’s boundary. They also compare alternative movements with similar starts and goals but different intermediate paths. A ranking objective favors the movement compatible with the supplied region over the alternative.
Sam: That tests whether the middle of the movement responds to the environment, not just whether it reaches the endpoint. Does this extra pressure cost anything?
Alex: Yes. Removing the ranking objective improves several scene-free motion-quality scores. Keeping it improves scene compatibility, smoothness, and contact in the scene-aware tests, with small trade-offs in path adherence. It’s a scene-conditioning regularizer, not a universal improvement to motion fidelity.
Sam: Let’s move to the evidence. How much scene-aware testing did they do?
Alex: The main constructed test set uses twenty indoor test scenes from LINGO. The authors manually specify feasible start and goal configurations, then plan object paths. Testing movable objects along those paths yields about eight hundred twenty trials.
Sam: What’s the most interpretable result against an earlier generator?
Alex: Against CHOIS, the share of frames with human–scene penetration falls from about thirty percent to fifteen percent. Penetration means the generated body overlaps scene geometry. MAMHOI also reduces object–scene penetration and improves contact and object-path adherence.
Sam: Could those gains simply come from correcting collisions after generation, rather than learning better movement?
Alex: The default results do not use the optional scene-collision refinement. The authors apply the same refinement to both CHOIS and MAMHOI as a separate comparison. Even without it, MAMHOI outperforms refined CHOIS on the reported scene measures, jerk, path error, and contact.
Sam: Lower overlap doesn’t necessarily mean a movement looks natural. Did anyone judge the sequences themselves?
Alex: The perceptual study collected eight hundred eighty-four pairwise judgments. Participants saw matched tasks and scenes with method identities hidden. Unrefined MAMHOI was preferred over CHOIS in about eighty-seven percent of judgments.
Sam: That supports the geometric results, but doesn’t turn collision correction into a naturalness guarantee.
Alex: The authors make that distinction too. Optional refinement reduces human–scene penetration but does not improve human preference in their comparisons. It also introduces small trade-offs in object-related measures. Correcting geometry and learning scene-adapted motion are complementary, not interchangeable.
Sam: What most limits how far we should generalize these results?
Alex: The main scene-aware set has no paired ground-truth human–object movement. It tests physical proxies and preferences under feasible endpoints and planned object paths. It doesn’t establish unrestricted task planning or fully valid physics. Penetration is reduced, not eliminated.
Sam: Given that scope, who should read the full document, and where should they begin?
Alex: Researchers generating scene-aware motion should start with Methodology, especially the affordance interface. Then read Results on Scene-Aware Test Set and the ranking-loss ablation. Those comparisons separate learned adaptation from collision refinement and expose the trade-off with scene-free motion quality.
Sam: And what should researchers outside motion generation carry away?
Alex: My verdict is that this is credible evidence for the interface within a constructed benchmark, not a guarantee of physical realism. The broader lesson is simple: teach where an interaction fits separately from how it moves, then connect the skills through a shared representation.
Sam: Thanks for listening.