Ilyes JAOUEDI, Gilles CHARDON, José PICHERAL
4 min
Abstract
We address the problem of localizing multiple sources in 3D by combining sensor array measurements with camera observations. We propose a fusion framework extending the covariance matrix fitting method with an unbalanced optimal transport regularization term that softly aligns sensor array responses with visual priors while allowing flexibility in mass allocation. To solve the resulting largescale problem, we adopt a greedy coordinate descent algorithm that efficiently updates the transport plan. Its computational efficiency makes full 3D localization feasible in practice. The proposed framework is modular and does not rely on labeled data or training, in contrast with deep learning-based fusion approaches. Although validated here on acoustic arrays, the method is general to arbitrary sensor arrays. Experiments on real data show that the proposed approach improves localization accuracy compared to sensor-only baselines.
Alex: Like elastic bands connecting camera sights to sound hotspots—they tug toward matches but stretch if something's off.
Sam: Yes. They mix this into CMF with a dial for strength: low trusts mics more, high leans on cameras. It balances mic patterns, angle costs, and power mismatches.
Alex: So it finds grid powers that fit the mics while softly aligning with camera rays. Why UOT instead of strict matching?
Sam: Strict matching demands camera object "masses" match sound powers exactly. But cameras can't tell a quiet speaker from a loud one—they just see the shape. UOT lets powers vary freely.
Alex: No labeled data required—just sound physics for patterns and room geometry for angles. How do they keep it fast enough for real time?
Sam: They use greedy coordinate descent. It tweaks one link at a time, always choosing the change that cuts total cost the most—like solving a puzzle by grabbing the piece that fits best right now.
Sam: In tests with a gym setup—four speakers, 32-mic array, smartphone camera—it updated in about 2.5 seconds on a laptop. That's real-time 3D tracking.
Alex: And it beat CMF alone, especially for depth in noise or at distance?
Sam: Yes, with consistently lower distance errors. The paper notes a small error limit at high signal quality, from speaker size and measurement limits. Still, adding visuals improves depth reliably, guiding direction without fixing powers.
Alex: What about uncertainty in camera depth?
Sam: They project object detections to a fixed plane using room references, then rely on angular costs—like compass bearings. Rays line up at zero cost, skipping depth details.
Alex: Smart—pure direction strengthens the mic depth. Works even with thousands of grid points.
Sam: The paper presents it as a proof-of-concept with clear accuracy gains on real data. It could extend to radar or other sensor arrays.
Alex: A practical advance: modular, no retraining, relying on physics and geometry for new setups.
Sam: Thanks for joining us on ResearchPod.