Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs | Yoonhyung Park et al. | ResearchPod

Transcript: Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a framework called Splash — and the puzzle it's trying to solve is this: how do you teach a compact robot to understand tactile sensations — like whether an object is slippery or firm — without making it forget how to see?

Alex: So this is about expanding a robot's senses without breaking its existing knowledge?

Sam: Exactly. When you train a small AI model to learn a new skill, it often overwrites what it already knows. This is a well-documented problem in machine learning called "catastrophic forgetting." It's like trying to learn a new language by erasing your memory of your native one.

Alex: So what's the mechanism they use to keep that existing knowledge safe?

Sam: They use a technique called parameter isolation. Think of the AI's brain as a library. Instead of rewriting every book on the shelves, they identify which books are critical for general knowledge and lock them away. The remaining sections — the ones that are rarely used — are where they do the new work, filling them with tactile data.

Alex: So the critical parts stay frozen, and the quiet, underused parts get repurposed?

Sam: Precisely. They call this a "dormant subspace." By only updating these specific, less-essential connections, they sidestep catastrophic forgetting. It's a way to expand the model's range without needing a bigger, more expensive system.

Alex: How do they actually decide which parts are dormant and which are critical?

Sam: They run a sample of visual data through the model and measure how much each connection contributes to the robot's vision and language skills. If a connection is weak or rarely active during visual tasks, it gets marked as dormant. Think of it like a performance review for every single connection in the AI's brain — the ones that aren't pulling their weight are the ones they feel safe reassigning.

Alex: Are there guardrails to make sure they don't accidentally reassign something important?

Sam: Yes. They keep the very first and very last parts of the model completely frozen. Those layers handle the most fundamental understanding — recognising that something is an image at all, and producing a final answer — so they're off-limits. Everything in between is fair game, as long as it passes the dormancy test.

Alex: And does being that selective actually produce better results than just retraining the whole model?

Sam: The evidence suggests it does. By being surgical about which connections they update, they achieved stronger tactile reasoning while actually improving the robot's general-purpose vision skills at the same time. The key insight is that targeted updates outperform wholesale changes — because when you change everything at once, the old knowledge has nowhere to hide.

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs

Yoonhyung Park, Minji Kim, Sungwon Moon, Jiyoung Lee

Jul 1, 2026·3:29·arxiv·Original Paper

SplashCatastrophic ForgettingVisuo-Tactile LearningParameter Subspace PartitioningMultimodal Large Language Models (MLLMs)Dormant Parameter Subspace

Listen

3 min

Summary

Research Question

Integrating tactile sensing into compact Multimodal Large Language Models (MLLMs) is essential for robotic dexterity, yet it presents a significant challenge: the limited parameter budget of small models (sMLLMs) makes them prone to catastrophic forgetting. When these models are fine-tuned to incorporate tactile data, they often lose their pretrained visual-language reasoning capabilities. This paper asks: how can we effectively align tactile signals with sMLLMs while preserving their original general-purpose intelligence?

Approach

The authors introduce Splash, a mask-isolated tactile alignment learning framework. Instead of updating the entire model, Splash identifies a dormant subspace—a subset of parameters that contribute minimally to pretrained vision-language tasks—using a visual-relative importance metric. During training, the model partitions its parameters into a frozen critical subspace (which acts as a stable anchor for visual knowledge) and a trainable dormant subspace (which is updated to internalize tactile alignment). This approach allows for unified, single-stage training without the need for complex multi-stage alignment or additional inference-heavy adapters.

Main Findings

Splash achieves state-of-the-art performance on three major visuo-tactile benchmarks (SSVTP, TVL, and TacQuad), consistently outperforming larger 7B-parameter models despite using compact 1B and 3B backbones. Crucially, the framework demonstrates superior preservation of general-purpose vision-language capabilities compared to traditional parameter-efficient fine-tuning methods like LoRA or (IA)3. By confining tactile adaptation to the dormant subspace, Splash avoids the trade-off between sensory expansion and reasoning performance, ensuring that the model remains capable of complex multimodal tasks after learning to perceive physical material properties like friction and compliance.

Why It Matters

For embodied robotics, deploying models on resource-constrained edge hardware is a primary requirement. Splash provides a practical, efficient path to equipping small robots with tactile intelligence. By eliminating the need for additional inference overhead and preventing the degradation of pretrained reasoning, this framework enables more capable and reliable robotic perception in real-world environments.

Key Terms and Definitions

Dormant Subspace — A subset of model parameters identified as having minimal impact on pretrained vision-language tasks, which is repurposed for learning new tactile modalities.
Catastrophic Forgetting — The phenomenon where a model loses its previously learned knowledge (e.g., visual reasoning) when it is updated to learn a new task (e.g., tactile sensing).
Visuo-Tactile-Language (VTL) Alignment — The process of mapping tactile sensory inputs into the same latent space as visual and linguistic representations to enable cross-modal reasoning.
Mask-isolated Adaptation — A training strategy that uses a binary mask to restrict gradient updates to specific parameter subsets, ensuring that critical weights remain frozen.
Tactile Perceptual Aliasing — A challenge where different materials produce similar tactile imprints, making it difficult for models to distinguish between them based on touch alone.

Abstract

Touch supplies the physical grounding needed to perceive intrinsic material properties, such as friction and compliance, that vision alone often cannot resolve. Recent efforts for equipping multimodal LLMs with this tactile sense, however, expose a zero-sum trade-off: the limited parameter budget of compact models forces a choice between acquiring the new sensory modality and preserving the established vision-language reasoning. We present Splash, a mask-isolated tactile alignment learning framework for MLLMs. Splash quantifies the significance of each pretrained parameter, and partitions the parameter space into a dormant and critical subspace. While the frozen critical subspace acts as a stable anchor to safeguard general visual knowledge, Splash updates the isolated dormant subspace to internalize tactile alignment towards LLMs. This selective, non-destructive expansion effectively prevents catastrophic forgetting and ensures non-destructive modality expansion. Extensive experiments show that Splash effectively achieves tactile reasoning without additional inference overhead in the LLM part, demonstrating state-of-the-art performance on visuo-tactile benchmarks, including SSVTP, TVL, and TacQuad, while preserving its original general-purpose capabilities.

Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs: Research Paper Podcast