Kunal Pratap Singh, Ali Garjani, Rishubh Singh, Muhammad Uzair Khattak, Efe Tarhan, Jason Toskov, Andrei Atanov, Oguzhan Fatih Kar, Amir Zamir
8 min
This paper introduces Test-Space Training (TST), a framework for building performant vision models that are specifically optimized for a user's deployment environment. Rather than relying on massive, generalist pre-training on diverse internet-scale data, the authors propose a specialization paradigm where an agent collects multimodal sensory data (RGB, depth, surface normals, etc.) within its own test space and uses this data to conduct self-supervised cross-modal pre-training.
The authors hypothesize that biological agents learn efficient representations of their surroundings by observing time-locked, multimodal sensory streams. They operationalize this through TST, which utilizes multimodal masked modeling to learn these representations. To enhance performance, they also experiment with scaling the dictionary of modalities by distilling outputs from off-the-shelf pseudolabeling networks (e.g., segmenters, object detectors) as additional modalities. The framework is evaluated through transfer learning on downstream tasks including semantic segmentation, object detection, and image captioning, benchmarking against powerful internet-scale generalist models like DINOv2 and CLIP.
The key finding is that collecting and training on rich multimodal data from the test space can equal or outperform generalist models trained on internet-scale datasets. The authors demonstrate that scaling the number of modalities within a test space is often more effective than scaling unimodal training data across many different spaces. They also establish a clear specialization-generalization trade-off: while pre-training on test-space data provides superior performance for that specific environment, it comes at the expense of versatility in entirely different, unseen spaces. Furthermore, they show that TST can successfully adapt generalist foundation models to specific test environments, and that specialized models can sometimes be smaller and more computationally efficient than their generalist counterparts.
This paper challenges the prevailing "more is better" approach to data collection for foundation models. By demonstrating that deployment-specific, multimodal self-supervision can rival massive internet-scale pre-training, it offers a viable path for creating specialized AI agents that are highly capable in specific domains. This paradigm has significant potential for privacy-sensitive applications, as it suggests that models can achieve state-of-the-art performance by training locally on an agent's own data, potentially removing the need for massive data harvesting from external users.
Cross-modal learning, i.e., learning to predict one modality from another, is a fundamental mechanism for self-supervision via leveraging multimodality. Many practical applications, e.g., deploying a household robot, involve devices that are equipped with a rich set of sensors that enable multimodal sensing in their test environment. This presents an opportunity to apply cross-modal learning to the multimodal data sensed by these devices to learn representations. Findings in developmental psychology also suggest that biological agents leverage it to build an effective representation of their surroundings. To study this, we propose a controlled setup, where we restrict a user device to just a given test environment. It results in a specialization setup where we attempt to develop a performant model for this specific test environment. Under this setup, we develop Test-Space Training (TST), which performs multimodal data collection in the test environment and performs self-supervised pre-training on it. We evaluate these models on various downstream tasks in the same environment. Under this setup, we find various interesting insights, such as collecting rich multimodal data only from the test environment and leveraging cross-modal learning, we can achieve competitive results with generalist models (e.g., DINOv2 and CLIP) pre-trained on large-scale internet datasets. This enables an alternative scenario where the need for external Internet-scale datasets for pre-training models is reduced. We also present a set of analyses and ablations that raise intriguing points on substituting data with (multi)modality, and how varying pre-training data enables a tradeoff between a model's abilities to specialise to a test environment, and generalize to held-out spaces.
Alex: Training and testing in the same building sounds like an obvious advantage. How do they avoid simply training on the answers they'll later evaluate?
Sam: In the main setup, pre-training receives no task-specific labels from that building. Afterwards, they add a task-specific prediction head and fine-tune using labeled images from other spaces. Evaluation then returns to the deployment space. The model knows its visual environment, but hasn't received its downstream task answers.
Alex: That makes this a different question from performance on unseen buildings. How did they make the comparison with general-purpose models fair?
Sam: The comparison models receive the same labeled transfer data. Those include DINOv2, a strong image representation model, and 4M, a multimodal model. They also compare against task-specific systems. At evaluation, the trained models use color images alone, apart from a separate multimodal baseline.
Alex: Before borrowing anything from those existing networks, how far did ordinary sensor data get them?
Sam: They start with color, depth, surface normals describing surface orientation, and edges extracted through simple processing. This sensor-based version substantially improves over no pre-training and over their color-only masked-modeling approach. But it still trails the strongest internet-trained generalists on the semantic tasks they evaluate.
Alex: Then the headline depends on what gets added. What does borrowing from existing networks actually involve?
Sam: They run those networks on the locally collected images and use their outputs as extra prediction targets. The paper calls these pseudo-modalities. They include learned feature maps, object boxes, segmentation masks, and edges. This amounts to distilling existing models' knowledge into a model specialized for the deployment space.
Alex: So the local model doesn't need direct access to those networks' original training images. But their outside knowledge still enters through the targets.
Sam: The authors explicitly make that distinction. The best-performing version cannot claim no external access. There's another qualification: their pre-training also mixes in unlabeled color images from the external transfer set. An ablation finds local data alone useful, but that mixture performs best.
Alex: Let's put a number on the benefit. How much better was the locally trained model, and on what?
Sam: On ScanNet++, using eight real indoor scenes as the test space, they evaluate semantic segmentation: labeling image pixels by category. The score measures average overlap between predicted and reference regions across classes. Their enriched local model scores about thirty-five, versus about thirty-one for DINOv2.
Alex: That's a useful improvement in that setting. Does the evidence extend beyond that particular dataset and task?
Sam: They also test reconstructed indoor scenes in Replica and generated houses in ProcTHOR. The enriched model consistently beats the general-purpose baselines on segmentation and object detection in their main comparison. Captioning is less decisive: training from scratch is broadly comparable to the multimodal generalist, rather than clearly superior.
Alex: Could a single particularly strong teacher explain the result, rather than learning across many kinds of information?
Sam: Their ablations argue against that simple explanation. Adding individual targets to color-only training helps, but none matches the full set. Removing any single extra modality from the rich set has a relatively modest effect. However, replacing trained networks with untrained ones hurts, so useful learned knowledge matters.
Alex: That supports combining complementary signals, not merely generating arbitrary transformations. But perhaps enough images from other houses could deliver the same local benefit.
Sam: They test that in generated houses. Even pre-training on three thousand similar but different spaces doesn't match pre-training in the actual deployment space. Separately, increasing the richness of local modalities works better than increasing external color-only data in their tested setting.
Alex: Does that mean we can get local expertise and broad generalization together just by adding more houses?
Sam: Not in their fixed-budget experiment. They hold model size and total dataset size constant, replacing some local observations with observations from other houses. Performance improves on a hundred held-out houses, but falls in the original test space. That's a specialization–generalization tradeoff under those constraints, not a universal impossibility result.
Alex: For deployment, I'd also worry about changing furniture and the cost of training a separate model for every location.
Sam: Both matter. In generated environments, changing lighting and small-object placement reduces performance; continued local pre-training recovers it. And this isn't demonstrated as lightweight training on a household device. The reported pre-training used sixty-four H100 GPUs, with the base model taking roughly twelve hours.
Alex: Smaller specialist models might save inference costs later, as the authors suggest. But that doesn't erase the upfront work or establish a break-even point for every deployment.
Sam: That's how I'd read it. My judgment is that the controlled comparisons make a persuasive case for local specialization. But the evidence comes from bounded indoor settings, with substantial training resources. It doesn't show that a specialist should replace a generalist when the operating environment is unknown or keeps expanding.
Alex: Given that scope, who should read the full paper, and where should they start?
Sam: Researchers in embodied perception, multimodal learning, or deployment-specific adaptation should read it. Start with “Scaling Modalities in the Test Space” for the central comparison and its dependence on existing networks. Then read “Measuring Specialization with TST,” followed by the appendices on transfer-data mixing and computational resources. For everyone else: more views of the wider world aren't always better than richer measurements of the world you actually use.
Alex: Thanks for listening.