CARBONCLIP: ENHANCE CARBON PREDICTION FROM SATELLITE IMAGERY VIA INTEGRATED STREET-VIEW SEMANTICS AND TEMPORAL CONTEXT TRAINING

CarbonCLIP addresses the challenge of predicting urban carbon emissions using only satellite imagery, which often lacks the fine-grained human-centric details necessary for accurate modeling. The authors propose a task-oriented multimodal distillation framework that bridges the gap between top-down satellite views and ground-level urban activities. During the pretraining phase, the model uses two primary branches: a spatial branch that aligns satellite imagery with textual descriptions of street-level scenes (generated by Large Multimodal Models), and a temporal branch that encodes monthly emission variations using sinusoidal embeddings. This process effectively transfers multimodal knowledge into a satellite-centered representation.

The primary innovation of CarbonCLIP is its ability to perform multimodal-informed inference while requiring only satellite imagery at deployment. By using LMMs to convert street-view imagery into structured textual descriptions, the framework bypasses the need for complex pixel-level geometric alignment between different data sources. Furthermore, the inclusion of a month-level temporal encoder allows the model to account for seasonal energy consumption cycles and phenological changes, which are critical for carbon emission patterns but often ignored in static visual models.

Existing methods for urban carbon monitoring often rely on heterogeneous data sources—such as point-of-interest data, transportation records, or socioeconomic statistics—that are difficult to harmonize and often unavailable in developing regions. CarbonCLIP provides a scalable solution by distilling rich contextual knowledge into a single-modality model. This allows researchers and urban planners to leverage the global coverage of satellite imagery while maintaining the predictive accuracy typically associated with multi-source data fusion, making it a robust tool for sustainable urban planning in data-scarce environments.