Towards a knowledge-enhanced single-cell foundation model

Recent single-cell foundation models (scFMs) have focused on scaling up the size of transcriptomic pretraining data to improve representation quality. However, this approach faces diminishing returns and high computational costs. This paper investigates whether incorporating structured biological knowledge—specifically cell-level text annotations and gene-level regulatory networks—can serve as a more efficient alternative to simply increasing the volume of raw transcriptomic data.

The authors developed scKITE (single-cell Knowledge-Integrated Transformer), which employs a two-stage pretraining framework. In the first stage, a Transformer encoder is pretrained using standard masked-expression reconstruction. In the second stage, the model is further trained using two lightweight auxiliary decoders that provide supervision through cell-level natural-language annotations and gene-level regulon predictions. These decoders are discarded after pretraining, leaving a general-purpose encoder that has been enriched with biological context. The authors systematically compared this knowledge-enhanced approach against standard data-scaling strategies across various pretraining corpus sizes.

scKITE demonstrates that biological knowledge can significantly shift the data-scaling curve, allowing for high performance with substantially fewer samples. Using only 179,067 pretraining samples—less than 0.5% of the data used by models like Geneformer or scGPT—scKITE consistently outperformed these larger models across diverse downstream tasks, including cell type annotation, batch integration, and perturbation-response prediction. The authors show that the auxiliary decoders effectively guide the encoder to prioritize biologically relevant genes, such as cell-type markers and transcription factor targets, leading to more robust and interpretable representations.