TypeProbe: Recovering Type Representations from Hidden States of Pre-trained Code Models

While modern code Large Language Models (LLMs) demonstrate high proficiency in code generation, it remains unclear whether they internalize formal type semantics or merely rely on superficial lexical patterns. This paper investigates whether these models encode type information within their hidden states, whether this information is shared across different programming languages (Java and Python), and if these representations are robust to adversarial lexical interference.

To test these questions, the authors developed a dataset of 90,000 operationally equivalent code snippets in Java and Python. They employed linear probing—a standard interpretability technique—to extract type information from the residual streams of two decoder-only models: SantaCoder-1.1B and CodeLlama-7B. The probing tasks included predicting function signatures, argument types, and result types. To ensure the models were not simply memorizing variable names, the authors introduced an adversarial partition where identifier names were systematically randomized to contradict their actual types.

The researchers found that both models develop linearly decodable type representations that transfer effectively across languages. Notably, probes trained on untyped Python code could still successfully recover type information in Java, suggesting that the models learn a latent, cross-lingual type manifold. While the models show strong performance in standard settings, their internal type representations are only partially robust to adversarial renaming. When variable names are misleading, the models' ability to correctly identify types degrades, indicating that lexical cues still exert significant influence over the internal state alongside formal type semantics.

These findings suggest that code LLMs are not just pattern-matching on tokens but are building internal representations that mirror formal programming concepts. This provides a theoretical foundation for why techniques like type-constrained decoding work and suggests that future interpretability efforts could use these internal representations to improve model reliability and steer code generation more effectively.