Does Bielik Know What It Doesn’t Know? Activation Dispersion Separates Entity Familiarity from Factual Reliability Across Model Scale

This paper investigates whether large language models (LLMs) possess an internal, pre-generation awareness of whether they are familiar with an entity mentioned in a prompt. Using four Polish-language Bielik models ranging from 1.5B to 11B parameters, the author examines whether activation patterns in the model's Multi-Layer Perceptron (MLP) layers can distinguish between well-known, obscure-but-real, and fabricated entities.

Rather than relying on complex training or multiple inference passes, the study employs two unsupervised, single-forward-pass statistics—Inverse Participation Ratio (IPR) and spectral entropy—to measure the dispersion of post-SwiGLU MLP activations. These metrics are evaluated at the prompt point (the end of the input sentence) to see if the model's internal state reflects entity familiarity before it begins to generate an answer. The study also compares these internal signals against behavioral factual reliability, measured by how often the models correctly answer questions about known entities, and audits the models for their tendency to refuse or hedge when faced with unknown information.

• Robust Detection: The dispersion measures successfully separate known from fabricated entities with high accuracy (AUROC 0.95–1.00) across all tested model sizes and entity domains (athletes, cities, writers, and musicians).
• Dissociated Scaling: The internal familiarity signal reaches a performance ceiling at the smallest model size (1.5B), whereas the model's ability to provide factually correct answers continues to improve significantly as the model scale increases.
• Alignment Gap: Despite having a clear internal signal that distinguishes familiar from unfamiliar entities, the models almost never abstain from answering, providing direct assertions for 99.88% of queries. This highlights a significant gap between internal knowledge and behavioral alignment.
• Limited Factual Detection: While the dispersion metrics are excellent at detecting familiarity, they are much less effective at distinguishing between correct answers and hallucinations for entities the model is already familiar with.