Quantifying information stored in synaptic connections rather than in firing activities of neural networks

While neural coding is traditionally studied through the lens of neuronal firing activity, the storage of information in synaptic weights remains less formally understood. This paper addresses this gap by developing an analytical framework to quantify the Shannon mutual information between data patterns and synaptic connections in a densely connected Hebbian network. The authors model data patterns as multivariate log-normal distributions—a choice supported by the heavy-tailed statistics observed in biological neural populations—and derive closed-form approximations for the information encoded by individual synapses and arbitrary ensembles of synaptic connections.

• Distributed Coding: The study demonstrates that information is stored collectively across the network. As the number of stored patterns increases, the information capacity of individual synaptic connections decreases, confirming that synaptic weights simultaneously encode partial information about multiple patterns.
• Synergistic Encoding: A central discovery is that the information encoded jointly by a group of synapses exceeds the sum of the information encoded by each synapse individually. This synergy indicates that synaptic connections do not act as independent storage units but rather as a collective ensemble where interactions between connections enhance total storage capacity.
• Scaling Laws: The authors find that the information capacity of synaptic ensembles scales approximately with the 1.1 power of their size. Furthermore, they estimate that the number of patterns a network can store scales with the 1.2 power of the network size (number of neurons), providing a new perspective on the storage capacity of Hebbian-based associative memory.

This framework provides a mathematically tractable way to interpret synaptic connectivity as a storage medium, offering a complementary perspective to firing-rate-based coding. By quantifying how information is distributed and synergistic at the synaptic level, the study offers insights into how neural networks might optimize memory storage and suggests that pruning strategies in artificial neural networks should account for the collective, context-dependent nature of synaptic contributions.