C. E. Shannon
8 min
Shannon introduces a mathematical definition of information based on the logarithmic reduction of uncertainty. By measuring information in binary digits (bits), he provides a universal language for quantifying the content of messages and the capability of transmission channels. A key insight is that information is not an inherent property of the message itself, but a measure of the freedom of choice in selecting a message from a set of possibilities.
The paper establishes that for any communication channel, there exists a specific 'channel capacity'—the maximum rate at which information can be transmitted with an arbitrarily small error probability. Shannon proves that as long as the rate of an information source is less than the channel capacity, it is possible to encode the source into signal sequences that can be reconstructed with near-perfect fidelity, regardless of noise, provided the encoding is sufficiently complex.
Shannon identifies entropy as the average amount of information produced by a source per symbol. High-entropy sources are inherently unpredictable, while low-entropy sources contain structure or 'redundancy' (like the predictable patterns in English prose). Redundancy can be leveraged to increase the reliability of transmission; although it reduces the effective transmission rate, it provides a buffer against noise, allowing the receiver to reconstruct corrupted parts of a message based on the inherent structure of the language.
This work transformed communication from an engineering practice of adjusting circuits into a rigorous mathematical science. By separating the transmission problem (reproducing signals) from the semantic problem (meaning), Shannon provided the foundation for modern digital systems, data compression, and error-correcting codes, enabling the efficient and reliable transmission of digital data across the globe.
Alex: His artificial English examples make that visible. As he preserves letter frequencies, then neighbouring-letter patterns, then word transitions, the generated text increasingly resembles English.
Sam: Those examples show the structure the theory can exploit. The central mathematical step concerns long messages. Under the source assumptions he uses, almost all the probability concentrates on messages whose individual probabilities are close on a logarithmic scale.
Alex: Why does that concentration help an encoder? The unlikely messages still have to be transmitted.
Sam: The likely messages can receive efficient descriptions, while rare messages use longer descriptions. In the noiseless proof, they are not simply discarded. As message blocks grow, their contribution to average transmission time becomes small, and the average description length approaches the source’s entropy.
Alex: That establishes the compression side: predictable structure reduces the average description length. Where does noise enter the accounting?
Sam: At the receiver, through whatever uncertainty remains about what was sent. Shannon calls that equivocation: average ambiguity after observing the received signal. Information transmitted is the uncertainty originally present, minus that remaining uncertainty.
Alex: Why not just subtract the fraction of corrupted symbols? That sounds like a more direct measure of what survived.
Sam: Shannon tests precisely that intuition. His example sends equally likely binary symbols at a thousand symbols per second. On average, one in a hundred arrives incorrectly.
Alex: Simple subtraction would credit the channel with nine hundred and ninety bits per second. But that treats the receiver as though it knew which symbols were wrong.
Sam: Accounting for that missing knowledge gives about nine hundred and nineteen bits per second instead. The uncertainty includes where the errors occurred. Shannon makes this operational with an imagined observer who sees both the sent and received messages.
Alex: What does that observer establish beyond a different way of describing errors?
Sam: The observer sends correction information over an additional channel. Shannon proves that the equivocation specifies the capacity needed to reduce the remaining errors arbitrarily far. Capacity means the maximum information rate a channel supports, so the ambiguity has a concrete communication cost.
Alex: We have a measure of information getting through. How does that become a guarantee that reliable transmission is possible?
Sam: Shannon maximizes that rate over possible inputs to define the noisy channel’s capacity. Then he proves that rates below capacity can support arbitrarily rare errors with suitable encoding. Above capacity, some uncertainty necessarily remains.
Alex: That is stronger than saying repetition helps. What makes the existence proof work?
Sam: He considers long signals and averages over a large collection of possible codes. Below capacity, the average error can be made arbitrarily small. Therefore at least one code must perform that well; the argument does not need to construct it explicitly.
Alex: So the receiver needs distinguishable possibilities, not a separate repetition of every symbol. The redundancy has to keep noise from making a signal look like another allowed message.
Sam: That is Shannon’s interpretation. Redundancy must suit the noise structure. He also gives a concrete code, using Hamming’s method, for a specially restricted channel: each seven-symbol block is either unchanged or contains a single error.
Alex: How much useful information survives in that example?
Sam: Four symbols carry the message; the others provide checks that locate the error. Under Shannon’s specified probabilities for those block outcomes, the code reaches capacity and corrects completely. But that restricted noise model is essential, not a general guarantee for every damaged block.
Alex: The general theorem still sounds more useful as a target than as an engineering recipe. Does Shannon acknowledge that gap?
Sam: Explicitly. He says following the proof to obtain nearly ideal coding is generally impractical. Approaching the limit also generally requires delay: longer blocks improve statistical discrimination, but the receiver must wait before deciding what was sent.
Alex: That covers messages made of discrete symbols. Speech and other continuously varying signals cannot usually be reproduced with infinite precision. Does the theory change its question there?
Sam: It asks how much information is needed for a chosen fidelity, meaning a specified quality of reconstruction. Exact continuous recovery generally demands infinite capacity. Approximate recovery instead requires a criterion, such as average squared error or the frequency of misunderstood words.
Alex: So the destination matters through what counts as an acceptable reconstruction. A waveform error and a misunderstood word are different judgments.
Sam: Shannon defines the least rate needed for that chosen quality, and connects it to channel capacity. He also derives continuous-channel limits under stated bandwidth, noise, and power constraints. These are conditional results, not one universal speed for every physical channel.
Alex: Who should read the full document, and where should they begin?
Sam: Researchers working on compression or reliable transmission should start with the fundamental theorems for noiseless and noisy channels, then their discussion sections. Read the noisy-channel discussion especially closely: it separates the existence of good codes from practical ways to find them.
Alex: And for researchers whose main problem is reconstructing a continuous signal?
Sam: Start with “Fidelity Evaluation Functions” and the following section on a source’s rate relative to fidelity. Shannon is explicit that his continuous treatment does not pursue maximum generality or full mathematical rigor. For everyone else, remember this: noise sets a limit on reliable speed, not an unavoidable error rate.
Alex: Keep that distinction in mind when judging a communication system.