Why the most surprising news contains the most information, according to Claude Shannon.

Why the most surprising news contains the most information, according to Claude Shannon.
Image: Intellect (generated)

In short: In information theory, entropy measures uncertainty. The more surprising an event, the more information it carries. If an event is certain, it provides zero information. This mathematical framework, introduced by Claude Shannon in 1948, defines the absolute limits of how efficiently we can compress and transmit data.

At its core, information entropy quantifies the average level of uncertainty associated with a random variable. Claude Shannon, who introduced the concept in his 1948 paper 'A Mathematical Theory of Communication', defined entropy as the expected value of an event's 'surprisal'. Surprisal is calculated using a logarithmic function of the event's probability; as an event becomes less likely, its informational value increases. Conversely, if an outcome is guaranteed, the surprise—and thus the information—is zero.

This concept provides a rigorous way to measure data. For instance, rolling a die has higher entropy than flipping a coin because the die has more potential outcomes, each with a lower probability. Shannon proved that entropy represents the absolute mathematical limit for lossless data compression. If a source of data is predictable—like English text, where certain letters and combinations like 'th' appear far more frequently than others—it has lower entropy and can be compressed more efficiently than a truly random sequence.

The mathematical formula for entropy, H(X), is the sum of probability-weighted log probabilities. While base 2 is standard in computing, yielding units called 'bits' or 'shannons', other bases like Euler's number (e) or 10 are used in different applications, resulting in 'nats' or 'bans'. This framework is not just for communication; it is formally analogous to entropy in statistical thermodynamics, where Gibbs's formula mirrors Shannon's. Today, these principles remain foundational to fields ranging from machine learning to combinatorics, defining the very limits of how we encode and understand the world's data.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/science-entropy-information-theory.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…