Shannon made uncertainty itself a measurable quantity
Information theory quantifies how messages are stored and sent under noise. Claude Shannon formalised the field in the 1940s after earlier Bell Labs work by Harry Nyquist and Ralph Hartley. A fair coin you have not seen carries one bit of entropy; seeing the face drops that entropy to zero.
Shannon’s July and October 1948 Bell System Technical Journal paper A Mathematical Theory of Communication framed communication as reproducing, exactly or approximately, a message chosen elsewhere. Historian James Gleick ranked it the most important development of 1948—deeper even than the transistor—and Shannon became known as the father of information theory. He had sketched ideas to Vannevar Bush as early as 1939. Work at Bell Labs was largely done by the end of 1944.
Nyquist’s 1924 telegraph-speed paper related intelligence transmission rate to the log of voltage choices; Hartley’s 1928 Transmission of Information set H = n log S for S symbols over n steps, with the decimal digit later called a hartley. Thermodynamics supplied math for unequal probabilities via Boltzmann and Gibbs; Rolf Landauer later linked information and thermodynamic entropy. Alan Turing used related statistics in 1940 against Enigma.
Entropy measures average surprisal of a source’s symbols; rarer symbols carry more self-information. Mutual information gauges dependence between variables and, in the long-block limit, equals the maximum reliable rate through a noisy channel—the channel capacity of the noisy-channel coding theorem. Units depend on the log base: bits (shannons) for log₂, nats for natural logs, decimal digits for common logs. Equiprobable messages maximise entropy; a thousand known bits transmit nothing, while a thousand fair random bits transmit a thousand shannons.
Coding theory seeks explicit schemes near capacity: source codes compress (ZIP files), channel codes detect and correct errors (DSL). Cryptographic codes form a third class. The theory underwrote Voyager deep-space links, compact discs, mobile phones, and parts of the Internet and AI, and ranges into neurobiology, cryptography, and black-hole physics. Shannon’s promise was asymptotic: enough uses of a channel achieve rates up to capacity with vanishing error—finding practical codes took years.
Source: Information theory
