Mutual information measures how much knowing one thing tells you about another
In short: Knowing the result of one thing can reduce your uncertainty about another. Information theory gives this idea an exact measure called mutual information, defined by Claude Shannon and measured in bits. It works where simple correlation fails, because it captures any kind of dependence, not only straight-line relationships. Here is how it works and why it matters.
Mutual information is a concept from probability theory and information theory that measures how strongly two random variables depend on each other. It captures how much observing one variable tells you about the other, and the result can be expressed in bits, nats or hartleys depending on the logarithm used. Claude Shannon defined and analysed the idea in his landmark 1948 work on communication, although he never gave it this name. The term was introduced later by Robert Fano, and the measure is also known as information gain.
The core idea is that mutual information compares the joint distribution of two variables with what it would be if they were independent, meaning the product of their separate distributions. Formally, it is the Kullback–Leibler divergence between the joint distribution and that product distribution. It equals zero exactly when the two variables are independent, and it is never negative. It can also be written as the reduction in uncertainty about one variable once the other is known, which links it to entropy and conditional entropy. Mutual information is symmetric, so the information X gives about Y equals the information Y gives about X.
A key advantage over the correlation coefficient is generality. Correlation only captures linear relationships between numbers, while mutual information is not limited to real-valued variables or to linear dependence. It asks how different the joint distribution is from the independent case, so it can detect many more kinds of association. The measure applies to discrete variables through probability mass functions and to continuous variables through probability density functions, where sums become integrals.
Estimating mutual information from data is harder than defining it. When samples from a joint distribution are available, Bayesian methods can be used to estimate it, and researchers have developed and extended such approaches over time. Variations, including normalised versions and extensions to more than two variables, have been proposed for different needs. Mutual information remains a foundational tool, used wherever it matters how much one measurement reveals about another.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Mutual information
To cite this card, link to https://intellectplus.co.uk/app/science-mutual-information.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Algebra · More in Science · Latest · A–Z · All topics