Likelihood is not probability, and confusing them can be disastrous
Toss a coin twice and get two heads. If the coin is fair, that outcome had a 0.25 chance, so the likelihood of fairness given those tosses is 0.25. It is tempting to conclude there is a 25% chance the coin is fair. That leap is a classic error, the same one behind the prosecutor's fallacy.
A likelihood function scores how well different versions of a statistical model explain the data actually observed. The models usually differ by a parameter, often written as theta, such as the probability that a coin lands heads. The same formula can be read two ways. Hold the parameter fixed and vary the data, and it is a probability distribution over outcomes. Hold the data fixed and vary the parameter, and it becomes the likelihood, a way of comparing candidate parameter values.
The coin shows the difference clearly. Let the heads probability range anywhere from 0 to 1. Two heads in a row has probability 0.25 if the coin is fair, but only 0.09 if the coin lands heads 30% of the time, so the data favour fairness over that bias. Plot the likelihood across every possible value and the area under the curve comes to one third, not one. Likelihoods are not probabilities of the parameter and need not add up to 1.
Turning a likelihood into the probability that a parameter is correct requires extra information: how plausible each value was before seeing the data. That is the job of Bayes' theorem, which produces what Bayesian statisticians call the posterior probability. Without that step, reading a likelihood as the chance a hypothesis is true can have serious consequences, as when a court mistakes the rarity of evidence for the probability of innocence.
In the rival frequentist approach, the parameter is treated as a fixed unknown rather than a random quantity. Its workhorse, maximum likelihood estimation, simply picks the parameter value that makes the observed data most probable, and uses a quantity called the Fisher information to judge how precise that estimate is.
Source: Likelihood function
