In regression, a factor can matter hugely alone yet vanish with company
Linear regression looks simple: draw the best straight-line relationship between an outcome and its causes. But the coefficient it reports for any one factor depends on what else is in the model. A variable with a strong relationship on its own can shrink to nearly nothing once another variable joins, and the reverse can happen too.
At heart, the method models an outcome as a weighted sum of explanatory variables. With one predictor it is simple regression; with several it is multiple regression. It was the first form of regression to be studied rigorously and used widely, because a model whose unknown parameters enter in straight-line fashion is simpler to estimate, and its statistical properties are simpler to derive. It serves two broad purposes: predicting outcomes for new cases, and explaining how much of the variation in an outcome can be traced to each factor.
The trouble lies in interpretation. In a multiple regression, each coefficient describes the expected change in the outcome for a one-unit change in that predictor while the others are held fixed, sometimes called its unique effect. Its marginal effect, measured by relating it to the outcome alone, can be very different. If another variable already carries the same information, the unique effect may be close to zero despite a large marginal one. If other variables soak up unrelated variation, a predictor's unique effect can grow even though its marginal effect is tiny.
What held fixed means also depends on the data. In a designed experiment, researchers genuinely set the predictor values. In an observational study, the phrase can only mean comparing subsets of data that happen to share a value. Critics argue that when predictors are correlated and were not assigned by design, multiple regression often fails to clarify how they relate to the outcome.
The usual fitting method is least squares, which minimises the sum of squared errors. Because squaring magnifies big misses, a few large outliers can pull the line towards them, so more robust cost functions are advised for messy data. Variations such as ridge and lasso regression add penalties to tame the coefficients. When errors follow a normal distribution, least squares gives the same answer as maximum likelihood.
Source: Linear regression
