Why some mathematical models cannot be solved in a single step

Why some mathematical models cannot be solved in a single step
Image: Intellect (generated)

In short: In linear regression, we can often find the perfect answer instantly. But when relationships become non-linear, the math breaks. To find the truth, we must abandon the search for a direct solution and instead embrace a process of endless, iterative refinement.

In standard linear least squares, the relationship between variables is straightforward, and the objective function is quadratic. This allows for a direct, closed-form solution. However, non-linear least squares is used when a model is non-linear in its unknown parameters. This complexity arises because the gradient equations—the mathematical tools used to find the minimum error—do not typically have a closed solution.

To navigate this, mathematicians use an iterative approach. The process begins by choosing initial values for the parameters. The method then approximates the complex, non-linear model using a linear one, specifically through a first-order Taylor polynomial expansion. This is the core of the Gauss–Newton algorithm. By treating the model as a series of linear approximations, we can calculate a 'shift vector'—a set of increments that nudges our parameters closer to the ideal fit.

During each iteration, a Jacobian matrix is calculated. This matrix is dynamic; because it depends on both the independent variables and the parameters, it changes every time the parameters are updated. We solve the resulting 'normal equations' to find the necessary adjustments. This cycle of approximating, calculating the Jacobian, and refining the parameters continues until the sum of the squared residuals—the difference between observed and predicted values—is minimized.

This technique is vital in economic theory for complex models like probit regression, threshold regression, and Box–Cox transformed regressors. In cases where observations are not equally reliable, researchers use a weighted sum of squares, where each data point is weighted by the reciprocal of its error variance, ensuring that more certain measurements have a greater influence on the final model.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/science-non-linear-least-squares.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…