The Hessian matrix tells a peak from a valley or a saddle
At the top of a hill, the bottom of a bowl and the middle of a mountain pass, the ground is momentarily flat. Slope alone cannot tell these apart. The Hessian matrix, a grid of second derivatives, measures how the surface curves in every direction and so reveals which kind of flat spot you are standing on.
For a function of several variables, the Hessian collects all its second-order partial derivatives into a square array describing local curvature. The German mathematician Ludwig Otto Hesse developed it in the 19th century, calling it a functional determinant; the modern name honours him. When those second derivatives are continuous, the matrix is symmetric.
Its best-known job is the second-derivative test. At a critical point where the gradient vanishes, a positive-definite Hessian signals a local minimum, a negative-definite one a local maximum, and mixed signs a saddle. If it is only semidefinite, the test says nothing. With two variables the determinant does the work, since it is the product of the two eigenvalues: positive means both share a sign, negative means they differ. Geometrically, the eigenvalues are the principal curvatures, the eigenvectors point along the principal directions, and the determinant equals the Gaussian curvature. Morse theory and catastrophe theory rely on it to classify critical points.
It also surfaces in geometry of curves. On a plane projective curve, a smooth point is an inflection point precisely when the determinant of the Hessian there equals zero, and Bézout's theorem then limits a cubic curve to at most 9 inflection points.
In optimisation, Newton-type methods use the Hessian because it supplies the quadratic term of a function's local Taylor expansion. The catch is size: storing it becomes infeasible for functions with huge numbers of parameters, such as neural network loss functions. Truncated-Newton and quasi-Newton algorithms get around this by approximating it, and one of the most popular quasi-Newton methods is BFGS.
Source: Hessian matrix
