How do we minimize complex functions in deep learning? It starts with the Hessian.

Ideas & Philosophy · Published 3 October 2026 Swipe more like this →

In short: Optimization is the engine behind deep learning, but navigating functions with many variables requires more than simple calculus. To find the minimum, we rely on Taylor series approximations, utilizing gradients for linear changes and the Hessian matrix for quadratic behavior. Understanding these tools is essential for mastering modern machine learning.

In his 2018 lecture on matrix methods, Professor Gilbert Strang explains that optimization is the fundamental algorithm powering deep learning. When dealing with a function of a single variable, we often use a Taylor series approximation. However, deep learning problems involve functions of many variables, requiring a shift from scalar calculus to vector-based mathematics.

For a function with n variables, the linear term of the Taylor series is represented by the gradient, a vector of partial derivatives. To capture the quadratic term, we use the Hessian matrix, denoted as H. This matrix contains second-order partial derivatives, where the entry Hjk corresponds to the derivative of the function with respect to variables xj and xk. A crucial property of the Hessian is its symmetry.

The computational cost of these methods is significant. While the gradient requires n derivatives, the Hessian involves approximately 1/2 n squared derivatives. For large-scale deep learning, calculating the full Hessian is often out of reach. Instead, techniques like backpropagation and automatic differentiation are used to speed up derivative computations, allowing us to navigate complex landscapes where n is large.

Beyond optimization of a single function, Strang highlights a parallel structure when solving systems of equations. When working with n functions of n variables, we look for the Jacobian matrix, which represents the first derivatives of the system. This framework allows researchers to approach the challenge of finding where a vector function equals zero, mirroring the logic used in gradient-based optimization.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

• Why second derivatives are the secret language of machine learning optimization
• How machines navigate high-dimensional landscapes to find optimal solutions
• How to efficiently calculate derivatives for complex systems modeled by ordinary differential equations.
• How to calculate gradients efficiently for massive engineering optimization problems
• Why the derivative is more than just a slope on a graph
• Mastering the DC-to-AC conversion process through Fourier analysis and bridge inverter control
• How dual numbers allow computers to calculate derivatives automatically and efficiently
• Why minimizing different norms changes the point you find on a line
• How bond mathematics reveals the hidden signals behind economic recessions and interest rates.
• Can you take the derivative of a function? Meet the calculus of variations.
• How mathematical models shape the modern landscape of financial markets and trading strategies
• Beyond the basics: How metric spaces bridge advanced mathematical disciplines

More in Mathematics · More in Ideas & Philosophy · Latest · A–Z · All topics