How do large language models learn to predict the next word?

But what is cross-entropy? | Compression is Intelligence Part 2 · 33:51 · Video: 3Blue1Brown · Watch at the source

In short: This video explores the mathematical origins of the loss function used to train large language models. It explains the concept of cross-entropy, providing a foundational look at how these systems measure and improve their predictive accuracy.

The video investigates the mechanics behind training large language models, specifically focusing on the role of cross-entropy as a loss function. By breaking down the mathematical principles that guide model learning, it offers a clearer view of how machine learning systems refine their outputs.

The presentation features custom animations created with the manim library, including visualizations of a 3D black-box model, chess game scenarios, and Lagrange multipliers. These tools help illustrate the complex processes that allow models to minimize error and improve performance.

Opens the Intellect+ feed with this card first. Swipe up for the next one.

Sources

To cite this card, link to https://intellectplus.co.uk/app/science-but-what-is-cross-entropy-compression-is-intelligence-part-2.

How we source, check and correct cards: Editorial standards.

General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.

Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?

Related

Finding something worth knowing…