How a language model picks the next word
In short: Text becomes tokens, tokens become numbers, layers weigh the context and score every possible next token.
1. The text so far is split into tokens: words or pieces of words. 2. Each token is turned into a list of numbers called an embedding. 3. Attention layers weigh how the tokens in the context relate to one another. 4. Softmax turns the model's scores into a probability for every possible next token. 5. A token is chosen and added to the text, and the process repeats.
Sources: Wikipedia, “Large language model (Wikipedia)” (CC BY-SA 4.0); Wikipedia, “Softmax function (Wikipedia)” (CC BY-SA 4.0). A simplified picture; the model calculates, it does not think like a person.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Large language model
- Also: Softmax function
- Also: hai.stanford.edu — What is a Large Language Model (LLM)? | Stanford HAI
- Also: www.britannica.com — Large language model | Definition, History, & Facts | Britannica
- Also: The Met — Model · CC0
To cite this card, link to https://intellectplus.co.uk/app/how-a-language-model-picks-the-next-word.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
