A word is defined by the company it keeps
In short: How does a machine understand the nuance of human language? By treating words like physical objects in a multi-dimensional map. Through word embeddings, computers translate text into mathematical vectors, ensuring that words with similar meanings sit closer together in a digital landscape of pure logic.
The foundation of word embeddings lies in distributional semantics, a method that quantifies meaning based on linguistic patterns. This approach stems from a 1957 observation by John Rupert Firth, who proposed that a word's identity is shaped by its surrounding context. In modern natural language processing, this translates to mapping words into real-valued vectors. When these vectors are positioned in a high-dimensional space, words with similar meanings—like 'coffee' and 'tea'—cluster together, while unrelated terms are pushed apart.
The evolution of this technology moved from sparse, high-dimensional models to more efficient 'distributed representations.' In 2000, Bengio et al. pioneered neural probabilistic language models to solve the 'curse of dimensionality.' This paved the way for the 2013 release of word2vec by Tomas Mikolov’s team at Google, which significantly accelerated training speeds and moved the technology from niche research into mainstream practical application.
Other notable developments include fastText, which breaks words into character n-grams, and contextually-meaningful models like BERT, which assign unique embeddings to every specific occurrence of a word to better handle multiple meanings. However, these mathematical maps are not neutral. Because they learn from human-generated data, they often inherit and amplify societal prejudices. A 2016 study by Bolukbasi et al. demonstrated that word2vec models trained on news text produced skewed associations regarding gender and race. For instance, the model might link men to professional roles while associating women with domestic ones. Researchers warn that without careful oversight, these embeddings can perpetuate and even intensify existing biases, making the study of their underlying mechanics a critical task for ethical AI development.
Opens the Intellect+ feed with this card first. Swipe up for the next one.
Sources
- Summary by Intellect · source: Word embedding
To cite this card, link to https://intellectplus.co.uk/app/science-word-embedding.
How we source, check and correct cards: Editorial standards.
General information, not professional advice. Spotted an error? Flag the card or see corrections and complaints.
Intellect Plus is a visual educational discovery platform for exploring fascinating facts and ideas. Brain Food, Not Brain Rot. What is Intellect Plus?
Related
More in Scientists, inventors and mathematicians · More in Science · Latest · A–Z · All topics
