Word embedding
A word embedding is a way of giving words a home in space — not the space of stars and galaxies, but a mathematical Coordinate system where meaning becomes geometry. Each word is assigned a vector, a list of numbers, and remarkably, words that mean similar things end up living near each other, like neighbors who never met but somehow share the same taste in furniture.
The Idea
Before embeddings, computers treated words as isolated symbols: "cat" and "kitten" were as unrelated as "cat" and "asteroid." Embeddings changed this by placing every word into a continuous vector space — often hundreds of dimensions — learned from how words actually behave in enormous piles of text. The guiding intuition, sometimes called the distributional hypothesis, is simple: a word is known by the company it keeps. Feed a Machine learning model enough sentences, and it discovers that "king" and "queen" orbit similar contextual neighborhoods, while "queen" and "toaster" do not.
What makes embeddings feel almost magical is that the resulting space captures relationships as directions. The famous demonstration — king minus man plus woman lands near queen — suggested that meaning itself has a kind of arithmetic, a Convergence of linguistics and vector algebra nobody quite expected to work so well.
How They're Learned
Early approaches like Word2vec and GloVe trained shallow networks to predict a word from its neighbors (or vice versa), nudging vectors closer together when words co-occurred often. This is a form of clustering in disguise — semantically related words drift into shared regions, forming loose neighborhoods without anyone hand-labeling them.
Later, embeddings became dynamic. Instead of one fixed vector per word, models built on Long short-term memory networks and, later, Attention mechanisms began generating embeddings that shift depending on sentence context — "bank" near a river differs from "bank" holding money. This shift was accelerated by the Attention Is All You Need paper, whose transformer architecture reshaped Neural machine translation and Sequence to sequence learning by letting every word attend to every other word when building its representation.
Uses and Quirks
Embeddings power translation, search, chatbots, and recommendation engines — quietly threading through countless Networked systems most people never see. But they also absorb the biases baked into their training text, inheriting human cognitive biases about gender, race, and profession, which researchers work hard to detect and correct.
There's something almost biological about the metaphor: just as anatomical fields organize tissue by function, embedding space organizes language by meaning, clustering concepts into vast, invisible neighborhoods of sense.