Word Embeddings

What you'll learn

Here is one of the most beautiful ideas in AI: meaning has geometry. If we represent each word as a vector, we can arrange them so that similar words sit close together and relationships become directions you can do arithmetic with. This is the embedding, and it's the foundation of every language model.

By the end of this module you'll be able to:

• Explain why one-hot vectors are wasteful
• Describe a dense embedding
• Read the vector space of meaning
• Understand how embeddings are learned from context

One-hot vectors

The naive way to turn a token ID into a vector is one-hot: a vector as long as the vocabulary, all zeros except a single 1. With 50,000 words that's a 50,000-long vector per word — enormous, and worse, every word is equally distant from every other. "cat" and "kitten" look exactly as unrelated as "cat" and "democracy." No meaning at all.

Dense embeddings

An embedding replaces that with a short, dense vector — maybe 300 numbers — that the model learns. Now each dimension can capture some shade of meaning, and similar words get similar vectors. Instead of 50,000 wasted slots, a few hundred rich ones.

The geometry of meaning

Plotted in space, embeddings reveal structure: words cluster by meaning, and — remarkably — relationships become consistent directions. Step through the classic example:

import numpy as np

# Each word is a learned dense vector (here just 4 numbers; real ones have 100s)
king  = np.array([0.9, 0.7, 0.1, 0.8])
man   = np.array([0.8, 0.2, 0.1, 0.3])
woman = np.array([0.7, 0.2, 0.9, 0.3])

result = king - man + woman
# result is closest to the vector for "queen" -> analogies emerge from geometry

How embeddings are learned

Where do these vectors come from? From a simple, powerful idea: a word is known by the company it keeps. Methods like Word2Vec slide a window over huge amounts of text and train a network to predict a word from its neighbours (or vice-versa). Words that appear in similar contexts get pushed to similar vectors. No labels needed — just text.

Contextual embeddings

Classic embeddings give "bank" one fixed vector, even though it means different things by a river and in finance. Modern models produce contextual embeddings: the vector for a word depends on the whole sentence around it. The mechanism that makes that possible is attention — our next, pivotal module.

Recap & quick check

Fixed embeddings can't tell which "bank" you mean. The idea that fixed that — and powers every modern model — is next, and it's the big one. Next up: Module 23 — Attention.