Learning · Visual explainer

Watch a transformer think — and learn.

Six steps, each one a demo you can just look at. Tap the formulas only if you want the maths.

The whole job

It guesses the next word

A model reads what's there and rates every possible next word. It picks a likely one, adds it, and repeats. That's all "AI writing" is — watch:

Step 1

Words become numbers

First the text is chopped into tokens and each gets an ID number, because a computer does maths on numbers, not letters. Type anything:

Step 2

Numbers get meaning

Similar words sit close together on a map the model learns. Click a word to see its nearest neighbours light up — meaning becomes location.

ƒ show the maths
xi = Embed(tokeni) ∈ ℝd
  • xᵢThe point (vector) for token i — its meaning as numbers; nearby points = similar meaning.
  • dHow many numbers per word (real models: hundreds). The map above shows just 2 so we can see it.
Step 3 · the big idea

Words look at each other

The "T" in GPT. Every word pulls in context from the others. Click a word — the thicker the line, the more attention it pays there. See how it reaches back to cat.

ƒ show the maths
Attention(Q,K,V) = softmax( QK⊤√dk ) V
  • QQuery — what the current word is looking for.
  • KKey — what each other word advertises. Q·K high = strong match = thick line.
  • VValue — the info pulled in from the words it attends to.
  • softmaxTurns the match scores into percentages that add up to 100%.
Step 4

Stack the same block, many times

One block = attention, then a little per-word network (FFN), with shortcuts (+) and rescaling (Norm) to keep it stable. Stack dozens. Watch a word flow through:

ƒ show the maths
a = x + Attention(Norm(x))
y = a + FFN(Norm(a))
  • + xShortcut — add the input back so deep stacks still train.
  • NormRescales numbers to a sane range each step.
  • FFNA small network that lets each word "think" on its own.
Step 5 · how it learns

Roll downhill until it's right

Training = make a guess, measure how wrong (the loss), then nudge the model's numbers downhill. The ball is the model; the valley floor is "perfect." Press step and watch it roll.

weight w
0.00
loss
—
step
0
ƒ show the maths
L=−∑y log ŷ   →   θ ← θ − η · ∇θL
  • LLoss — one number for how wrong the guess was (the ball's height). Lower is better.
  • y / ŷThe true next word / the model's predicted probabilities.
  • θAll the model's numbers (weights). Training adjusts them.
  • ∇θLGradient — which way is uphill. We step the opposite way.
  • ηStep size (learning rate). Too big overshoots; too small crawls — try the slider.

A real model swaps "one ball" for hundreds of billions of weights and "guess a number" for "guess the next token" — but the loop is exactly this: guess → measure → roll downhill, trillions of times.