What is a neural network?

A neural network is just a big stack of adjustable knobs (called weights). Numbers go in, the network multiplies them by its knobs and adds them up, and an answer comes out. “Learning” means slowly turning the knobs until the answers are right.

Training a network to classify

Below is a real neural network running in your browser, learning to separate two groups of dots. Press Play and watch the colored boundary bend until it sorts them correctly — and the loss (how wrong it is) fall toward zero.

Forward pass → backprop → weight update

Above you watched the boundary move. But HOW does a weight know which way to turn? Here is a tiny real network (2 inputs → 2 hidden → 1 output). Step through one full training iteration and watch the exact value of every weight change.

Training a tokenizer (BPE)

A subword tokenizer is also trained — but differently from the network above. It starts from single characters, then repeatedly glues together the most frequent neighboring pair. Press Step (or Play) to watch it build a vocabulary of reusable pieces from this little corpus.

Masked Language Modeling — fill in the blank

Once the tokenizer above has split text into pieces, the encoder learns what those pieces MEAN. This is how modern text encoders (BERT, RoBERTa, XLM-R) actually learn — with no human labels. Hide a word with [MASK] and the model must predict it from context. Repeat over billions of sentences and it absorbs grammar, facts, and meaning. Below is a REAL multilingual BERT predicting live in your browser.

Downloading the real BERT weights… (loads automatically — a moment)

Token labeling & the embedding table

Before any learning, two setup steps turn tokens into something a network can use. (1) Every distinct token from the corpus is given a number — its label/ID. (2) Each ID points to a row of N numbers — its vector. Those numbers begin random; training (next) makes them meaningful.

Learning an embedding from scratch

A network can't read letters — only numbers. So every token is handed a short list of numbers called its embedding (here just 2 numbers, so we can plot it). At the start these numbers are RANDOM. Watch the network learn better ones, live.

Turning words into numbers (embeddings)

Networks only understand numbers, so each word becomes a list of numbers — a vector. Through training, the network learns to place words with similar meaning near each other. Click a word to see its nearest neighbors.

Now in 3D — add your own words

Same idea, but live and in 3D with a REAL model. It turns each word into a 384-number vector; we project that to 3D so you can see it. Drag to rotate, use +/− to zoom, click a word for its closest relatives. Add any word — even Korean — and watch where it lands.

Why does the same sentence cost more in Korean?

AI language models don't read words — they read tokens (chunks of text). Type the same meaning in English and Korean below, and watch how many tokens each one becomes. The counts come from GPT's real tokenizer, running right in your browser.

Tokenization: how machines slice up language

Type anything (try English and Korean!). The same text is split three ways. Notice how the counts and pieces differ.

Trade-offs at a glance

WordPiece vs SentencePiece vs BPE — the real models

The panel above uses GPT’s BPE. Here are three real tokenizers from actual models, running live in your browser. Watch the markers: BERT uses ## (glue to the previous piece), XLM-R uses ▁ (a space before a word).

Normalization: when identical-looking text isn't equal

Before tokenizing, text is cleaned. The sneakiest case: two strings that look identical but are stored as different bytes — so the computer says they're NOT equal. This bites Korean especially (and is the cause of broken filenames between Mac and Windows).

Unicode forms — “한글”

The token tax: low-resource languages pay more

Every bar below is the SAME sentence — “Artificial intelligence is changing the world.” — in a different language, measured with GPT's real tokenizer. English is cheapest because the tokenizer saw the most English. Languages with less training data get shattered into more, smaller pieces.

Korean ↔ English glossary

Self-attention — which tokens matter?

Each token builds a query and compares it against every other token's key. The softmax of those scores decides how much of each token's value to mix in — so a word can “look at” the words it depends on.

Guess where the word looks

The highlighted word is ambiguous. A person uses the rest of the sentence to figure out what it means — and so does attention. Which word do you think it points to?

Putting a Transformer layer together

One block = multi-head attention → add & normalize → feed-forward network → add & normalize. Stack a few dozen of these and you have GPT or BERT.

Build a Transformer block

You've seen the finished block — now assemble one yourself. Click the four pieces in the right order, bottom to top. Data flows up from the input embeddings to the output.

Reading vs generating

Same attention machinery, one difference: what each token is allowed to look at. Green means “can look”.

Row = the token doing the looking · column = the token being looked at