200 lines of pure math. No frameworks. No shortcuts. From random noise to coherent text, step by step.
Every model begins with a choice: what to learn from. The dataset defines the vocabulary, the patterns, the soul of what the model will become.
Before a model can learn, it must learn to read. Text becomes integers, the only language mathematics understands. Every unique character is a token; the rest is just index arithmetic.
Thousands of tiny adjustments, each one nudging the model toward understanding. Loss is its confusion. Watch it fall as the model begins to see the patterns in language.
After training, look inside. Embeddings reveal how the model organised meaning in high-dimensional space, projected down to two dimensions. Attention shows which tokens the model focuses on at each position.
The model speaks. Temperature controls the boldness of each prediction, a dial between certainty and creativity. At zero it picks the most likely token. At high values, pure chance.
Perplexity measures how surprised the model is by unseen text. A perfect model would always predict the next token correctly. Human language models achieve around 10 to 50. Ours starts at vocab size.