A transformer language model is built from scratch on Shakespeare to show how ChatGPT works underneath.
It comes from the 2017 paper Attention Is All You Need, a landmark AI paper that reads like a fairly ordinary machine translation paper. The authors did not fully anticipate its impact: the same architecture, with minor changes, was copy-pasted into a huge amount of AI applications over the next five years, including the core of ChatGPT.
GPT is short for generatively pre-trained transformer. The transformer is the neural net that does all the heavy lifting under the hood, and it comes from that 2017 paper. ChatGPT itself is a very serious production-grade system, trained on a good chunk of the internet with a lot of pre-training and fine-tuning stages on top.
Rather than a chunk of the internet, the lecture uses Tiny Shakespeare: a concatenation of all of Shakespeare's works in a single file of about one megabyte, roughly one million characters. On it a character-level transformer language model is trained to model how characters follow each other, so it can then generate infinite fake Shakespeare.
Really all that is required is a proficiency in Python and some basic understanding of calculus and statistics. It also helps to have seen the earlier videos on the same YouTube channel, in particular the Make More series, which defines smaller and simpler neural network language models like multilayer perceptrons and introduces the language modeling framework.
nanoGPT is a repository on the speaker's GitHub for training transformers on any given text. It is a very simple implementation: just two files of about 300 lines each, one defining the GPT model, one training it on a text dataset. Trained on OpenWebText it reproduces GPT-2's smallest 124 million parameter model.
Tokenizing means converting raw text into a sequence of integers according to some vocabulary. Here it is done at character level: the 65 sorted characters of Tiny Shakespeare, with simple encode and decode lookup tables. Google's SentencePiece and OpenAI's tiktoken use subword schemes instead; GPT-2's encoding has about 50,000 possible tokens.
Every token at each position emits two vectors: a query, roughly what am I looking for, and a key, roughly what do I contain. The affinities are the dot products of one token's query with all the other keys, so aligned pairs interact strongly. A third vector, the value, is what actually gets aggregated.
They call it scaled attention. With unit Gaussian keys and queries, the raw weights have variance on the order of the head size, here 16; multiplying by one over its square root keeps the variance at one. That matters because those weights feed softmax, which sharpens toward one-hot vectors on extreme numbers.
Multi-head attention is just applying multiple attentions in parallel and concatenating the results. With an embedding size of 32, four communication channels of eight dimensions each concatenate back to 32, similar to a group convolution. Running it brought the loss down to roughly 2.28, from about 2.4 with a single head.
Two optimizations keep deep networks optimizable. Skip or residual connections, from the 2015 residual learning paper, let you fork off the pathway, compute, then add back, giving gradients an unimpeded superhighway from the supervision to the input. Layer norm normalizes rows, per token, and is applied before the transformation in this pre-norm formulation.
“ChatGPT is a probabilistic system, and for any one prompt, it can give us multiple answers”
“Now the query vector, roughly speaking, is what am I looking for?”
“Attention supports arbitrary connectivity between nodes.”
“So the self-attention is the communication.”
“Let's build GPT: from scratch, in code, spelled out..” https://www.youtube.com/watch?v=kCc8FmEb1nY. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/let-s-build-gpt-from-scratch-in-code-spelled-out.html.