Recording intelligenceAI-generated brief · check the source for context

Let's build GPT: from scratch, in code, spelled out.

1:56:20 recording · AUTO · 1 speaker

Watch the original

Executive Summary AI
  • The lecture opens with ChatGPT writing a haiku about AI and prosperity, and a mock breaking-news article about a leaf falling from a tree, to show that it is a probabilistic system that answers the same prompt differently each time.
  • The neural network underneath is the transformer, proposed in the 2017 paper Attention is All You Need — GPT stands for generatively pre-trained transformer — and the plan is to rewrite the speaker's nanoGPT repository, two files of about 300 lines, from an empty file.
  • The training data is the tiny Shakespeare dataset, a single file of about one megabyte and roughly one million characters, tokenized at the character level into a vocabulary of 65 symbols and split 90% train / 10% validation, then sampled in chunks of block size 8 stacked into batches.
  • The simplest baseline, a bigram model built on an embedding table, starts at a loss of 4.87 against the 4.17 expected from 65 uniform classes, and training with AdamW for 10,000 steps brings it to about 2.5 with still-garbage output.
  • Self-attention is then derived step by step — every token emits a query, a key and a value, affinities come from the dot product of queries and keys, scaled by one over the square root of the head size and masked with a lower-triangular matrix — and adding multi-head attention, a feed-forward layer, residual connections and layer norm drives the validation loss from 2.4 to 2.06.

Brief overview

A transformer language model is built from scratch on Shakespeare to show how ChatGPT works underneath.

  1. Attention is a data-dependent communication mechanismTokens emit queries and keys, dot-product them into affinities, and aggregate values from the nodes they find interesting.
  2. Masking makes it a decoder blockSetting future positions to negative infinity before the softmax stops later tokens from leaking the answer to earlier ones.
  3. Depth needs residuals and layer normSkip connections give gradients a superhighway back to the input, and pre-norm layer norm took the validation loss from 2.08 to 2.06.

Questions this recording answers

10 questions, each answered where it is said
Where does the transformer architecture behind ChatGPT come from?

It comes from the 2017 paper Attention Is All You Need, a landmark AI paper that reads like a fairly ordinary machine translation paper. The authors did not fully anticipate its impact: the same architecture, with minor changes, was copy-pasted into a huge amount of AI applications over the next five years, including the core of ChatGPT.

What does GPT actually stand for?

GPT is short for generatively pre-trained transformer. The transformer is the neural net that does all the heavy lifting under the hood, and it comes from that 2017 paper. ChatGPT itself is a very serious production-grade system, trained on a good chunk of the internet with a lot of pre-training and fine-tuning stages on top.

What model and dataset are built in this lecture?

Rather than a chunk of the internet, the lecture uses Tiny Shakespeare: a concatenation of all of Shakespeare's works in a single file of about one megabyte, roughly one million characters. On it a character-level transformer language model is trained to model how characters follow each other, so it can then generate infinite fake Shakespeare.

What do I need to know before following along?

Really all that is required is a proficiency in Python and some basic understanding of calculus and statistics. It also helps to have seen the earlier videos on the same YouTube channel, in particular the Make More series, which defines smaller and simpler neural network language models like multilayer perceptrons and introduces the language modeling framework.

What is nanoGPT and what is inside the repository?

nanoGPT is a repository on the speaker's GitHub for training transformers on any given text. It is a very simple implementation: just two files of about 300 lines each, one defining the GPT model, one training it on a text dataset. Trained on OpenWebText it reproduces GPT-2's smallest 124 million parameter model.

How is raw text turned into numbers for the model?

Tokenizing means converting raw text into a sequence of integers according to some vocabulary. Here it is done at character level: the 65 sorted characters of Tiny Shakespeare, with simple encode and decode lookup tables. Google's SentencePiece and OpenAI's tiktoken use subword schemes instead; GPT-2's encoding has about 50,000 possible tokens.

How does self-attention decide which past tokens to listen to?

Every token at each position emits two vectors: a query, roughly what am I looking for, and a key, roughly what do I contain. The affinities are the dot products of one token's query with all the other keys, so aligned pairs interact strongly. A third vector, the value, is what actually gets aggregated.

Why are attention scores divided by the square root of the head size?

They call it scaled attention. With unit Gaussian keys and queries, the raw weights have variance on the order of the head size, here 16; multiplying by one over its square root keeps the variance at one. That matters because those weights feed softmax, which sharpens toward one-hot vectors on extreme numbers.

What is multi-head attention and does it help?

Multi-head attention is just applying multiple attentions in parallel and concatenating the results. With an embedding size of 32, four communication channels of eight dimensions each concatenate back to 32, similar to a group convolution. Running it brought the loss down to roughly 2.28, from about 2.4 with a single head.

Why add residual connections and layer norm to a deep transformer?

Two optimizations keep deep networks optimizable. Skip or residual connections, from the 2015 residual learning paper, let you fork off the pathway, compute, then add back, giving gradients an unimpeded superhighway from the supervision to the input. Layer norm normalizes rows, per token, and is applied before the transformation in this pre-norm formulation.

Key Quote

“ChatGPT is a probabilistic system, and for any one prompt, it can give us multiple answers”

— Let's build GPT: from scratch, in code, spelled out., at 0:51▶ watch at 0:51
Key Quote

“Now the query vector, roughly speaking, is what am I looking for?”

— Let's build GPT: from scratch, in code, spelled out., at 1:04:13▶ watch at 1:04:13
Key Quote

“Attention supports arbitrary connectivity between nodes.”

— Let's build GPT: from scratch, in code, spelled out., at 1:15:39▶ watch at 1:15:39
Key Quote

“So the self-attention is the communication.”

— Let's build GPT: from scratch, in code, spelled out., at 1:26:09▶ watch at 1:26:09

Cite this

“Let's build GPT: from scratch, in code, spelled out..” https://www.youtube.com/watch?v=kCc8FmEb1nY. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/let-s-build-gpt-from-scratch-in-code-spelled-out.html.

Report this page
What is wrong with this page?