Frontier AI has no single winner: ideas spread freely, so budget and hardware decide.
Sebastian says "winning" is too broad. By 2026 no company holds technology no other company can access, because researchers constantly rotate between labs. The differentiators are budget and hardware constraints, not proprietary ideas, so he sees no winner-takes-all scenario. DeepSeek already won the hearts of open-weight people.
Answered around 17:26Nathan says DeepSeek is losing its crown as China's preeminent open model maker. Z.AI with its GLM models, MiniMax, and Kimi from Moonshot have shone more brightly in the last few months. The new DeepSeek models are still very strong, and DeepSeek kicked off the movement they build on.
Answered around 20:01Nathan expects this to last a few years. US tech and IT companies will not pay for API subscriptions to Chinese companies over security concerns, so open weights are how those firms gain influence and take part in a huge growing US AI expenditure market. It works.
Answered around 21:20Sebastian uses the Codex plugin for VS Code: a convenient chat interface with access to his repository. He knows Claude Code is more agentic, touching more things and doing a whole project, but he is not comfortable there yet, maybe a control freak, and wants to see what happens.
Answered around 36:32GPT-OSS was the first open-weight model really trained with tool use in mind. Tool use means the LLM can run a web search or call a Python interpreter. Sebastian calls it a huge unlock against hallucinations: asked who won the 1998 soccer World Cup, it searches FIFA and answers France.
Answered around 46:46Sebastian counts a handful of tweaks: mixture of experts, group query attention instead of multi-head, RMSNorm replacing LayerNorm, and a different nonlinear activation, like swapping sigmoid for ReLU. Nothing fundamental changes; you convert one model into the other by adding components. His book's GPT-2 has roughly 124 million parameters.
Answered around 58:00It is a tweak that makes a model larger without consuming more compute in each forward pass. The expensive fully connected layer becomes many feed-forward networks, say 256, and a router picks which expert each input token uses. Few active experts is called sparse; dense always uses one module.
Answered around 55:29Nathan defines it as a power law: the x-axis combines compute and data, the y-axis is held-out prediction accuracy on next tokens, text the model has not seen. Scaling laws arrived when people found that relationship was very predictable. Three axes now exist: pre-training, reinforcement learning, and inference time.
Answered around 1:02:54DeepSeek's famous pre-training number was about $5 million at cloud market rates. Olmo 3, detailed in Section 2.4 of its paper, cost roughly $2 million in cluster rental including engineering issues and multiple seeds. Serving hundreds of millions of users runs to billions; a thousand GPUs cost about $100,000 daily.
Answered around 1:07:07Sebastian says the name exists because having pre-training and post-training with nothing in the middle sounded weird. It uses the same algorithm as pre-training but is specialised, for example on long-context documents, which are too scarce for the main phase. It also puts quality content last, countering catastrophic forgetting.
Answered around 1:20:51“So winning is a very broad term.”
“But on the other side of things, there's a lot of ominous technology from China where there's way more labs than DeepSeq.”
“And with code, what's nice is you can verify it.”
“Pre-training has gotten extremely expensive.”
“AI State of the Art 2026.” https://media.blubrry.com/takeituneasy/ins.blubrry.com/takeituneasy/lex_ai_ai_sota_2026.mp3. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/ai-state-of-the-art-2026.html.