Recording intelligenceAI-generated brief · check the source for context

AI State of the Art 2026

4:39:52 recording · EN · 3 speakers

Listen to the original

Listen to the episode
Executive Summary AI
  • Lex Friedman frames the year through the DeepSeek moment of January 2025, when DeepSeek R1 reached near state-of-the-art performance with allegedly much less compute and for much cheaper.16:36
  • Sebastian argues no company will hold technology nobody else has, because researchers rotate between labs, so the differentiating factor becomes budget and hardware constraints rather than proprietary ideas.18:10
  • Nathan expects Chinese open-weight releases to continue for a few years and more open model builders throughout 2026 than in 2025, because US companies will not buy API subscriptions from Chinese firms for security reasons.21:20
  • Sebastian says the architecture has barely moved since GPT-2 — mixture of experts, group query attention and RMSNorm are tweaks, not new architectures — so the real gains now come from mid-training, post-training and inference-time scaling.58:32
  • Nathan puts DeepSeek's pre-training at about $5 million at cloud market rates and Olmo 3's cluster rental at about $2 million, against serving costs of billions, and expects $2,000 subscriptions after this year's $200 tier.1:07:07

Brief overview

Frontier AI has no single winner: ideas spread freely, so budget and hardware decide.

  1. No clear winner at the frontierResearchers rotate between labs, so ideas are not proprietary — budget, hardware and serving cost decide.
  2. Open weights are China's distribution playUS firms avoid Chinese APIs, so open weights buy influence; Nathan expects more open builders in 2026 than 2025.
  3. Architectures barely changed since GPT-2MoE, group query attention and RMSNorm are tweaks; RLVR and inference-time scaling produced this year's capability jump.

Questions this recording answers

10 questions, each answered where it is said
Who is ahead in AI right now, China or the United States?

Sebastian says "winning" is too broad. By 2026 no company holds technology no other company can access, because researchers constantly rotate between labs. The differentiators are budget and hardware constraints, not proprietary ideas, so he sees no winner-takes-all scenario. DeepSeek already won the hearts of open-weight people.

Answered around 17:26
Which Chinese labs are now ahead of DeepSeek on open models?

Nathan says DeepSeek is losing its crown as China's preeminent open model maker. Z.AI with its GLM models, MiniMax, and Kimi from Moonshot have shone more brightly in the last few months. The new DeepSeek models are still very strong, and DeepSeek kicked off the movement they build on.

Answered around 20:01
Why do Chinese companies keep releasing open-weight models?

Nathan expects this to last a few years. US tech and IT companies will not pay for API subscriptions to Chinese companies over security concerns, so open weights are how those firms gain influence and take part in a huge growing US AI expenditure market. It works.

Answered around 21:20
Which AI coding tool does Sebastian use, and why not Claude Code?

Sebastian uses the Codex plugin for VS Code: a convenient chat interface with access to his repository. He knows Claude Code is more agentic, touching more things and doing a whole project, but he is not comfortable there yet, maybe a control freak, and wants to see what happens.

Answered around 36:32
What is tool use in an LLM, and why does it matter?

GPT-OSS was the first open-weight model really trained with tool use in mind. Tool use means the LLM can run a web search or call a Python interpreter. Sebastian calls it a huge unlock against hallucinations: asked who won the 1998 soccer World Cup, it searches FIFA and answers France.

Answered around 46:46
How much has the architecture changed from GPT-2 to today?

Sebastian counts a handful of tweaks: mixture of experts, group query attention instead of multi-head, RMSNorm replacing LayerNorm, and a different nonlinear activation, like swapping sigmoid for ReLU. Nothing fundamental changes; you convert one model into the other by adding components. His book's GPT-2 has roughly 124 million parameters.

Answered around 58:00
What is a mixture of experts and why is it used?

It is a tweak that makes a model larger without consuming more compute in each forward pass. The expensive fully connected layer becomes many feed-forward networks, say 256, and a router picks which expert each input token uses. Few active experts is called sparse; dense always uses one module.

Answered around 55:29
What is a scaling law, technically?

Nathan defines it as a power law: the x-axis combines compute and data, the y-axis is held-out prediction accuracy on next tokens, text the model has not seen. Scaling laws arrived when people found that relationship was very predictable. Three axes now exist: pre-training, reinforcement learning, and inference time.

Answered around 1:02:54
What does it cost to train a model versus serve it?

DeepSeek's famous pre-training number was about $5 million at cloud market rates. Olmo 3, detailed in Section 2.4 of its paper, cost roughly $2 million in cluster rental including engineering issues and multiple seeds. Serving hundreds of millions of users runs to billions; a thousand GPUs cost about $100,000 daily.

Answered around 1:07:07
What is mid-training and how does it differ from pre-training?

Sebastian says the name exists because having pre-training and post-training with nothing in the middle sounded weird. It uses the same algorithm as pre-training but is specialised, for example on long-context documents, which are too scarce for the main phase. It also puts quality content last, countering catastrophic forgetting.

Answered around 1:20:51
Key Quote

“So winning is a very broad term.”

— Sebastian Rashka, AI State of the Art 2026, at 17:26▶ listen at 17:26
Key Quote

“But on the other side of things, there's a lot of ominous technology from China where there's way more labs than DeepSeq.”

— Nathan Lambert, AI State of the Art 2026, at 20:01▶ listen at 20:01
Key Quote

“And with code, what's nice is you can verify it.”

— Sebastian Rashka, AI State of the Art 2026, at 39:39▶ listen at 39:39
Key Quote

“Pre-training has gotten extremely expensive.”

— Nathan Lambert, AI State of the Art 2026, at 1:06:34▶ listen at 1:06:34

Cite this

“AI State of the Art 2026.” https://media.blubrry.com/takeituneasy/ins.blubrry.com/takeituneasy/lex_ai_ai_sota_2026.mp3. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/ai-state-of-the-art-2026.html.

Report this page
What is wrong with this page?