This opening lecture explains reasoning in LLMs and the four methods the course will teach from scratch.
Reasoning comes in two types. System 1 is fast and immediate, about 95% of our thinking, driven by intuition, past experience and instinct, like answering what is the capital of India. System 2 is slow, logical, lazy and indecisive, like choosing which movie to watch tonight.
No. AI is not made to think like humans, it is made to answer like them. ChatGPT 3.5, out in November 2022, was crazy good on System 1 tasks needing fast answers, but weak on puzzles, thinking and planning, showed no step-by-step process, and hallucinated random answers.
Imagine representing a healthcare, pharma or finance company and delivering a high-stakes application that impacts people's lives. Would you give the client a direct answer, or show the step-by-step reasoning that led the LLM there? Clients trust the second case much more. That is the power of reasoning models.
Research continued and O1, the first reasoning model, was released by OpenAI on September 20, 2024. Asked the same question, which movie shall I watch tonight, it reasoned for a couple of seconds and showed it was thinking through comedic styles, dramedy, action and classic options.
Three things, mainly. It was completely open source, while OpenAI has never revealed the architecture of any model, so we finally saw a working reasoning architecture. Its accuracy was comparable to O1. And it achieved reasoning by pure reinforcement learning, which was a tremendous breakthrough.
Test time compute is the number of computing resources used by the model during inference, and inference is the process that leads the trained model to give you an answer. It is like an exam: preparation is the training stage, actually writing the answers is the inference stage.
On Roger's tennis balls, a regular LLM answers in three tokens: answer, is, 11. Forced to think first, Roger started with five, two cans of three is six, five plus six is eleven, it needs 4 plus 11 plus 5 plus 3, twenty-three tokens, seven to eight times more compute.
The brain team at Google Research introduced it in 2022. You give the model a worked example, Roger started with 5 balls, 2 cans of 3 is 6, 5 plus 6 is 11. The model then answers new questions, like the cafeteria's 23 apples, the same step-by-step way.
Inference time compute scaling, pure reinforcement learning, supervised fine tuning with reinforcement learning, and pure supervised fine tuning with distillation. The last two come with hands-on work: providing reasoning to a model using pure RL, and providing reasoning to a model using the process of distillation.
You take a large model with reasoning capabilities, like DeepSeek R1, and generate input output pairs, say 500 questions and 500 answers carrying the entire reasoning, also called supervised fine-tuned data. Fine-tune a small non-reasoning model such as Qwen or Microsoft's Phi on it, and it gains reasoning.
“I am his twin brother, don't confuse me with him.”
“System 1 helps us think immediately for an answer.”
“AI is not made to think like humans. AI is made to answer like humans.”
“It says wait, wait, wait that's an aha moment I can flag here.”
“Test time compute is the number of computing resources used by the model during inference.”
“So the researchers at DeepSeek actually did not expect this at all.”
“Lecture 1 - Reasoning LLMs from Scratch - Series Introduction.” https://www.youtube.com/watch?v=k2QEElQ9eeM. Transcript summary by WhipScribe, https://whipscribe.com/transcript-pages/lecture-1-reasoning-llms-from-scratch-series-introduction.html.