Course Overview

The course opener, taught jointly by Aakanksha Chowdhery and Azalia Mirhoseini, walks the whole arc in one sitting — pre-training scaling laws, the emergence of chain of thought, the post-training stack behind ChatGPT, inference-time scaling, reasoning models, agents — then spends the last twelve minutes on grading and project rules. The slides carry most of the specifics; much of the numeric content is never spoken aloud.

Setup and scaling laws 00:30–04:40

The title slide 01:03 lists Prof. Aakanksha Chowdhery and Prof. Azalia Mirhoseini, a CAs: heading left blank, and the course website https://cs329a.stanford.edu — the captions render this as "cs239a", a transcription error. Stated background between them: Google Brain (where they met), Google DeepMind, work on Claude at Anthropic and on Gemini, and research at Reflection AI. Second offering, with updated lectures and one extra homework.

The growth chart 04:38 plots model size in billions of parameters on a log y-axis (0.01–1000) against 2018–2022: ELMo (94M), BERT-Large (340M), GPT-2 (1.5B), Megatron-LM (8.3B), T5 (11B), Turing-NLG (17.2B), GPT-3 (175B), Megatron-Turing NLG (530B), plus a dashed extrapolation annotated "GPT4 is estimated to be here!" The spoken version names PaLM at 540B as the last point, but PaLM is not on this chart — the 530B marker is Megatron-Turing NLG.

Few-shot learning and emergence 04:40–10:30

Three reasons models keep growing: continuous improvement with scaling, few-shot learning, and emergent behavior — "capabilities that only emerge in larger models". The zero/few-shot slide is the GPT-3 figure, both panels annotated "No gradient updates are performed.": zero-shot is Translate English to French: then cheese =>; few-shot inserts sea otter => loutre de mer, peppermint => menthe poivrée, plush girafe => girafe peluche.

The companion grid 10:02 runs eight tasks (mod. arithmetic, IPA transliterate, word unscramble, Persian QA, TruthfulQA, grounded mappings, multi-task NLU, word in context) with accuracy on y against Model Scale (Training FLOPs) from 10¹⁸ to 10²⁴, for LaMDA, GPT-3, Gopher, Chinchilla and PaLM against a dashed Random baseline. Same shape everywhere: flat at random, then a knee. The slide credits this figure to the same Wei et al. chain-of-thought paper as the previous slide.

What ChatGPT actually added 10:30–19:00

The stack is walked against a (Pre-)training → Fine-tuning → Inference graphic with the fine-tuning box highlighted. The alignment slide 13:40 plots sensibleness and safety as percent against model size 2B / 8B / 32B / 128B, each with a dashed Human ceiling, a Finetuned LLM curve and a Base LLM curve. Both fine-tuned curves are near-flat across a 64× size range and sit above the base model at every size — nearly meeting the human line on safety, leaving a visible gap on sensibleness. That is the visual argument that alignment comes from fine-tuning, not scale.

The RLHF slide 17:41 lists the ranking criteria as exactly three — Correctness, Helpfulness, Specificity — where the narration adds harmlessness as a fourth. Its figure runs RLHF and DPO side by side over the same prompt, x: "write me a poem about the history of jazz": RLHF goes preference data → maximum likelihood → reward model → sample completions / label rewards → LM policy under reinforcement learning; DPO goes preference data → maximum likelihood → final LM, with no reward model at all. DPO is on the slide and never mentioned aloud.

Inference as the third scaling axis 19:00–28:00

Pre-training plus fine-tuning was the whole story "until a year and a half ago". The lab's own paper is the case study: Large Language Monkeys 19:56 (Brown*, Juravsky*, Ehrlich*, Clark, Le, Ré, Mirhoseini, 2024, Scaling Inference Compute with Repeated Sampling). The slide draws Step 1 — an LLM, literally a monkey at a typewriter, emitting candidate snippets for "Input a number from stdin and …" — and Step 2, a Verifier box glossed "(e.g. unit tests, proof checkers, majority voting)" picking one. It names two failure modes: Problem 1 (coverage): can we generate a correct solution? and Problem 2 (precision): can we identify a correct solution from the generated samples? The precision half is slide-only, and it is the distinction the rest of the course rests on.

It seems like the models already know a whole lot more than what you get out of them when you just ask them once.

Q&A sharpens three points. Why this is not brute-force search 24:40: the answer space is vastly larger than what the model proposes, so sampling from the model is far more sample-efficient — 10,000 samples is "not a whole lot" for IMO-level problems. On latency: parallel samples parallelize, so the real trade-off is cost. On temperature: past roughly 1.2 output degrades into gibberish. Then DeepSeek (December 2024), the o-series and Gemini thinking models joined test-time scaling to fine-tuning 28:10: test-time search generates high-quality synthetic data on problems with known answers, and that data is trained back in. That closed loop is the "self-improving" in the course title.

Reasoning models 28:00–40:00

The principles slide 31:31 names five behaviours: Problem Analysis, Task Decomposition, Self-Evaluation ("through RL, thinking models learn to finetune their chain of thought and refine the strategies it uses"), Self-Correction, Alternative Proposal.

The "better at reasoning, not at writing" claim is carried entirely by a bar chart 36:56: "Human preferences by domain: o1-preview vs GPT-4o", x = win rate vs GPT-4o (%) 0–100 with a dashed line at 50 and error bars. Reading off the axis, Personal Writing ≈48% and Editing Text ≈50% sit at or below parity; Computer Programming ≈60%, Data Analysis ≈60% and Mathematical Calculation ≈72% sit above it.

The long Q&A that follows is transcript-only — the camera stays on the speakers 35:20–40:20. Repeated sampling improves coverage/pass@k, whereas reasoning training improves pass@1: "a lot of what we cover in train-time or test-time scaling really comes down to it learns which is correct". Models prefer their own traces even when better ones are available 37:50. And on how reasoning is taught: "I don't think there's a published piece of work that really covers this bit… it's a bit of both".

From LLMs to agents 40:20–48:00

A section slide, "What's New? From LLMs to Agents", marks the pivot: chatbots and reasoning models are still single-turn — "fun to interact with, but they're not necessarily accomplishing a task for you" — and what changed recently is Claude Code, Codex and Deep Research completing workflows end to end.

The components slide 45:15 lists the pieces — LLM calls, verifiers, critics/judges, tool calls, search — composed as Prompt Chaining, Routing, Parallelization, Orchestrator, Evaluator/Judge, Verifiers, closing on what is effectively the syllabus: "Agentic workflows require planning, multi-step reasoning and self-improvement." Only the evaluator loop and the parallel fan-in are actually drawn; routing and orchestration are a bullet list plus narration, not diagrams.

Coding agents, and why they work now 47:00–54:30

"AI Coding Agents" 48:15 puts the SWE-agent figure on screen: an LM Agent, an Agent–Computer Interface exposing LM-friendly commands (Navigate repo, Search files, View files, Edit lines) outbound and LM-friendly environment feedback inbound, and a Computer holding a Terminal and a Filesystem listing sklearn/, examples/, README.rst (Yang et al., 2024). The speaker flags it as dated: "I should have updated the slide for Claude Code."

Asked why the same architecture became reliable 48:50: the architecture barely changed; what changed is more capable models and better RL — RL with verifiable rewards, train-time scaling working — plus a loop in which models generate their own unit tests. That surfaces the generator–verifier gap: generating plausible content is easy, judging it needs a feedback loop, and for creative writing human feedback is the bottleneck. Asked why RL produces such large jumps if pre-training already contains the capability 51:40, the honest answer is that opinions differ, there is no consensus, and "that whole loop is not completely well understood".

Where this already pays off is repetitive work 54:27 — the Devin slide lists data warehouse migrations, ETL development, data cleaning and preprocessing beside a screenshot of "Devin's Workspace" with Shell / Browser / Editor / Planner tabs — plus code migrations, version upgrades and unit tests.

Support agents, research agents 54:30–58:00

Customer support: the pattern is defined on the slide as "LLM calls, external information + tools, actions such as issuing refunds; update tickets, success via user-defined resolutions", illustrated with a Google Cloud Live Transcription flow captioned "transcribes all interactions between customer and agent automatically, including PII reduction".

Research agents 55:37 uses the STORM pipeline — Topic → Prewriting {References, Outline} → Writing → Full-length Article, labelled Research via Question Asking, with the worked topic printed as "2022 Winter Olympics Opening Ceremony" (Shao et al., 2024); it reappears in a homework. Then the AI-scientist pipeline 56:48 (Lu et al., The AI Scientist, 2024): Idea Generation (LLM Idea/Plan Innovation → Novelty Check via Semantic Scholar → Idea scoring/archiving) → Experiment Iteration (Experiment Template → Code Δ via LLM & aider → Exec Script → Experiments ⟳ Update Plan → Numerical Data/Plots) → Paper Write-Up (Manuscript Template → Text Δ → Manuscript → LLM Paper Reviewing). The hallucination-as-brainstorming point gets its own slide: a New York Times headline, "How Hallucinatory A.I. Helps Science Dream Up Big Breakthroughs", over a rendered protein structure.

Course logistics 58:00–1:09:42

"CS329A at a Glance" 1:00:50: Test-time Scaling; Self-improvement Techniques (Verifiers, Feedback, RL, Search); Evolution Strategies; Tool-use/Actions; Augmenting LLMs with Retrieval/Memory; Planning and Multi-step Reasoning; Robust Agent Evals; guest lectures from frontier AI labs and startups; Multimodal Agents & Robotics. Prerequisites 1:01:25: Python/NumPy at CS106A/B/X level, familiarity with LLM foundations (e.g. CS229S, CS224N), and research experience — the Where: and When: fields on that slide are left blank.

Groups form via Canvas; the screenshot 1:03:50 shows self sign-up enabled and "groups are limited to 4 members", next to Unassigned Students (59) and Groups (0). API credits are promised. The project slide 1:05:20 is explicit: acceptable — new evaluation datasets for agentic systems, investigating the reliability of an agentic system, hill-climbing a benchmark, improving self-improvement techniques from papers covered in class; negative examples — a survey paper, or "a demo agentic app with no specific eval criterion", restated aloud as wanting "something more than vibe coding". Milestones: proposal, mid-quarter presentation in class (recorded video for NDO students), final report, poster.

That is the 50/50 homework–project split described aloud, but the internal weighting is slide-only: the final report alone is 35%, more than any two homeworks, while the poster is 2.5%. Edstem handles asynchronous questions, Gradescope project milestones. Late policy 1:09:05, strictly enforced because the class is large: up to 4 total late days, at most 2 per assignment, 25% penalty per additional late day, the final project must be on time, no exceptions. Final slide: audits are NOT allowed. And one detail that exists only in the audio — the poster session is announced as December 12, 4:00–6:00 PM with industry attendees 1:07:20; the slide titled "Course Project Poster Presentation" reads only "Final project poster presentation:" with the date field blank.

Test-Time Compute Scaling

The densest plot lecture in the course. Three papers — Large Language Monkeys, Scaling LLM Test-Time Compute Optimally, and Archon — carried almost entirely by figures whose axes, fitted constants and legends are never read aloud. The through-line is that inference has its own scaling law, that the law only converts into capability through a verifier, and that the space of ways to spend inference compute is large enough to be worth searching automatically.

The framing slide 00:29 is a three-box pipeline — (Pre-)training → Fine-tuning → Inference — with the inference box alone in a dotted outline under "What's New? A New Frontier for Scaling is Inference!" Everything that follows happens without touching a parameter.

The coverage curve, past the headline 01:53–04:16

The Monkeys setup slide 01:53 draws the two-step loop — an LLM (a cartoon monkey at a typewriter) emitting candidates, then a Verifier glossed "(e.g. unit tests, proof checkers, majority voting)" — and splits the problem in two, in text never spoken: Problem 1 (coverage): can we generate a correct solution? and Problem 2 (precision): can we identify a correct solution from the generated samples? Part 1 covers the four-panel reasoning result 02:35; the agentic figure is the one to look at here.

Why the curve is a power law 04:43–11:05

After a recap of Kaplan et al. 2020 04:16, the inference analogue 04:43 is stated as an exponentiated power law — the slide's wording; the transcript says "exponential power law" — with the formula on screen:

c = exp(a·k^b) — k: number of samples; a & b: curve fitting parameters; c: coverage, the fraction of problems solved by at least one sample.

Then the lecture goes somewhere the captions never signal: a second paper, printed only in the footer — "How Do Large Language Monkeys Get Their Power (Laws)?", Rylan Schaeffer, Joshua Kazdan, John Hughes, Jordan Juravsky, Sara Price, Aengus Lynch, Erik Jones, Robert Kirk, Azalia Mirhoseini, Sanmi Koyejo, ICML 2025. It is the source of the next three slides.

Verification is the binding constraint 13:01–19:13

The verification slide 13:01 lists exactly three easy cases: formal proofs for math, unit tests for coding, and "AI as a compiler: e.g. generate CUDA and use source Pytorch as verifier" — the third a lab project, illustrated with a KernelBench figure the room camera renders unreadable.

The follow-up slide 15:56 explains why. "The Correct Generations Could Be Very Rare" plots Correct Sample Percentage (0–100%) against Problem Index (Sorted by Correct Fraction) for the same four model/dataset pairs, with green bars for "Problem is correct with majority voting" and dark red for "incorrect with majority voting". The red band sits entirely at the left edge, where the correct fraction is a few percent — the hardest problems are solved two or three times in 10,000, so a frequency-based selector cannot see them.

Discussion 19:13–26:20

The discussion slide 19:13 carries three prompts verbatim: the main takeaways and where to build; how to dynamically adjust sampling strategies given the "needle-in-the-haystack" precision problem at large k; and how to mitigate flaky tests, false negatives and inconsistent test-case construction. The eight minutes that follow are transcript-only — the recording stays on the room camera. Raised from the floor: refutation instead of verification, simulation-based verifiers in physics or molecular dynamics, and ensembling many verifiers, which previews Weaver in the next lecture. The 10,000-samples-per-problem dataset is on Hugging Face and shrinking the gap is offered as a course project 24:27.

Parallel versus sequential 26:53–34:00

The pivot slide 26:20 is two lines: "Monkeys showed repeated sampling is very effective! What are other ways to scale compute at inference?" The comparison diagram 27:54 uses one worked prompt throughout — Q: If 4 daps = 7 yaps, and 5 yaps = 3 baps, how many daps equal 42 baps? — with Parallel Sampling fanning out to independent answers ("LM proposes answers independently, in parallel") over Sequential Revisions as a chain ("LM proposes a sequence of revisions, each conditioned on previous revisions").

The reward-model slide 29:17 is a two-line definition: "Outcome Reward Model assigns a score to the final answer when it is correct"; "Process Reward Model assigns a score for each step in the model's response and not just the final answer." The beam-search figure 30:43 makes PRM-guided search concrete: from a Question node, four candidates per level, each in a dashed box (the key reads dashed box = Apply Verifier, green = Selected by verifier, red = Rejected by verifier), annotated "Select the top-N samples at each step using the PRM" and, at the bottom, "Select the best final answer using the verifier." The compute-optimal slide 33:15 puts three layouts side by side: Parallel Best-of-N (8 independent samples), Sequential Revisions (a chain of 8), and Combining Sequential/Parallel (two chains of four — "verifier selects the best answer within each chain", then "across chains").

The methodology slide 34:00 is more specific than the narration, which says only "the palm model": MATH benchmark, 12k train and 500 test questions; PaLM 2-S* (Codey) base model; difficulty defined by binning pass@1 estimated from 2048 samples into five quantiles; the PRM trained with the MATH-Shepherd approach; the revision model finetuned on trajectories of incorrect answers followed by a correct one.

The second discussion slide 42:26 adds a detail the narration skips entirely: "Paper mentions that the PRM800k dataset was not useful due to distribution shift across different models." The discussion again runs on the room camera 42:55–45:00.

Archon: searching the space of inference architectures 45:38–1:02:59

The framework slide 46:42 — Archon: An Architecture Search Framework for Inference-Time Techniques, Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, Azalia Mirhoseini — draws three columns. Inputs: Target Benchmark(s), Inference Call Budget, Available LLMs, and a stack of Inference Time Techniques. Optimizer: an ITAS Optimizer (Hyperparameter Selection) in a loop with Proposed Archon Architecture and Sample Benchmark Results. Outputs: an Optimized Archon Architecture drawn as layers G G … G → C → R → F → Output.

The operations table 48:08 stays on screen for several minutes with one row highlighted at a time; the inference-cost column is the part the narration never gives.

The "Natural Language Unit Tests" slide 55:35 shows the generated tests verbatim. Problem: "Check for balanceness of round brackets in an input string", with Example Input "Some random text (with nested (brackets [inside] and even more)) closing brackets )) here!" Five LLM-Based Unit Tests follow, in prose rather than code — among them "Given a string that starts with a closing bracket like ')(', the solution should output 'NO' immediately" and "When a closing bracket is encountered, it must match the most recently opened bracket that hasn't been matched yet."

Just like in deep learning, we are adding layers in pre-training and the model gets better. It seems like these careful additions of inference layers are helping the model become more and more accurate. 58:48

The search is constrained by four printed rules 1:00:17: one type of op per layer; the first layer is Generator ops only; everything else can go anywhere, but a Critic must precede a Ranker or a Fuser, and a Unit Test Generator must be followed by an Evaluator. The optimizer comparison 1:00:49 plots Performance Relative to Optimal Architecture (%), 0–100 against Number of Configurations Tested, 0–7000 on MT Bench and Arena-Hard-Auto: Bayes Optimization reaches 100% within roughly 600–900 configurations on both, while Greedy Search climbs in steps to 100% near 5700–5900 and Random Search only gets there at 6500–7000.

The results table 1:02:04

Reproduced from the slide. All Archon rows produce a single final response; "Infer. Calls" is the measured call count.

Two things the table shows that the "average of 14.1%" headline does not. The open-source-only Archon beats GPT-4o on MT Bench by 23 points and on MATH by 3.4, but is below both frontier models on MixEval (86.2% against 87.5% and 89.1%). And the general-purpose configurations, tuned across all benchmarks rather than one, give up only a few points to the task-specific ones while using fewer calls. The slide credits TextGrad (Yuksekgonul, Bianchi, Boen, Liu, Huang, Guestrin, Zou) and Mixture-of-Agents (MoA) (Wang, Wang, Athiwaratkun, Zhang, Zou, 2024) as the comparison systems. The MixEval Hard entry for Closed Src. Task Specific is printed as 72.6 without a percent sign — a slide typo, not a different unit.

The lecture ends on a discussion slide it never reaches 1:02:59: how Archon could adapt architectures to new queries online rather than offline, and where the gain would outweigh the added cost and latency.

Robust Verification

Four papers, four years, one question: once a model can generate a correct answer somewhere in its sample set, how do you find it automatically? The lecture runs from a trained outcome verifier (GSM8K, 2021), through process reward models built on human step labels, to process labels generated with no humans at all, to an ensemble of imperfect off-the-shelf verifiers weighted by weak supervision.

The reading list 00:54

The agenda slide reappears before each section with the current paper un-greyed: Training Verifiers to Solve Math Word Problems (Cobbe et al. 2021), Let's Verify Step by Step (Lightman et al. 2023), Math-Shepherd (Wang et al. 2023), and Shrinking the Generation-Verification Gap with Weak Verifiers (Saad-Falcon et al.) — dated 2025 on its first appearance and 2023 on every later copy 31:41.

1. Training Verifiers to Solve Math Word Problems 01:23–21:00

Two motivations on the slide: LLMs hallucinate and confidently present wrong solutions, and there was no suitable math reasoning benchmark — SQuAD and OpenBookQA test language understanding and QA, MATH is competition-level, and none expose intermediate reasoning steps.

GSM8K is 8.5K grade-school problems selected for quality, diversity, multi-step reasoning, and natural-language rather than pure-math solutions 02:21. A corner bar chart counts papers per year 2021–2025 for GSM8K, HumanEval, ASDiv and MMLU — near zero through 2022, then roughly 900 for GSM8K and just over 1,000 for MMLU in the tallest year.

The verifier outputs the probability that a solution is correct, and its training data labels itself: fine-tune the generator for 2 epochs, sample 100 completions per training problem, mark each against the human ground-truth answer, train the verifier for one epoch. It is a language model with a small scalar head predicting per token, trained on a joint objective — binary correctness plus ordinary next-token prediction — with question tokens masked out 05:24. The ablation slide contrasts sentence-level scoring with token-level and settles on the score after the last token; a red arrow is drawn on the diagram from that phrase to the final per-token output box 09:43.

Asked why it declines rather than plateaus 17:20: the plot is the output of the whole system, not coverage. Across 800 near-identical candidates of which one is right, the verifier's precision degrades and it starts ranking a wrong solution first. The paper shipped at 100 samples. The other durable argument for verification 20:13 is that the generator stays general — nothing is specialised to one dataset.

2. Let's Verify Step by Step 21:29–37:00

The method slide splits the two reward models at the point of supervision 22:30. A math problem goes to a Generator emitting Step 1, Step 2, …, Final ans; from the final answer hangs one box R_N: 1/0 annotated "Obtained from ground-truth" — the ORM's entire signal. The PRM build adds a stack R₁: 1/0, R₂: 1/0, … annotated "Labeled by human", scored at inference as P(R₁=1) · P(R₂=1) · … · P(R_N=1). Three named reasons follow 25:17: credit assignment, false positive avoidance — outcome supervision may reward a solution that reaches the right answer with wrong reasoning — and AI alignment, meaning interpretable, human-endorsed reasoning.

The labels are the expensive part: PRM800K is 800k step-level labels over 12k math problems, each step positive, negative or neutral, collected with an active-learning filter for convincing wrong answer solutions — on the slide, "solutions that are rated highly by the current best PRM but result in the wrong answer" — which proved 2.6× more data efficient than random sampling 26:43.

Both reward models are fine-tuned from GPT-4, which is also the generator; ORM pairs are (sample, final-answer correctness), PRM pairs are (sample, step-wise correctness from PRM800K).

A label-efficiency plot follows 30:02: best-of-500 solve rate against solutions labelled per problem. Both PRM variants end near 59% but start at 48.6% with a single label per problem, while the two ORM variants start near 26% and reach only ≈52% and ≈48.5%.

Out-of-distribution generalisation 30:58

PRM wins every row. The spoken claim that majority voting generalises better than the ORM holds on three of the four exams but reverses in aggregate — ORM 63.8% against majority voting's 61.3% — carried entirely by AMC10/12.

On reward hacking 34:27: what stops a model skipping the reasoning and jumping to an answer the PRM likes? Here nothing is being trained — the generator is untouched and human annotators would mark a skipped step down. The failure mode appears when the PRM is used to fine-tune the generator, and worsens once the step labels are themselves model-generated — the next paper.

3. Math-Shepherd: PRMs without human annotation 37:51–51:31

The definition on the slide: the quality of a reasoning step is its potential to reach the correct answer. From a step, sample N continuations and score by whether the decoded answers match the gold one — hard estimate (HE) is 1 if any continuation ever gets it right, soft estimate (SE) is the fraction that do.

The class names both failure modes before they appear 41:25: hard problems give no signal (if no rollout succeeds every step scores zero, penalising unusual-but-correct approaches), and wrong steps on a lucky trajectory get labelled correct. The hard-vs-soft slide 43:56 plots validation loss against N = decoded paths (1, 4, 16, 64, 256): the soft curves fall monotonically (7B 2.77 → 1.76, 13B 2.50 → 1.47, 70B 2.50 → 1.38) while 70B:Hard turns around after N=4 and climbs back to ~2.33. Soft looks better on loss; the best downstream results came at N=4 either way, so hard was used for simplicity.

At N=256 against baselines 44:46, GSM8K reaches ~93% for Shepherd versus ~92% ORM and ~88% self-consistency; on MATH, ~44.6% against PRM800K (~41%), ORM (~40.4%) and self-consistency (~39.4%) — an automatically-labelled PRM beating the human-labelled one. Table 1, verification over 256 outputs:

Then the loop closes with step-by-step PPO against that PRM. Table 2, greedy decoding 50:36: Mistral-7B: MetaMATH 77.9 / 28.6 (GSM8K / MATH), +RFT 79.0 / 29.9, +ORM-PPO 81.8 / 31.3, +Math-Shepherd-PPO 84.1 / 33.0; LLaMA2-7B follows the same ordering, 66.6 / 19.2 up to 73.2 / 21.6. Asked why they stopped at greedy decoding, the answer is Table 3 51:02, which re-ranks the PPO'd Mistral-7B over 256 outputs to 89.1 / 43.5 with self-consistency + Math-Shepherd, versus 86.6 / 38.0 for the best row without RL. The model generates its own step labels, trains its own reward model, improves itself against it, then is re-ranked by it — with gains visibly plateauing.

To make PRMs reward self-correction, the suggestion is a rubric for each step plus tool use — a calculator or SymPy settles the arithmetic in the running example 47:51.

4. Weaver: shrinking the gap with weak verifiers 51:31–1:06:00

The motivation slide asks whether the gap can be closed by scaling inference compute, in particular by ensembling multiple weak verifiers 52:03. "Weak" does not mean bad — these are the best available verifiers, all imperfect. The pool spans reward models, PRMs and ORMs, and LLM judges: a model asked directly whether an answer is correct.

Weaver's pipeline is score, weight, select 56:21: normalise verifier outputs to a common scale and filter out low-quality ones; use weak supervision to estimate verifier accuracies from minimal labelled data (~1%); combine and take the highest-confidence response. The diagram shows LM judges emitting False/True and reward models emitting scalars like 0.3, normalised into a column 0.0, 0.5, 0.7, 0.2, …, weighted, and reduced to a verdict: Accept or Reject. Filtering matters as much as weighting.

The theory slide cites Snorkel, Ratner et al. (2017), sets up n queries × k solutions × m verifiers, and assumes each verifier captures independent aspects of correctness, giving Pr(Y=1|S₁…Sm) = ∏ᵢ Pr(Sᵢ=s̄ᵢ|Y=1)·Pr(Y=1) / Pr(S₁…Sm) 57:27. The intuition 58:31: if every verifier agreed on every sample the ensemble would teach nothing — the signal lives in their disagreement. Against a naive average of the same verifiers 59:28, Weaver adds +4.7% on GPQA Diamond over an oracle top-5 pool (64.5 → 69.2) and +10.1% over the total pool (62.1 → 72.2); +3.3% and +6.4% on MATH 500; +2.4% and +5.1% on MMLU Pro.

Read across the table 1:02:26: an 8B generator verified by a pool of ≤8B models averages 70.0%, roughly what the 70B class gets from majority voting; the 70B generator reaches 86.2% against o3-mini's single-sample 86.7%. These are end-to-end selection accuracies, not coverage. The 70B row's pool is written "72B and below" on the slide and spoken as 70B.

A second plot 1:05:51 traces success rate against inference compute per query (2⁻⁴–2¹⁰ ExaFLOPs), annotating +18.2 accuracy points on GPQA Diamond, +10.5 on MATH 500 and +14.7 on MMLU Pro at matched compute, and 97.3% compute saved at matched accuracy. Both checkpoints are open-sourced.

Recap and closing questions 1:06:45–1:12:59

Two recap slides: outcome-based verification improves performance during both training and inference; process supervision beats outcome supervision, though outcome reward also scales with data; verifier training data can be labelled automatically and makes a strong RL/PPO reward model; an ensemble of weak verifiers can outperform a larger single model, then be distilled past every individual verifier in it.

We are using test-time scaling, but by bringing more verifiers, rather than sampling a single verifier more. 1:07:15

The closing Q&A is transcript-only — the camera holds on the room with the recap slide behind. Beyond math, model-generated unit tests become the verifier, previewed as the "code monkeys" paper 1:08:14. On whether repeated sampling eventually collapses into training-time data generation, leaving a model asked only once 1:10:47: that is the direction, but sharpening the log-probs onto a single answer risks losing the creativity and diversity of solutions.

Learning from Feedback with Tools and Code

Three papers, one loop: the model produces something, something outside the model judges it, and the judgement goes back into the context or into the gradient. ReAct gets the judgement from a Wikipedia search API, RLEF from a Python test runner, Constitutional AI from the model reading a written rule. What differs is only where the feedback comes from — and every one of the three has a regime, visible on the slides, where it makes the model worse.

ReAct: reasoning and acting 01:00–22:30

The prior-work slide 04:01 states both halves of the gap: chain of thought's problem is that "the model's reasoning is an internal state that is not grounded (a static black box) → frequent hallucinations"; WebGPT-style decision-making LLMs have "no explicit reasoning procedure" and rely on expensive datasets and human feedback.

The formalism slide 07:47 makes the trick explicit. An agent observes o_t ∈ O and acts a_t ∈ A given c_t = (o_1, a_1, …, o_{t-1}, a_{t-1}, o_t); learning π(a_t|c_t) directly is "complex and expensive". So augment the action space with language:  = A ∪ L. A thought is an action in language space, â_t ∈ L; thoughts do not affect the environment, they only produce c_{t+1} = (c_t, â_t). Implementation 09:18: a frozen PaLM-540B few-shot prompted with trajectories of actions, thoughts and observations; reasoning tasks alternate thought and action, decision tasks use thoughts sparsely.

The action space 16:51 is deliberately impoverished — search[entity] returns the first 5 sentences of a wiki page or 5 similar entities, lookup[string] simulates ctrl+F, finish[answer] ends the task — with the slide's own note: "purposefully weak action space to simulate human interaction with Wikipedia." Baselines 17:27 ablate ReAct trajectories: Standard drops thoughts, actions and observations; CoT drops actions and observations; CoT-SC samples 21 trajectories and takes the majority; Act drops thoughts. The two hybrids back off from one to the other on failure.

Plain ReAct loses to chain of thought on HotpotQA and wins on Fever; only the hybrids clear CoT-SC, and every prompting number sits far under supervised SoTA.

Grounding does not remove failure; it moves it. ReAct hallucinates on 0% of failures against CoT's 56% — but its reasoning-error rate is 47% against CoT's 16%, and another 23% of its failures are the search returning nothing useful.

Below the table sit two grouped bar charts, y-axis HotpotQA EM (0–30), x-axis size (8b, 62b, 540b), panels learning = prompt and learning = finetune, four bars each. Under prompting, Act and ReAct are the worst methods at 8b — around 5–6 EM against Standard's ~15 — and only catch up at 540b. Under finetuning the order flips: Act and ReAct are tallest at 8b and 62b, ReAct peaking near 33 EM. The 540b finetune group is empty; it was not run.

The success-rate column is the harsher one: 40.0 against a human expert's 59.6, on a task where errors cascade. The weaknesses slide 22:20 names the costs — "large complex tasks with large action space require more demonstration, which can exceed in-context learning", and multiple reasoning-action steps raise inference time and cost. In today's thinking-mode models all of this happens automatically, because they were distilled on exactly these traces 11:50.

RLEF: grounding code LLMs in execution feedback 27:20–44:00

The contributions slide 28:49: an end-to-end RL fine-tuning framework where actions = code, observations = execution feedback, binary reward on a held-out test; iterative refinement that repairs code with feedback before it becomes training data ("high quality data for PPO"); and two techniques, two-tier tests and a hybrid token/turn-level policy.

The two-tier slide 35:19 is headed "Avoid info leakage!!": public tests are used during iteration, are a limited set for speed and guide solution selection; private tests are hidden during generation, give full suite coverage and drive PPO training. Pressed on why the split matters when neither set is training data, the answer is that training on the same signal you use as feedback in the outer loop is the leak that counts 42:30.

The PPO slide 32:33 spells out the reward the narration only calls binary: r(s_t,a_t) = +1 if end of episode and all tests pass, −1 if end of episode and any test fails, −0.2 if a_t does not contain valid code. The return subtracts a KL penalty, R = r − β log(π(a_t|c_t)/ρ(a_t|c_t)), with A_t = R − V(c_t). The labelled split: policy at token level versus value function at turn level, which "evaluates complete responses, uses the last token of the prompt" and assigns a single advantage to all tokens in the response — closer to GSPO than to per-token credit.

RLEF hurts the 8B model single-turn on CodeContests (11.8 → 9.7) and on MBPP+; the gains are all multi-turn. The 70B+RLEF model at 40.1 MT is well past gpt-4o-2024-05-13 at 24.3, while still trailing it single-turn on HumanEval+. The slide's own bullet on the base rows: "base models rarely benefit from access to faulty solutions and execution feedback."

The slide skipped at 43:20 is a strengths/weaknesses page carrying the section's sharpest sentence:

Single-turn performance is hurt by RLEF training. Also performance improvements are observed with up to 5 turns, but benefits plateau beyond this limit.

The class had already pressed on the same seams 38:00–40:00 — binary feedback is plausibly enough only because these problems are small ("not more than 100 lines of code"), and rewarding only the final solution may teach the model to lean on repair.

Constitutional AI: feedback from written principles 46:20–60:30

Two stages 51:43: supervised learning fine-tunes on data generated by self-critique and revisions; RLAIF trains a preference model on those responses plus the constitution, then fine-tunes the LLM to maximise it. The supervised results slide 51:20 shows three panels sharing a y-axis of PM Score against Number of Revisions (0–4), coloured by Parameters from about 10¹⁰ to 5·10¹⁰: harmlessness rises (largest model −0.4 → ~2.3), helpfulness falls (~0.05 → ~−0.85), HH rises for every size — and nearly all the movement happens between revision 0 and revision 1.

Two questions worth keeping. On amending a constitution 52:20: post-training is maybe 5% of compute and reruns often, so updates are cheap — but the real problem is continual learning, and getting a model to reliably forget is unsolved. On validating AI feedback 57:20: you still want a human-checked validation set for the preference model, "not scaled to 10,000 or that level", but not zero. The follow-on slide 60:25 lists RLAIF vs. RLHF (2023), Self-Refine (2023) and Training Language Models to Self-Correct via Reinforcement Learning (2024), with the caveat that self-critique is often harder than critique by others.

Recap and closing Q&A 62:43–71:00

After the recap slide 62:43 the recording stays on the room camera, so the last eight minutes are transcript-only — including the aside at 63:20, "sorry, this slide was not meant to be there," whose slide never appears on screen. The best exchange is on whether RL post-training makes hand-crafted frameworks obsolete 67:00: yes if you can define the search space, no because for most tasks you cannot. "If you wanted to build a finance agent, if you want to build a legal agent, what is the set of steps they follow?" ReAct matters because it fixes a workflow — and the workflow is domain-specific, which is exactly why it does not generalise.

Planning and Multi-Step Reasoning

Three ways to make a model work in more than one step: LATS runs Monte Carlo tree search over agent actions at test time, SPRINT fine-tunes a reasoning model to emit independent plans together so their executions can run concurrently, and SWiRL does step-wise RL on synthetic trajectories with every tool call moved offline, out of the training loop. The slides carry the search-tree mechanics, the verbatim agent traces and the benchmark tables that the spoken lecture only gestures at.

LATS: Language Agent Tree Search 00:00–22:50

The framing slide is a trip-planning checklist 01:00: Reason ("What is my budget? Where should I go to?"), Act ("Browse the web to gather general consensus or ask around for opinions… read subreddits, travel blogs"), Search ("According to feedback/info gathered online, further plan and research"). The gap LATS targets is that a single LLM plan-then-execute pass does not produce a diversity of candidate solutions, so the paper imports MCTS from the RL literature into the agent loop.

The related-work slide 05:30 draws the contrast precisely: Math-Shepherd "scores outcomes of the reasoning process based on some simple verifier", whereas "LATs scores action outcomes based on self-reflection and observations from the environment". The bottom line, underlined: "LATS Extends ReAct to incorporate planning into reasoning and action". The intuition slide 06:45 gives the decomposition LATS = CoT + Thinking fast and slow + ReAct and a loop diagram: the LLM Agent emits Actions into the Environment, which returns Obs/Reward into a Context box; Context feeds an Evaluation/Self-reflection box; both feed Memory; Memory feeds Tree Search, which returns "Values" and hands the "Best Node" back to the agent.

The maze walkthrough 08:00–16:20

The method is taught on a running toy: "Task: Navigate through a maze to reach an exit. Initial Observation: 'You are in a dimly lit room. There are two doors: one on the left and one on the right.' Goal: Reach the exit (a terminal state with a reward, say r=+1)." Each step adds an ASCII tree in the middle of the slide plus a prose box on the right.

Benchmarks 17:20–19:20

HotPotQA is run as "Multi-hop QA … requiring retrieval over 2+ Wikipedia passages", with an "oracle setup — environment provides correctness feedback", on a "subset of 100 questions, 3 few-shot examples". Exact-match numbers (slide credits Tables 2, 3 of the paper):

Two things the numbers show that the narration does not: Tree of Thoughts on this task is barely better than ReAct (0.39 vs 0.32) and well behind Reflexion, and the best LATS configuration is the one that combines internal chain-of-thought with external ReAct, not the one with the largest sample budget.

The slide says "1.18 real-world products, 12k human instructions" — the magnitude suffix is missing on the first figure. The claim made aloud, that LATS gets "close to human experts" without fine-tuning, holds for the Score column (75.9 vs 82.1) but not for success rate: LATS's 38.0 SR is below the fine-tuned baseline's 45.0 and far below the expert's 59.6. The bolding on the slide is confined to the first block, i.e. LATS is best among the prompting methods.

The summary slide 19:20 is blunt about costs: "Multiple iterations and sampling", "Potentially impractical for time-sensitive scenarios", "Cost-benefit analysis not thoroughly analyzed", and under Assumptions on "environment": "May not work for environments where actions are irreversible. How to adapt to these environments?"

If the agent is running a transaction — paying for a service — you cannot simply back up the tree.

The discussion slide 20:20 poses five questions; two are sharper than the ones actually debated: whether LATS could decompose proactively rather than only after a failure is detected, and that "its improvements only exist within one session and do not persist beyond the inference context. How might we transition from this to a learned paradigm?" — which is precisely what the next two papers do. In the room, the questions raised were whether other bandit algorithms could replace UCT 21:00 (answer: yes, the contribution is the platform) and how repeated actions across the tree are handled 22:15 (answer: repeats inside a trajectory raise the visit count that feeds UCT; the structure is assumed to be a tree, not a fully connected graph).

SPRINT: planning in parallel 23:00–50:10

The motivating slide 24:00 reproduces two DeepSeek-R1 training curves side by side. Left, "DeepSeek-R1-Zero AIME accuracy during training": x-axis Steps 0–8000+, y-axis Accuracy 0.2–0.9, with r1-zero-pass@1 (blue) climbing from about 0.16 to 0.71 and r1-zero-cons@16 (red) from about 0.26 to 0.86; two horizontal dashed references, o1-0912-pass@1 at ≈0.74 and o1-0912-cons@64 at ≈0.83, with the red curve crossing the upper dashed line near step 6300. Right, "average length per response during training": the same x-axis against 0–12,000 tokens, rising from roughly 500 to roughly 9,500. Attribution on the slide: DeepSeek-R1, DeepSeek-AI (2025).

The framework slide splits the system into "a planner and a pool of executors that concurrently carry out these plans", with a post-training recipe on the left and an inference picture on the right showing three stacked "Plan" boxes fanning into parallel lanes.

The inference-process slide 31:10 shows what the learned output format buys you: the planner emits a single <Plan_i> block containing several "Thinking … <prompt_i.1>…</prompt_i.1>" entries; each is routed to its own Executor, which returns <execution_1>…</execution_1>; the executions are labelled "(2) Parallel execution" and then "(3) Syncing" back into a shared Running Context, from which the planner either starts the next round or terminates with the Final Answer. Nothing about the architecture changes — as the answer to a question from the room puts it 35:50, "it's always next token"; the tags are simply a format the runtime can branch on.

What the numbers actually say 36:30–46:20

Training recipe on the slide: 6k DeepSeek-R1 thinking trajectories on the MATH train set, filter out samples with low parallelization opportunities, SFT DeepSeek-R1-Distill-Qwen-7B on the reformatted trajectories. The results plot is accuracy (%) against average number of sequential tokens, both linear: SPRINT at ≈2,400 tokens / 92.5%, R1-Distill-32B at ≈2,300 / 91.5%, RFT at ≈2,800 / 91.0%, SoT at ≈3,850 / 90.8%, R1-Distill-7B at ≈2,000 / 89.1%. Two corrections follow from reading the axes. The accuracy gain over the same-size baseline is +3.4 points (89.1 → 92.5), and SPRINT does not use fewer sequential tokens than either the 7B baseline or the 32B model — it sits slightly to the right of both; what it does is beat a 4× larger model at roughly the same sequential cost. The saving is against RFT and SoT.

Against RFT the MATH500 token saving is ~15%, against SoT-reasoning ~36% — the "about 40%" quoted aloud 44:40 matches the latter comparison, not the headline baseline. The out-of-domain margins are narrow in accuracy (+1.0 on Countdown, +0.5 on GPQA-Diamond over RFT) but large in tokens on Countdown (2284 vs 4917, a 54% cut).

The parallelization-pattern chart 40:50 plots number of problems (0–500) against stage number 1–11, stacked by MATH difficulty Level 1–5, with a dashed "Parallelized" line overlaid. The line peaks at stage 2 (~310 problems parallelized) rather than stage 1 (~230) and decays to near zero by stage 8; the stacked bars show that the problems still alive at stages 6–11 are almost entirely Level 4 and 5. So "harder problems require more iterative planning" and "more parallelism early" are both visible in the same figure.

The honest slide is the last one 45:30: "Even Larger Relative Saving for Longer Sequences" bins problems by how many tokens the RFT model generated. On MATH500 the reduction runs −5% (0–4k), 17% (4k–6k), 19% (6k–8k), 39% (8k–10k). On Countdown, −42% (0–2k), 28%, 52%, 65% (6k–8k). On GPQA-Diamond, −28% (0–6k), 9%, 25%, 45% (12k–15k). On short problems the plan/execute scaffolding is a net loss, sometimes badly so. The closing note 46:20 is that the Sonnet 4.5 system card's prompt encourages the model to use tools at least 100 times — an indication of how far past the short-trace regime real work now runs. The "Next Steps" slide 49:40, which the frame sampling skips, names three: use GRPO "to let the model find parallelization strategies beyond supervised samples", overlap independent tool calls, and "employ hardware-optimized implementations that convert sequential token reduction into lower latency" — i.e. the wall-clock speedup is not yet demonstrated.

SWiRL: step-wise RL with the tools switched off 50:10–1:14:55

Title slide 50:20: SWiRL: Synthetic Data Generation & Multi-Step RL for Reasoning & Tool Use, Anna Goldie*, Azalia Mirhoseini*, Hao Zhou, Irene Cai, Christopher D. Manning, COLM 2025. The challenges slide 51:30 states the third problem exactly: RLHF, RLAIF and RLEF "have mainly focused on single step optimizations, assigning outcome rewards to an entire chain of reasoning and tool call steps only at the end of a trajectory". The design-goals slide 53:10 adds two lines beyond the five capability bullets: "We also want to avoid calling tools during the training process and not be bottlenecked by the slow, expensive, and buggy tool calls", and "We want the model to generalize these reasoning and tool use capabilities across tasks and tools".

Stage 1 55:00 generates trajectories by iterative prompting, then splits the filtering into two branches drawn explicitly on the diagram: "All steps are judged to be reasonable by an LLM (all process label are positive)?" → Process-filtered data; "Final answer is correct given golden answer?" → Outcome-filtered data.

The objective slide 1:03:20 writes it as J(θ) = E_{s∼T, a∼π_θ(s)}[ R(a|s) ], with τ = (s₁, a₁, …, s_K, a_K) a synthetic trajectory, each state s_i containing "state s_{i−1}, action a_{i−1}, tool call response to a_{i−1}", and the note "R(a|s) is the quality of the generated action a given the context s, assessed by a reward model. No golden labels are used."

The experimental-setup slide 1:06:10 is visibly a draft: synthetic data generated by Gemma-2-27b, sourced from HotPotQA and GSM8k; "We collect 50k trajectories from HotPotQA sourced from ?? problems" split into four 10k subsets (process filtered, outcome filtered, process and outcome filtered, random); "We collect ?? trajectories from GSM8K sourced from ?? problems"; base model for fine-tuning also Gemma-2-27b; Gemini 1.5 Pro grades the process reward; and a final bullet reading "TODO: Some info on RL policy, number of TPUs, number of steps of RL training". The judge model is not mentioned aloud; the answer given to a question is only that no judge was trained, just prompted.

Results 1:07:10–1:14:20

The filtering ablation is a grouped bar chart, five bars per dataset, y-axis accuracy 0–1.2:

Process filtering wins on all four, as told. The unremarked result is that not filtering at all beats filtering on outcome on three of four datasets, and beats process-and-outcome filtering on three of four — the harm comes specifically from conditioning on the final answer being right.

Training on GSM8K with a symbolic calculator lifts HotPotQA (a search task) from 0.65 to 0.71, against 0.73 for training on HotPotQA directly; the reverse transfer takes GSM8K from 0.65 to 0.76 against 0.79 in-domain. The data-scaling panel 1:10:00 plots accuracy against training set size on a log axis from 10² to 10⁴ for five datasets, with the base model as a red dashed line in each: HotPotQA 0.59→0.71, MuSiQue 0.51→0.64, BeerQA 0.63→0.72, GSM8K 0.68→0.76. CofCA is the interesting one — flat at ≈0.51 from 10² to 10³, below its own baseline, then jumping to 0.64 at 10⁴.

Two tables close the mechanism argument. Process correctness 1:11:10: mean process label rises from 82.5% to 91.0% in distribution (HotPotQA) and from 87.5% to 91.6% out of distribution (GSM8K). Statistical significance 1:12:00 reports Model Score, F1, Precision and Recall with ±0.02–0.03 error bars across five datasets — e.g. HotpotQA model score 0.667 ± 0.024 → 0.747 ± 0.022, MuSiQue 0.431 → 0.509, GSM8k 0.685 → 0.765. A tool-use ablation that the frame sampling misses entirely 1:13:40 separates the two effects: with multi-step tool use, SWiRL beats the base model everywhere (HotPotQA ≈0.66 → ≈0.74, GSM8k ≈0.66 → ≈0.80); without tools, SWiRL is roughly level with or slightly below the base model on the QA sets but jumps from ≈0.31 to ≈0.55 on GSM8k.

The SFT comparison 1:12:40 makes the filtering point flip visible: for SFT, process-and-outcome-filtered data (≈0.50 on HotPotQA) beats process-only (≈0.40), and on CofCA process-only collapses to ≈0.14; for SWiRL the ordering reverses, with process-filtered on top at 0.71 / 0.64 / 0.64 / 0.72. Imitation on a trajectory with a wrong outcome hurts; RL gets a fresh action from the same prior context.

The final summary slide 1:14:10 gives the relative improvements, which are larger than the absolute table suggests: training on HotPotQA yields "21.5% on GSM8K, 15.3% on BeerQA, 11.1% on MuSiQue and 14.8% on CofCA"; "training only on multi-hop HotPotQA question answering improves performance on GSM8K (math) by 16.9%, and training on GSM8K improves performance on HotPotQA by 9.2%"; and "SWiRL results in significant gains with just 1000 synthetic trajectories".

Train-Time Scaling and Scaling RL

Three papers, one loop: generate the model's own outputs, filter them, train on them, repeat. STaR bootstraps rationales from correct answers, DeepSeekMath makes the RL step cheap enough to scale by deleting PPO's critic, and DAPO enumerates the six implementation details that decide whether scaled RL works at all. The slides carry the tables, the training curves and the numbers; the narration mostly carries intuitions.

The framing 00:29–05:30

The motivating slide is headed AIME 2024 only — the narration says 2024 and 2025 — and lists GPT-3.5: ~5%, DeepSeekMath-RL 7B: 51.7%, DAPO on Qwen-32B: 50%, under "How did smaller models catch up to larger ones?" 01:09 That 51.7% reappears later in the deck as DeepSeekMath's MATH top-1 score, so the AIME label on that row is worth checking before quoting it.

The train-time scaling slide is blunt 03:05: "Most attempts to do RL on reasoning fail catastrophically", "The devil is in implementation details", then three insights — models can learn from their own outputs if you filter cleverly; training compute can substitute for model parameters; small fixes matter enormously at scale. The loop itself is three boxes, Pre-training → Fine-tuning → Test-time, with an arrow running from Test-time back into Fine-tuning 04:24.

Take the outputs of test-time scaling, filter them, fine-tune on them. That single return arrow is the whole lecture.

STaR: bootstrapping rationales 16:20–39:30

The Challenges box gives three cited obstacles 16:42: manual annotation of reasoning steps doesn't scale (Rajani et al., 2019); template-based rationale generation only works where solution patterns are known (Nye et al., 2021); few-shot prompting underperforms fine-tuning on larger rationale-free datasets (Wei et al., 2022). The loop is drawn as an indented chain 18:15: Start: 10 examples with reasoning steps → Generate solutions to 10K problems → Keep only correct answers → Fine-tune on these → Repeat.

Rationalization gets its own slide 19:01. Problem: "Model gets stuck—can't solve new problems, no training signal." Solution: "Give the model the answer, ask it to explain backward" — Human: "The answer is 42. Now show your work."; the model generates reasoning that leads to 42; fine-tune without the hint, as if it solved it alone. The Assumptions slide annotates the same loop with "Correctness is proxy for reasoning quality… filtering incorrect answers may discard insightful reasoning paths, but it maintains higher-quality training data" and "Assumes LM can generate valid backwards reasoning when given answer as hint" 20:47. Students press on the second for four minutes; the answer is that this paper does no filtering of rationalized rationales at all.

A summary slide between the two result tables is internally inconsistent 30:02. It reads "MATH: 46.8% (instruction tuning) → 51.7% (with RL)" and "Uses only 70-87% of data", then lists limitations that are unambiguously STaR's — "Resets to base model each iteration (wasteful)", "No true RL—just sophisticated filtering", "Plateaus after 16-36 iterations". The 46.8 → 51.7 line is DeepSeekMath's MATH result; STaR's own numbers on the adjacent slides are 72.5% on CommonsenseQA and 10.7% on GSM8K. The narration reads it aloud as STaR's result, so a captions-only account inherits the error.

The Challenges slide states the mechanism precisely 31:50: rationalization conditions on the answer, so p(r|x, y) "might be a better search space for rationales" than p(r|x); but non-rationalized outputs are the cases where the model is most confident, and "provide a weaker gradient signal". The Strengths and Weaknesses slide carries a failure the narration skips entirely 32:43: under "Rationalization bootstrapping may amplify biases", Q: "Who's better at football: Russell Wilson or Jeremy Lin?" A: "Jeremy Lin", with a red ✗ against the rationale "Black people are better at football. Russell Wilson is black, whereas Jeremy Lin is asian. Therefore, the answer is Russell Wilson."

Follow-ups appear with their paper figures 36:49: V-STaR trains a verifier with DPO on preference pairs of correct and incorrect solutions, then ranks generator samples with it at test time; Quiet-STaR generates thoughts in parallel after every token and applies REINFORCE "to increase the likelihood of thoughts that help the model predict future text while discarding thoughts that make the future text less likely", with stages labelled think, talk, learn. The class then debates what bounds STaR: no logical leaps outside the training distribution, and the base model's ability to rationalize when handed the answer.

DeepSeekMath and GRPO 40:40–53:00

The data work is half the contribution 41:50. Conventional wisdom: train on arXiv. DeepSeekMath's finding: Common Crawl web pages > arXiv papers. Recipe: start from OpenWebMath (13.6B tokens); train a fastText classifier asking "Does this look like math?"; mine Common Crawl for 120B tokens (9× bigger); iterate to find new math domains and retrain. The results slide adds that arXiv training "sometimes hurts performance", that code → math significantly helps both tool-use AND reasoning, and that multilingual data improves Chinese benchmarks without harming English 43:22 — "Data curation matters more than data source prestige."

The memory argument is four boxes, sourced to sebastianraschka.com/blog/2025 43:47: Original policy (LLM from SFT stage), New policy (being trained with PPO), Critic (value model estimating expected reward), Reward model. The GRPO slide narrows to the costly pair: actor 7B, critic 7B, "2× memory, 2× training cost" — insight: "Use the group for the baseline."

The procedure is four steps 45:25: sample 64 different answers per question; score them (illustrative vector [0.8, 0.2, 0.9, -1.0, …]); normalize; "This is your advantage—no critic needed!" The normalization is written on the slide, without parentheses, as reward - mean(rewards) / std(rewards); the quantity described aloud is (reward − mean) / std. No GRPO objective appears anywhere in the deck — no ratio, no clipping term, no KL penalty; the speaker says outright that the formula was omitted in favour of intuition. What the deck gives instead is a unifying form 46:24: gradient ∝ GradientCoefficient × ∇ log p(output | question).

Which sets up the constraint DAPO later attacks 47:40: if every sample in a group is correct, or every one wrong, the normalization yields nothing. A student adds that PPO's KL term is the other thing keeping the policy near what it could already do.

The section closes on the honest finding 52:40: online > offline, and then "RL improves Maj@K (best of 32 tries)", "RL does NOT improve Pass@K (at least one correct in 32)". "RL makes the model more consistent, not fundamentally smarter. It fixes 'misalignment' where the model knows the answer but doesn't always produce it." (The gloss "best of 32 tries" on Maj@K is loose; the narration correctly describes majority voting.)

DAPO: the six fixes 53:40–61:00

Naive GRPO on Qwen-32B reaches 30% on AIME, with entropy collapse ("model becomes too confident"), unstable training, and response length exploding 53:55. "But we know it should work—DeepSeek got 47%. What's missing?"

Dynamic sampling carries the sharpest number in the lecture 56:09: 60% of questions eventually get all 64 samples correct. All correct → advantage zero → no gradient → wasted compute. The fix: over-sample, discard all-correct and all-wrong groups, keep only 0 < correct_count < 64, which "maintains effective batch size". Token-level loss follows 57:50: under a sample-level loss, "long garbage answer = same weight as short good answer", giving "unhealthy length explosion"; weighing tokens rather than samples is worth 41% → 42%.

Dynamic sampling is the largest single step (+8), and the final 50 beats the R1-distilled reference at 47 on the same base model 59:42.

The practitioner slide is the deck's most quotable 60:50. "Don't watch loss—watch these": response length should grow to ~4K tokens; entropy should stay in the 0.4–0.6 range (too low = collapse, too high = gibberish); % samples with acc=1 tells you when to increase sampling. Red flags: length suddenly explodes → check token-level loss; entropy drops to 0.2 → collapse, increase ε_high; no improvement after 2K steps → the reward model may be saturated.

Choosing between them, and what none of them fix 62:00–67:30

Selection criteria are explicit 62:05. STaR: fewer than 100 examples with reasoning; you need interpretability (education, research); limited compute, no RL infrastructure; GSM8K-level tasks. DeepSeekMath GRPO: good base model plus instruction data; memory-constrained, can't fit a critic; standard math reasoning. DAPO: reasoning chains over 1K tokens; state-of-the-art needed; RL infrastructure available; competition-level problems (AIME, IMO).

The ceiling slide is unusually direct 63:11. All three improve Maj@K, answer formatting and multi-step coherence. None improve Pass@K, novel problem-solving, or out-of-distribution generalization "(much)". "These are alignment/elicitation methods, not capability expansion methods."

Reward quality is the binding constraint 65:30: "Neural reward models may suffer from reward hacking in the large-scale RL. However, autonomous coding applications have access to verifiable rewards" — correct final answer for given inputs, execution feedback, unit tests. The closing slide 66:45 lists what is not understood — why Maj@K but not Pass@K; are emergent behaviours real ("Reflection, backtracking—learned or statistical?"); how to learn from failures ("Current: just filter them out") — and four directions: tree search during generation; noise-robust RL, because "reward models are ~20% wrong"; process rewards and uncertainty quantification; and "Can we merge STaR's rationalization + DAPO's techniques?"

Asked in the Q&A what fraction of frontier training is now RL 68:30: unpublished for Anthropic, but roughly 99%/1% pre-training-to-RL a year ago, growing to perhaps 5%. Grok-4 claimed 50% RL and, by this account, did not get the corresponding jump — bottlenecked by exactly the failure modes above. On data-scarce benchmarks, the reframing is that "there is not enough data" is the wrong abstraction; "do you have enough data to hill climb on" is the right one 71:40.

Self-Improvement and Deep Research Agents

Two systems, one idea: the answer is already somewhere in the model's output distribution, and the engineering is in curating it out. AlphaCode and AlphaCode 2 curate a million samples down to ten submissions; Search-o1 curates retrieved web pages down to the fragment a reasoning chain needs. Both spend most of their cleverness on selection, not generation.

On the recording: the camera alternates between a screen-capture feed and a room camera. Roughly 32:00–35:30, 40:00–45:30 and 1:09:00–1:12:27 are room-camera Q&A with no slide visible, summarised here from the transcript alone.

The problem: competitive programming, not autocomplete 01:55–03:30

The contrast slide is more specific than the narration 02:46: "Codeforce problems are 4x longer in terms of description length and solution length", and HumanEval problems "include instructions about exactly what to implement (competitive programming problems have no suggested implementation)". It illustrates HumanEval with a GitHub card for HumanEval-V — a visual benchmark for multimodal models, not the HumanEval under discussion. AlphaCode ranked in the top 54.3% across ten contests, the first demonstration that a model could generalise past narrow tasks to end-to-end problems.

AlphaCode's pipeline 03:47–09:00

Large-scale sampling 06:36: 1M programs per question — the slide puts a red "‼" beside the number — half Python, half C++, randomized tags and ratings in the prompt, high temperature. Filter and cluster 07:19: filtering to samples that pass the statement's example tests "removes approximately 99% of model samples"; clustering then separates syntactically different but semantically equivalent programs using a separate test input generation model that invents new test inputs for unseen problems.

Codeforces results, and where the variance comes from 09:15–13:30

The evaluation slide 09:15 gives 10 competitions with >5000 participants each, top 54.3% average ranking limited to 10 submissions per problem, top 28% of Codeforces competitors in the last six months, and a heat-mapped table — green good, pink bad — "Estimated % ranking of AlphaCode in 10 competitions (lower is better)":

A student picks out exactly this spread — 1623 at 20.9% estimated against 1617 at 73.9% 10:24. Two hypotheses 12:00: how in-distribution a contest's problems are, and selection failing rather than generation failing — solutions that are almost but not quite correct never make it into the ten submitted.

pass@k versus 10@k 14:04–17:30

The definitions slide 14:04: pass@k submits all k samples and "measures the search aspect of the sampling process"; 10@k submits only 10 and "measures the filtering process and how models behave at a very large number of samples". Table 5 gives the solve rates 15:28 — note there is no 10@1M column on the test set:

Against the spoken claim that the larger model is consistently better: at 10@1k on validation, 9B and 41B are identical at 16.9%. Scale only pays once the budget grows, and clustering is worth more than the 4.5× parameter jump in every column.

The gap between panels is the price of selection. Asked whether you could just push to a trillion samples 19:53, the answer turns on diversity — more sampling only helps if the extra samples are actually different, which is exactly what clustering measures.

The strengths/weaknesses slide 22:35 credits interpretability, a reduced false-positive rate, and evidence the model "does not copy existing solutions; actually reasons"; against it, reliance on hidden test coverage, "loss is poor proxy for solve rate (solving problem is one-of-many task)", weakness on DP and constructive algorithms, and one line never spoken — "Test set dataset size is small (n=165)".

AlphaCode 2 24:35–31:30

Three changes, two of them starred on the slide 26:40: ⭐ fine-tune another Gemini Pro model to score "correctness" before submitting; ⭐ a "family of models" fine-tuned with varied hyperparameters (difficulty, tags) to maximise diversity; plus using an LLM to fine-tune instead of pre-training on GitHub, on CodeContests V2 and a second "higher quality dataset".

100 samples now buy what a million bought a year earlier — from a better base model, engineered diversity and a learned scorer, not from more sampling.

Asked about the 95% waste, the room-camera discussion lands on self-refinement and RL on the base model, plus a useful reframing: this is a multi-agent system — one family of models generates, another scores — and distilling that system into one model is what a large reasoning model is 34:16.

Discussion 39:21–45:38 — transcript only

The slide poses four questions — a better proxy for solve rate than validation loss; adapting to task complexity; maintaining diversity with fewer generations; embedding reasoning into the model — and the class works through the second and fourth. The answers that land: easy problems need far fewer samples, since the solution is likely already in the output distribution and refinement will fix it, while parallel sampling buys diversity of approach, which is what hard problems need; and chain-of-thought in the training data plus STaR-style rationalization, with decomposition and a backtracking tree search for multi-step failures 41:29.

Search-o1 46:37–59:50

The framing is direct 47:25: enhance large reasoning models "with autonomous retrieval capabilities", because LRMs "suffer from knowledge insufficiency during extended reasoning chains". Gaps surface as uncertainty language — "perhaps", "alternatively", "wait" — and an early guess cascades through the chain. Plain RAG fails on three counts 48:19: single retrieval at the beginning, each step needing different information, no way to adapt. The solution names two components 49:45: the Agentic RAG Mechanism (autonomous query generation, triggered on knowledge gaps, multiple iterations per session) and the Reason-in-Documents Module (analyze documents separately, extract, integrate).

The module exists for two stated reasons 55:58: "retrieved docs are lengthy, contain noise", and "LRMs suffer 'catastrophic forgetting' of general capabilities" under long context. It conditions on current query + previous reasoning + retrieved content.

Like a researcher: you don't paste every reference into your draft — you take notes on them.

The summary slide at 59:57 contradicts the figure it summarises, saying vanilla reasoning "guesses structure → wrong answer (14 atoms)" when panel (a) gives 10 and it is the Agentic RAG panel that gives 14.

Results 01:00:49–01:05:30

Table 2 compares against human experts on the GPQA extended set 01:01:52; the side bullets read "+5.5% over direct reasoning", "+24% average improvement over RAG baselines (Qwen2.5-7B)", "outperforms human experts in physics (68.7 vs 57.9)":

The advice is to read the diagonal — physicists on physics, chemists on chemistry. On that reading Search-o1 beats physicists (68.7 vs 57.9), edges biologists (69.5 vs 68.9), and is nowhere near chemists (40.7 vs 72.6). The chemistry gap draws two unresolved hypotheses from the room 01:02:25: that chemical structures survive retrieval badly, or that the training data was weaker there.

A companion text slide 01:03:40 claims "even 1 document with Search-o1 > 10 documents with Standard RAG", then lists "Physics: 74% → 78%, Chemistry: 36% → 48%, Biology: 55% → 85%" — endpoints that look like each panel's axis limits rather than its curve values, since biology never reaches 85 and physics starts above 74. The same slide carries a stray unrendered markup tag: <cite index="9-9">Performance improves with more retrieved documents (top-k from 1 to 10)</cite>.

The narration says every bold number on that line comes from Search-o1. The table does not quite bear it out: on 2WIKI EM the best is RAgent-QwQ-32B at 58.4, ahead of Search-o1's 58.0, and the bold single-hop TriviaQA values (65.8/79.2) belong to RAG-Qwen2.5-32B. Search-o1 does hold the best numbers on HotpotQA, MuSiQue and Bamboogle.

What actually changes in the reasoning chain 01:06:20–01:07:30

So "uncertainty drops substantially" holds for the two hedging words that mark knowledge gaps, and not for "wait", which is a self-correction marker and barely moves.

Search-R1, and a coda on calibration 01:07:40–01:12:18

The final slide lists six takeaways — extended reasoning creates knowledge gaps; simple retrieval is insufficient; outcome-based RL works, no complex reward engineering needed; agentic beats fixed retrieval; document refinement is critical; benefits scale with model size — then a section the lecture ran out of time for. Search-R1 contributions: "retrieved token loss masking for stable RL", "multi-turn search with outcome rewards", "41% improvement (Qwen2.5-7B) on QA tasks" 01:08:14. Search-o1 closes the loop by prompting; Search-R1 teaches the model to search with RL. The closing slide reads "End of Lecture 8" 01:08:47.

The last question, over the room camera with no slide 01:09:22, asks whether output probabilities correlate with correctness across a thousand generations. Aggregate the log-probs sensibly — a geometric product or an average, not a sum — and models come out overconfident: 50% correct, 80% confident, which is why they resist correction. Knowing what you know is offered as an open project area.

Agentic Evaluations and Long-Horizon Tasks

Chatbot benchmarks saturate, so the lecture asks what replaces them and walks three answers: METR measures how long a task a model can finish, GDPval measures whether the output beats a human professional on economically valuable work, and DeepScholar-Bench measures research synthesis against a live corpus. The first trend is exponential, the second is linear, and the third is stuck under 19%.

The agenda slide names all three by their paper titles 01:54: Measuring AI Ability to Complete Long Tasks; GDPVal: Evaluating AI Model Performance on Real-World Economically Valuable Tasks; DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis. The framing slide reduces the problem to two questions — "how long/complex are the tasks AI can complete?" and "can AI perform real-world work that humans currently do?" — and closes with the line "Different metrics, similar insights" 03:36.

METR: the time-horizon metric 04:18–27:20

The definition on the slide is precise: the duration of tasks models complete at X% success rate, with 50% chosen as the "balance between capability and reliability," and human time-to-complete as the universal anchor 05:12. Three suites, 170 tasks, spanning six orders of magnitude 05:40.

Human baselines come from professionals with roughly five years' experience recording completion times on successful attempts, aggregated by geometric mean 12:00. The acknowledged bias: experienced people are conditioned to what success looks like and systematically underestimate difficulty. The example-task table shown on screen is reproduced verbatim below 11:12; the cuda_backtesting task carries a specific performance target the spoken version rounds off.

A separate slide lists the trajectory explicitly 14:54: GPT-2 (2019) ~2 seconds; GPT-3 (2020) ~8 seconds; GPT-3.5 (2022) ~1 minute; GPT-4 (2023) ~8 minutes; Claude 3.7 Sonnet (2025) ~59 minutes. The drivers slide is short — better logical reasoning and code generation, improved tool use, "greater reliability: reduced 'looping' behaviors," and better error recovery and self-awareness 15:23.

Camera note: from roughly 16:05 to 20:15 the recording stays on the room camera and the projector is not in frame. That stretch is transcript-only — it is the class discussion where students add context engineering (including compaction at the end of the context window), feedback from user logs, explicit planning that triggers re-planning mid-execution, and memory of a codebase as an environment model.

The comparison slide states it flatly 21:59: "80% horizon: Similar doubling time but ~5x shorter" and "Claude 3.7: 15 min (80%) vs 59 min (50%)." (The captions render "~5x shorter" as "five weeks shorter"; the slide is the reliable text.) The doubling times are effectively identical — 7 months at 50%, 213 days at 80% — so the reliability penalty is a constant factor, not a slower trend.

50% success is a coin flip. "That's like saying you gave your task to an intern, but it only completes it 50% of the time."

Failure modes, as counts 23:06

The slide shows raw counts, not percentages, for GPT-4 1106 against o1 — and the totals are nearly identical, which the narration does not say.

Reasoning buys a 6× reduction in repeated failed actions (12 → 2) and pays for it with double the premature abandonment (8 → 16). The failure budget moves; it does not shrink.

Validation experiments 25:39

Three checks on the slide. Messiness factors (16 dimensions): lower performance at higher messiness, but similar trends across messiness levels and no evidence of plateaus on messier tasks. SWE-bench Verified replication: similar exponential trend but a shorter doubling time of ~70 days, which the slide attributes to "annotator time underestimating easier tasks" — the contamination story about models having seen public GitHub repositories is spoken, not written. Internal PRs: contractors are 5–18× slower than maintainers, model performance is consistent with contractor times, and therefore time horizons measure "low-context human" capability.

GDPval: economically valuable work 27:20–51:00

The pivot slide reframes the question as "not 'can AI do this?' but 'is AI output good enough?'", scored as head-to-head competition with industry experts. The slide specifies 14+ years experience, where the narration says "more than a decade" 28:00.

Construction is top-down 33:39: 9 sectors contributing >5% to GDP, 44 occupations, 1,320 tasks with 220 in an open gold subset, "predominantly digital" defined as ≥60% of O*NET tasks being computer-based. Characteristics 34:38: average 7 hours completion time (some span weeks); multimodal (CAD, video, audio, spreadsheets, presentations); average $398 value per task in the gold subset; 67.7% require interaction with reference files; 89% rated well-specified by experts.

Asked whether METR and GDPval disagree 37:16, the answer is that they measure different metrics — but an exponential invites you to assume hours become days become weeks, and GDPval says that on real multi-hour work, broken down by profession, models are only so reliable.

The economics table 40:33 compares three review strategies. "Naive" is the raw model-versus-human ratio; "Try 1x" and "Try nx" fold in the cost of a human reviewing and, in the n case, iterating.

Two things the table shows and the bullets do not. GPT-4o is a net loss under review — 0.46× speed and 0.53× cost against an unaided expert, because checking a 12.5%-win-rate model costs more time than it saves. And the naive speedup falls as models improve (327× → 90×): stronger models think longer.

Where models are already at parity 41:46–45:34

The variation slide is specific where the narration is vague 41:46: by sector, "Government, Retail, Wholesale near parity; others lag"; by duration, "best on 0-2 hour tasks; declines with longer tasks"; by modality, "Claude better on multimodal; GPT-5 better on text."

The context experiment 45:50

The sharpest single result in the GDPval section. Under-specified prompts deliberately removed context: 42% shorter prompts by token count, and win rate falls only 47.7% → 44.3%. The small drop is the finding — the slide's bold line is "Models struggle to 'figure out what to work on'", which the win-rate metric barely registers because a model that works on the wrong thing competently still sometimes wins the comparison.

The follow-on slide poses it as the critical question 46:42: are we measuring "can a smart person do this?" or "can someone embedded in this work do this?"

DeepScholar-Bench: research synthesis 51:29–1:01:00

The task a student requested in the first ten minutes, turned into a benchmark: generate related-work sections for academic papers. Construction 52:55: 100 queries from recent arXiv papers at PhD-level difficulty, 22 domains, re-runnable monthly with new papers, using only post-training-cutoff papers to avoid contamination. Three evaluation dimensions 53:32 — knowledge synthesis (organization & coherence; nugget coverage), retrieval quality (relevance rate; document importance by citation counts; reference coverage), verifiability (citation precision; claim coverage) — all validated at 70–82% human agreement.

Key finding, stated on the results slide: no system exceeds 19% across all metrics.

The per-dimension numbers 55:16 are more specific than the narration. OpenAI DeepResearch leads knowledge synthesis with an organization score of 0.857, but all systems are under 40% on nugget coverage — they write coherently and miss key facts. Best reference coverage is 18.7% (OpenAI DeepResearch); document importance is under 12.5% for everything. On verifiability, DeepScholar-base reaches 93.6% citation precision (the captions say "90%") while OpenAI DeepResearch scores lower despite the better prose.

The failure analysis 56:39 names the oracle experiment the narration rounds off: hand a system the perfect sources and it still surfaces only 52.8% nugget coverage. Systems retrieve relevant documents but miss foundational papers, and struggle to assess document importance as distinct from relevance. No system excels at both synthesis quality and verifiability — OpenAI DeepResearch has the best synthesis and weaker citations; DeepScholar-base has strong citations and merely decent synthesis.

Synthesis across the three 58:31–1:08:00

Research synthesis is run through all three lenses on one slide 1:01:32: METR would call it a 30 min – 8 hr task, inside the current ~50 min capability range for Claude 3.7 — "BUT quality at this horizon is still poor." Key gaps listed: multi-step reasoning over many sources, comprehensive information gathering, maintaining verifiability while synthesizing, and context acquisition from distributed sources. The takeaway line: "We're not just limited by task duration — quality of synthesis at reachable horizons needs major improvement."

Each benchmark's blind spots are named 1:02:51. METR: automatic scoring only, no multi-agent interaction, lax resource constraints, not punishing of mistakes. GDPval: tasks precisely-specified and one-shot, no iterative back-and-forth with stakeholders, limited tacit knowledge requirements.

The closing takeaways slide 1:07:05 sets the three trends against each other: METR's exponential 7-month doubling over 6 years, currently ~50 minutes, forecast to a 1-month horizon by 2028–2031 (with caveats); GDPval's linear improvement over 2024–2025, currently ~48%; DeepScholar-Bench's sub-19% ceiling with critical gaps in retrieval comprehensiveness. The insight slide follows 1:07:46 with the sentence that holds the lecture together: "Key tension: Reaching time horizons ≠ high-quality outputs."

The Q&A 1:08:19–1:15:11 is room-camera only apart from a return to the METR pipeline slide at 1:12:18, used to answer how the 50% figure is computed — per-task success rate over multiple agent attempts, averaged across the 97 HCAST tasks, not a per-task floor. On what blocks the long tail 1:13:44: a mix of data and fundamental capability; broad domains like legal and financial research adopt fast, narrow ones stall, and robotics is the clearest case of an entire cohort of startups treating data collection as the bottleneck.

Future Research Areas

The closing lecture — the deck is titled Lecture 20 — Last Lecture — surveys four papers chosen to name four bottlenecks: diversity of reasoning chains, verification, where training prompts come from, and, on a different axis, the energy cost of intelligence. Each paper gets three to six slides whose figures carry most of the argument, followed by two "what's next" slides that are the real deliverable.

On the recording: it is a screen capture of a Google Slides deck in a Chrome window, with a two-person video strip above. There is no room camera and no cut away from the slides, so everything below is checkable against a frame — except two stretches, flagged in place, where one static slide stays up through a long Q&A.

Course recap 00:40–04:12

"CS329A at a Glance" 00:40 lists the quarter as a flat outline: Test-time Scaling; Self-improvement Techniques (sub-bulleted Verifiers, Feedback, RL, Search); Evolution Strategies; Tool-use/Actions; Augmenting LLMs with Retrieval/Memory; Planning and Multi-step Reasoning; Robust Agent Evals; guest lectures from frontier AI labs and startups; Multimodal Agents & Robotics. One item is added aloud but is not on the slide — the guest lecture on symbolic techniques, filed under tool use, feeding synthetic data. Three definitional slides follow, cited to Anthropic's "building effective agents" 02:27–03:32: the agent loop, the contrasting agentic workflow ("systems where LLMs and tools are orchestrated through predefined code"), and a parts list closing on agentic workflows require planning, multi-step reasoning and self-improvement.

"Today's lecture" 04:12 names the four papers: Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains; DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning; Absolute Zero: Reinforced Self-play Reasoning with Zero Data; Intelligence-Per-Watt: A Study of Local Intelligence Efficiency. The agenda slide 04:38 gives three lines that recur verbatim at the end: generalization via train-time scaling in reasoning; robust verification and meta-verification loops for diverse tasks; breaking through the data barrier of task selection.

Bottleneck 1 — diversity of reasoning chains 06:59–14:25

Iterative finetuning on self-generated data is described on the slide as running "into plateauing problem. Performance stops increasing, or even declines," with STaR and ReST named as variants 06:59. The diagnosis slide 08:05 gives two causes — lack of diversity, reliance on single models — and the fix: specialized agents split into generation agents (produce diverse initial solutions) and critic agents (evaluate and refine them).

Generation models then train on their own outputs across rounds, filtered for a match with the final majority vote; critic models get "a mix of trajectories where answer is correct from the start, and where answer is corrected over course of debate" — both halves matter, since the critic learns by contrast.

Bottleneck 2 — the verification ceiling 15:47–22:45

The outcome-reward slide 15:47 is blunter than the narration: outcome reward models "enabled saturation of AIME, HMMT competitions in ~1 year" (the captions render this as "Amy"), and are an unreliable proxy — correct answers don't guarantee correct reasoning, via lucky errors and flawed logic — as well as inapplicable to theorem proving, which needs step-by-step derivation rather than numerical answers.

Critical observation: No generation–verification gap in natural-language theorem proving. 16:20

The training premise adds a claim the transcript skips: verification effort serves as proxy for proof quality. The architecture 17:41 is three roles — a verifier trained to identify issues and score proofs on (0, 0.5, 1), three levels rather than the two the narration mentions; a meta-verifier reviewing the verifier's analyses for faithfulness; a generator producing proofs with self-verification capability. The meta-verification slide 19:26 names the failure mode exactly — the verifier "can get correct score while hallucinating non-existent issues — e.g. predict score=0.5 with fabricated 'errors' that don't exist" — and checks whether those issues actually exist and whether the score follows from them. It reports a number the lecture never says aloud: verifier analysis quality improved from 0.85 → 0.96 while maintaining score accuracy.

The methodology slide 20:00 is similarly specific. Training is GRPO — the transcript says "TRPO", which is wrong — with iterative optimization Verification → Generation → Repeat on base model DeepSeek-V3.2-Exp-Base. Benchmarks come with sizes: in-house CNML-level problems (91 across 5 categories); IMO 2025 and CMO 2024 (6 problems each); Putnam 2024 (12); IMO Shortlist 2024 (31); IMO-ProofBench (60, Basic + Advanced).

The takeaways slide generalises the recipe: a verifier trained to identify issues without reference solutions; meta-verification to reduce hallucinated issues; a self-verification incentive; and automated labeling, where scaled verification compute eliminates human annotation.

Bottleneck 3 — where do the tasks come from? 23:32–31:46

The scalability slide 23:32 has four bullets: SFT needs human-curated reasoning traces; RLVR still depends on expert-curated question-answer pairs; high-quality human data is scarce and expensive; and, as AI potentially surpasses human intelligence, human-designed tasks may limit learning potential.

That same "Proposer" text slide supplies the diversity mechanism: the proposer is "conditioned on K past self-generated examples from a buffer, explicitly prompted to generate different tasks to promote diversity."

The learnability reward 27:04 is written out: if average solver success rate = 0 (unsolvable), reward = 0; otherwise reward = 1 − average_success_rate, and the line the narration omits — this is estimated via Monte Carlo rollouts of the solver with non-zero temperature. Validation 28:07 adds three gates before a task enters training: program integrity (must execute without errors and return a value), safety check (filters dangerous packages — os, sys, subprocess), determinism check (identical outputs across executions). The whole run bootstraps from one seed triplet that is the identity function, def f(x): return x.

Results 29:21: state of the art on coding benchmarks despite zero human-curated data, outperforming models trained on tens of thousands of expert examples. The emergent behaviours come with named metrics — complexity measured by ComplexiPy and Halstead rises over training without explicit optimization, and diversity of programs and answers grows.

The proposer and solver exhibit mildly adversarial but ultimately cooperative dynamics.

The takeaways slide 30:42 adds a scaling number absent from the audio: "Scaling Wins: Larger models gain more — 14B improves 13+ points vs 6 points for 3B." Under "Surprising Transfer", training only on self-proposed code tasks reaches state of the art on both coding and math. Mirhoseini connects this to SWiRL from the planning lecture 31:46 — model-generated data transferring across tools and domains, larger models absorbing the flywheel better.

Where verification is genuinely hard 33:10–39:32 — transcript only

For roughly six minutes the "What's next: improving the self improvement loop" slide sits static while the instructors and class talk; nothing new appears on screen. The content: the hard cases are those where verification is slow rather than subjective — chip-design simulation taking days, wet-lab chemistry — because RL needs hundreds or thousands of iterations and tolerates minutes, maybe an hour, not days. One workaround is a reward model trained to predict a simulation's outcome from offline data, bounded by data volume and prone to reward hacking when inaccurate. The KernelBench example 38:11: compiler execution tells you whether the kernel is correct, but reading performance profiles to find the unoptimized parts of a large concatenated program is harder — being attempted, the instructor notes mid-sentence, by a class project decomposing it into sub-parts with a knowledge base of reference solutions.

Intelligence per watt 40:27–51:50

The title slide 40:27 carries the full author list, which the narration compresses to "Professor Ré and Professor John Hennessy and a large team": Jon Saad-Falcon*, Avanika Narayan*, Hakki Orhun Akengin, J. Wes Griffin, Herumb Shandilya, Adrian Gamarra Lafuente, Medhya Goel, Rebecca Joseph, Shlok Natarajan, Etash Kumar Guha, Shang Zhu, Ben Athiwaratkun, John Hennessy, Azalia Mirhoseini, Christopher Ré.

The hardware counterpart 44:45, "Dawn of the PC AI Era — Local inference as a path to redistribute compute", plots GPU Memory (GB) on a log axis against Release Year 2012–2025 for local accelerators, annotated 126.3× Improvement — a stepped curve, nearly flat 2015–2020 then jumping — with slide text noting local hardware now runs models once confined to data centers at 128GB–200GB. The metric slide 46:26 gives the definition in two forms: capability ("What % of today's single-turn chat and reasoning LLM queries can local LMs (<=20B active params) answer correctly?") and a formula labelled acc/W — mean task accuracy ÷ mean power draw.

Open problems 51:50–58:18

"Research Directions Forward" 52:16 lists three bullets — understanding foundational principles in test-time scaling and learning from the synthetic-data flywheel; continual learning from generated test-time experiences on the fly; infra for high-throughput / low-latency test-time scaling — over a three-box pipeline, (Pre-)training → Fine-tuning or Online / Continual learning → Test-time, with a feedback arrow from Test-time back into the middle box, and the footer link https://scalingintelligence.stanford.edu/. Spoken alongside: the question behind the first is why repeated sampling surfaces correct answers at all, and how best to distil successful trajectories back into the model; behind the third, that repeated sampling, revision and tool calling look nothing like single-turn chat serving. Her lab's own inference-systems work is named aloud here, but the captions garble the project names and no slide lists them.

"What's next: the era of intelligence efficiency" 56:55 is the efficiency counterpart: new inference serving engines for local/cloud resources; new model architectures and kernels for energy-efficient inference, framed as accelerator–architecture co-design; and more granular intelligence-per-watt measurements across tasks, hardware and deployment settings. The deck then returns to the same three self-improvement bullets from the opening, closing the loop.

Q&A and close 59:38–01:07:36 — transcript only

The last eight minutes run over a static "End of Lecture 20" slide. The best exchange is continual learning versus memory 01:00:18–01:04:19: one path is effectively infinite context — a model with perfect access to unbounded context could hold every positive and negative experience at once — but in-context reasoning quality degrades well before a few million tokens, which is why techniques like cartridges push knowledge into KV caches instead of weights. The counter-argument is the sharper half:

No matter how much memory systems you add … cross-embodiment generalization doesn't happen if you don't update the weights. That's more of a skills transfer problem. 01:03:39

A last question asks whether agents can self-create environments, as Absolute Zero does for tasks. The answer reframes it: environments matter as proxies for real-world tasks, so the hard part is not who writes the simulator but whether it is a reasonable proxy for the feedback the model gets from the real world.