Day 18 — Bonus April 28, 2026

Reasoning Models & Test-Time Compute for Agents

After the 17-day arc, the topic that’s quietly re-architecting every layer underneath: a third scaling axis that buys capability not from more parameters or more data, but from more thinking. Why o-series, R1-class, and budget-forcing models force you to rethink the ReAct loop, the eval harness, and your unit economics — together.

o-series
DeepSeek-R1
RLVR
Budget Forcing
Inference Scaling
CoT Faithfulness
Tool Hallucination
Curriculum
Bonus · Day 18 of 17 + 1
Core Concept

Test-Time Compute — the Third Scaling Axis That Eats the Agent Loop

For a decade the deal in deep learning was simple: better answers come from more parameters (model scaling) and more data (data scaling). In 2024 a third axis became visible and, by April 2026, dominant: more thinking at inference time. A reasoning model is one that has been trained — usually via reinforcement learning with verifiable rewards (RLVR) — to spend variable amounts of compute generating private chain-of-thought tokens before producing the user-visible answer. The accuracy / compute tradeoff is now a dial, set per-request, not a property fixed at training time.

Analogy first. Think of a senior engineer reviewing a tricky migration. Junior version: blurts an answer. Senior version: spends fifteen quiet minutes drawing on a whiteboard, sketches three plans, eliminates two, sanity-checks the third against edge cases, then writes the answer. The whiteboard never leaves the room. Same person, same knowledge — the difference is thinking budget. A reasoning model lets you set how big that whiteboard is, and how much time the model spends in front of it, on every single call.

Mechanism. A reasoning model is an autoregressive LLM that has been post-trained on a long-form chain-of-thought objective. The dominant recipe, popularized by DeepSeek-R1 (Jan 2025), is roughly: (1) cold-start with a small set of long, high-quality reasoning traces; (2) run large-scale RL where the only reward signal is whether the final answer matched a verifier (math equality, unit-test pass, JSON schema fit); (3) let the policy gradient discover, on its own, that “wait, let me reconsider”, self-checking, and backtracking are strategies that increase reward; (4) optionally, distill the long traces into smaller open models. The result: longer responses on hard prompts, near-identical responses on easy ones, with a smooth log-linear improvement curve as you give the model more output tokens to work with.

The three knobs you now control at inference

1
Knob
Thinking budget (max reasoning tokens)
How many private CoT tokens may the model spend before it has to commit?
In Claude this is the thinking.budget_tokens parameter; in OpenAI it’s reasoning.effort (low / medium / high); in Gemini it’s the “thinking budget” setting; in open models like s1-32B it’s implemented as budget forcing — literally appending "Wait" to extend, or "Final Answer:" to truncate. Higher budget = better on hard prompts, more latency, more cost.
2
Knob
Parallel sampling (best-of-N)
How many independent reasoning trajectories do we sample, and how do we pick a winner?
Snell et al. (2024) showed that the optimal allocation between “deeper one trajectory” and “wider many trajectories” depends on prompt difficulty. Easy prompts: one trajectory, light budget. Hard prompts: many parallel trajectories, vote with a learned verifier or majority-of-answers. This is why you sometimes see n=8 or n=64 in modern agent traces — it’s the parallel knob.
3
Knob
Tool / search interleaving
Does the model think first then act, or interleave reasoning with tool calls?
First-generation reasoning models (o1) thought silently then emitted one answer. Modern variants (o3, Claude extended thinking, Gemini 2.5 Pro thinking) interleave: think → call a tool → observe → think more. This is ReAct, but with budgeted reasoning steps. The agent loop didn’t go away — it now has a thinking-tokens column in addition to a tool-calls column.

Why this matters for agents specifically

An agent is a loop: think → act → observe → repeat. Pre-reasoning-models, every “think” step was the same size — one forward pass, one short response. The whole burden of “harder problems need more thinking” fell on more loop iterations, which meant more tool calls, more state, more places to fail. Reasoning models compress some of that loop into a single call. A task that used to take eight ReAct steps now takes three, but each step is a fatter-tailed compute draw.

The architectural shift inside frameworks: LangGraph’s deepagents harness, OpenAI’s Agents SDK, and Anthropic’s Claude Agent SDK have all moved from “short turn, many turns” toward long turn, fewer turns, with explicit thinking-token accounting. The eval harness has to evolve with it: you no longer measure “steps to solution” alone; you measure (steps, thinking tokens, tool calls, $$ spent) jointly, and the Pareto frontier across those is the real product surface.

# Pre-reasoning: many short steps Plan → Tool → Plan → Tool → Plan → Tool → Plan → Answer 8 short LLM calls, ~600 tokens each # With reasoning: long steps, fewer of them [Think 12K tok] → Tool → [Think 4K tok] → Answer 3 calls; 1 fat reasoning, 1 thin verify # Cost picture old: 8 × (in+out) // short calls, low p99 new: 3 × (in+OUT) // fewer round-trips, fatter tail
The right mental model: reasoning models trade orchestration complexity for per-call latency variance. Your framework problem gets easier; your latency-budget problem gets harder.
Numbers Worth Memorizing
15.6 → 71.0
DeepSeek-R1 AIME 2024 pass@1, before vs after RL (paper)
+27%
s1-32B over o1-preview on competition math (paper, Jan 2025)
~10×
Output-token ratio: reasoning vs non-reasoning on hard prompts

Sources: arXiv 2501.12948 (DeepSeek-R1) and arXiv 2501.19393 (s1). The ~10× figure is a representative ratio reported in vendor docs and replication writeups; exact value varies by prompt class and budget setting.

Papers to Know

Three Papers That Explain Where Reasoning Models Came From and Where They’re Going

Published · Nature Seminal · 2025
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
DeepSeek-AI (Daya Guo, Dejian Yang, et al., 200+ authors) · arXiv 2501.12948 · Jan 2025 · published in Nature, Sept 2025
First openly published recipe for an o1-class reasoning model. The headline contribution is “DeepSeek-R1-Zero,” trained with pure RL on verifiable rewards starting from a base model with no supervised reasoning data — and showing that self-reflection, verification, and backtracking emerge as policy-gradient strategies, not as imitation of human traces. The full DeepSeek-R1 then re-introduces a small cold-start dataset for readability. AIME 2024 pass@1 jumps from 15.6% to 71.0% (86.7% with majority voting), matching OpenAI o1.
Why it matters: Made the recipe public. Within four months, every major open-source lab had a clone (Open-R1, SimpleRL, SkyThought) and the assumption that “reasoning ability is a closed-lab moat” collapsed. Also — and rare for an arXiv paper of this size — the work was peer-reviewed and accepted to Nature.
arxiv.org/abs/2501.12948 →
arXiv preprint · 2024 Seminal
Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters
Charlie Snell, Jaehoon Lee, Kelvin Xu, Aviral Kumar (Google DeepMind / UC Berkeley) · arXiv 2408.03314 · Aug 2024
The paper that named the third scaling axis. Studies two test-time compute strategies — sequential revision (longer reasoning chains) and parallel sampling with a process verifier — and asks: given a fixed compute budget, when does it pay to scale parameters vs. scale thinking? Finds that for prompts of easy-to-medium difficulty, an LLM with optimally-allocated test-time compute can match a 14× larger model. For very hard problems, parameter scale still wins.
Why it matters: Pre-dated o1 by ~weeks; provided the theoretical scaffolding everyone now cites. The “compute-optimal scaling” framing is what lets product teams justify spending $0.50 of inference on a single agent step instead of training a $50M bigger model.
arxiv.org/abs/2408.03314 →
arXiv preprint · 2025 Recent
s1: Simple Test-Time Scaling
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, Tatsunori Hashimoto (Stanford / U. Washington) · arXiv 2501.19393 · Jan 2025
The minimum-viable reasoning model. Curates only 1,000 high-quality reasoning traces (s1K), supervised-fine-tunes Qwen2.5-32B-Instruct on them for ~26 minutes on 16 H100s, and introduces budget forcing — controlling test-time compute simply by appending "Wait" to the model’s output to make it think longer, or by truncating thinking with "Final Answer:". The resulting s1-32B beats o1-preview on AIME24 and MATH by up to 27%.
Why it matters: Demolished the “you need millions of RL rollouts and a frontier base model” assumption. The budget-forcing trick is now the canonical way to expose the test-time-compute knob in open inference servers (vLLM, SGLang).
arxiv.org/abs/2501.19393 →
GitHub Pulse

Six Repos That Are the Working Substrate of the Reasoning-Model Stack

Star counts verified on github.com on the day this was generated. Counts are approximate and move quickly.

deepseek-ai/DeepSeek-R1
flagship
~92K ★
Official repo for DeepSeek-R1 / R1-Zero / six distilled variants (1.5B → 70B), MIT-licensed, weights on Hugging Face.
Why it matters: the canonical open reasoning model. Read the README to internalize the four-stage training pipeline.
huggingface/open-r1
replication
~26K ★
Hugging Face’s fully open reproduction of DeepSeek-R1 — data pipeline, GRPO training scripts, distillation recipes.
Why it matters: where the recipe became hackable. If you want to fine-tune a reasoning model on your domain, this is the entrypoint.
volcengine/verl
infra
~21K ★
Volcano Engine RL training framework. Industrial-strength PPO / GRPO / DAPO infrastructure used by multiple frontier labs.
Why it matters: closest open-source thing to the RL infrastructure inside DeepSeek and Qwen labs. The serious replication target.
huggingface/trl
foundation
~18K ★
Transformer Reinforcement Learning. The library most labs reach for first to do PPO / GRPO / DPO on top of HF Transformers.
Why it matters: every reasoning-model tutorial in 2025-2026 starts here. Lower barrier than verl, slower at scale.
OpenRLHF/OpenRLHF
infra
~9.4K ★
Ray + DeepSpeed-based RLHF / RLVR framework. Strong support for distributed PPO and reward-model serving.
Why it matters: the practical “step 2” after you outgrow trl on a single node and don’t want to write your own Ray cluster.
simplescaling/s1
paper code
~6.6K ★
Code, dataset (s1K), and trained checkpoints from the s1 paper. Includes the budget-forcing inference script.
Why it matters: smallest end-to-end reasoning model you can actually train and serve yourself. Best teaching artifact in the space.
Community Pulse

What the Loud Voices Are Actually Saying

AK
Andrej Karpathy
@karpathy
Karpathy has argued that the reasoning era opens a new product surface he calls the “cognitive core” — a small model (a few billion parameters) that deliberately sacrifices encyclopedic knowledge in exchange for a Matryoshka-style architecture in which capability and reasoning are dials the user can turn at test time, including on-device. The thesis: knowledge belongs in retrieval; what should live always-on, locally, is the thinking engine.
Context: Karpathy’s X post on the “cognitive core” idea (x.com/karpathy/status/1938626382248149433) and his 2025 year-in-review writing.
LW
Lilian Weng
lilianweng.github.io
Weng has surveyed test-time compute as the next clean “capability dimension” in her May 2025 essay Why We Think. She frames thinking tokens as a Kahneman-style System-2 budget bolted onto a System-1 LLM, treats CoT as a latent-variable formulation, and connects the post-o1/R1 wave to the broader literature on iterative refinement, decoding-time search, and faithfulness-of-reasoning concerns.
Context: lilianweng.github.io/posts/2025-05-01-thinking/ · ~40-minute read, the de-facto literature review for engineers entering the space.
SW
Simon Willison
simonwillison.net
Willison has been one of the most consistent public chroniclers of the reasoning-model rollout: documenting in real time how DeepSeek-R1 made chain-of-thought visible (via <think> tags) at the moment OpenAI was hiding it; tracking pricing collapse on R1-class API endpoints; and noting how cheap reasoning has reshaped what hobbyist developers can ship from a laptop.
Context: ongoing coverage on simonwillison.net and his Substack newsletter through 2025-2026.
HC
Harrison Chase
@hwchase17 · LangChain
Chase’s recurring 2025-2026 thesis: “better models alone won’t get your agent to production.” Even with a reasoning model in the loop, the work shifts to context engineering — what you put in the prompt, how you offload to a filesystem or memory tier, when to spawn subagents. LangChain’s deepagents harness is explicitly framed as “works because LLMs are getting better at reasoning”: you can lean on the model for plan quality only if you’ve solved context first.
Context: Chase’s LangChain blog and his Sequoia Capital podcast appearances on long-horizon agents (2025-2026).
Platform Deep-Dive

How the Four Frontiers Expose Reasoning Today

PlatformReasoning surfaceKnob you setVisible CoT?
Claude (Anthropic) Extended thinking on Opus / Sonnet / Haiku 4.x lines. Reasoning interleaves with tool calls inside a single agent turn. thinking.budget_tokens (e.g. 1024 → 32000). Streamed thinking blocks in the response. Yes — full thinking blocks are returned to the developer; the API also supports redaction.
OpenAI o-series + GPT-5.4 o3 / o4-mini / GPT-5.4 reasoning variants. First family to give reasoning models full agentic tool access (web, Python, files, vision, image gen). reasoning.effort: minimal / low / medium / high. Tool use is decided by the model, not declared by the caller. No — reasoning tokens are summarized but not exposed verbatim; you pay for them but can’t read them.
Gemini 3.1 (Google) Thinking mode on 2.5 Pro and successor 3.x. Long-context (1M+ tokens) plus thinking budget makes it the default for very large codebases / document corpora. Thinking budget configurable in API and AI Studio; supports parallel tool calls during thinking. Partial — thought summaries are shown by default, raw thoughts gated.
OpenClaw Skill-based agent runtime. Reasoning is delegated to whichever model the SKILL.md targets — Claude, R1-distill, or local SLM — with thinking budget passed through the skill manifest. Per-skill reasoning_budget field. Default low for routine skills, high only for deliberate reasoning skills. Depends on the underlying model. Skills that target Claude inherit visible thinking; skills that target o-series inherit hidden.
Practical implication for product: the “visible CoT” column is becoming a regulatory and trust battleground. EU AI Act guidance is converging on a position that high-stakes agents must produce auditable reasoning artifacts — which favors models that expose chain-of-thought verbatim.
Vocabulary

Twelve Terms That Separate Tourists from Practitioners

test-time compute
The compute spent generating tokens at inference, distinct from the compute spent during pre-training. The third scaling axis.
✗ Don’t conflate with “inference cost” in general — this term specifically refers to compute spent on thinking, not on producing the final answer.
“Snell et al. showed test-time compute can substitute for ~14× parameter scale on easy-to-medium prompts.”
RLVR
Reinforcement Learning with Verifiable Rewards. RL where the reward signal comes from a programmatic checker (math equality, unit tests, JSON validator), not a learned reward model.
✗ Not the same as RLHF — RLHF uses a reward model trained on human preferences; RLVR sidesteps that.
“DeepSeek-R1-Zero is the existence proof that RLVR alone can elicit reasoning behaviors.”
GRPO
Group Relative Policy Optimization. PPO variant introduced by DeepSeek that drops the value network and computes advantages from group statistics across sampled rollouts.
✗ Don’t describe GRPO as “just PPO” — the value-network removal is the operational point that makes large-scale RL on LLMs tractable.
“We swapped PPO for GRPO and our memory footprint dropped ~30%.”
budget forcing
Inference-time technique that controls thinking length by appending tokens like "Wait" (to extend) or "Final Answer:" (to truncate) directly into the model’s decoded stream.
✗ Not the same as setting max_tokens — budget forcing manipulates the content the model conditions on, not just the cap.
“The s1 paper’s entire test-time-compute knob is budget forcing.”
thinking tokens
Tokens emitted by the model into a private chain-of-thought channel that may or may not be returned to the caller, but are billed and time-bounded.
✗ Don’t assume “hidden” means “free” — OpenAI bills hidden thinking tokens at output rate.
“Our p99 latency blew up because we set thinking-tokens too generously on a hot path.”
process reward model (PRM)
A reward model that scores intermediate reasoning steps (not just the final answer), used to guide search through reasoning trajectories.
✗ Don’t conflate with outcome reward model (ORM); PRMs reward the steps, ORMs reward the conclusion.
“We use a PRM at inference for tree search and an ORM during RL training.”
CoT faithfulness
The degree to which a model’s exposed chain-of-thought reflects the actual computation that produced the answer.
✗ A “readable” CoT is not a “faithful” CoT — recent papers show models routinely produce post-hoc rationalizations.
“We can’t use the model’s CoT for compliance audit until we’ve measured faithfulness on our task distribution.”
distilled reasoner
A smaller open model fine-tuned on long reasoning traces sampled from a larger reasoning teacher (e.g. R1-Distill-Qwen-32B from R1).
✗ Don’t expect a distilled reasoner to do RLVR-style self-correction at the same depth as its teacher; the strategy space gets shallower with distillation.
“For our latency budget we shipped a 14B distilled reasoner instead of calling the 671B teacher.”
parallel sampling (best-of-N)
Generating N independent completions and selecting one via majority vote, learned verifier, or PRM-scored re-rank.
✗ Not the same as “temperature sweeping”; best-of-N is structured search with a selection rule.
“Scaling test-time compute by best-of-32 with a PRM ranker gave us +6 points on AIME for ~5× the cost.”
tool hallucination (in reasoning models)
Failure mode in which RL-tuned reasoners invoke tools that don’t exist, hallucinate parameters, or use distractor tools confidently. Documented to amplify proportionally with reasoning capability.
✗ Don’t treat as a generic LLM hallucination — it’s structurally tied to outcome-only RL rewards.
“Our R1-distill agent had a 4× tool-hallucination rate vs the same model pre-RL on SimpleToolHalluBench.”
reasoning-effort header
API parameter (most prominently reasoning_effort on OpenAI) that selects an internal preset for thinking budget without exposing exact token counts.
✗ Not standardized across vendors — Claude exposes raw budget_tokens; Gemini exposes a budget; OpenAI hides it behind a level enum.
“On our research-grade questions we go effort=high; on autocomplete-class queries, minimal.”
cognitive core
Karpathy’s name for a hypothetical small (few-billion-parameter) reasoning model deliberately stripped of encyclopedic knowledge so it can run always-on locally as the “kernel” of personal LLM computing.
✗ Not a shipped product yet — treat as a thesis pointer, not a category.
“The cognitive-core thesis explains why every device vendor wants a 4B reasoner on-chip.”
Expert Questions

Five Questions That Make Engineers Lean In

Q1
Where does test-time compute stop substituting for parameters — what does the Snell et al. compute-optimal frontier look like at the hardest tail of your distribution, and how do you detect that you’ve crossed it in production?
Q2
If RLVR with outcome-only rewards reliably amplifies tool hallucination (Yin et al., “The Reasoning Trap,” arXiv 2510.22977), what does the joint training objective look like — do you bolt on a tool-faithfulness PRM, or do you intervene in the reward function itself?
Q3
When the model’s exposed chain-of-thought is the artifact you depend on for audit, how do you measure CoT faithfulness on your own task distribution — counterfactual rerunning, attribution-graph methods, or something cheaper?
Q4
Reasoning models compress agent loops by absorbing what used to be multiple ReAct steps into a single fat call. At what loop length does that compression invert — where the latency-tail of one fat reasoning call exceeds what the orchestrator could have done with cheaper tool-mediated short steps?
Q5
Distillation from a reasoning teacher transfers traces but not necessarily strategy — in your experience, which capabilities survive distillation (math, code) and which collapse (multi-hop self-correction, novel-domain backtracking)?
CGO Lens

What Reasoning Models Actually Mean for botlearn.ai

Three product / commercial implications worth saying out loud in your next staff meeting

⚠ Watch-list for the next 90 days: (1) the EU AI Act’s emerging guidance on reasoning-trace auditability (favors visible-CoT vendors), (2) the price floor on R1-class API endpoints (every step down expands the addressable market for AI-native education), (3) the open-source vs. proprietary gap on tool-use faithfulness benchmarks (where the moat is actually being contested).
Curriculum Tracker

17 Days + 1 Bonus — Where We Are

D01 Full Agent Stack
D02 Memory Architecture
D03 Planning & Tool Use
D04 RAG Deep Dive
D05 Agent Frameworks
D06 Benchmarks & Eval
D07 Multi-Agent Systems
D08 Computer Use Agents
D09 Code Agents
D10 Long-Horizon Tasks
D11 Agent Safety
D12 Agent Economics
D13 Research Frontiers
D14 OpenClaw Deep-Dive
D15 A2A Protocols
D16 Agentic Commerce
D17 Synthesis (Capstone)
D18 Reasoning Models & Test-Time Compute · today