Reasoning Models — How Test-Time Compute Works

You ask Claude or ChatGPT a hard question, and instead of an instant answer, you get a spinner. Ten seconds. Twenty. Sometimes longer. That pause used to mean something was broken.

Now it means something else entirely. The model is spending compute — real GPU cycles — thinking through your problem before it writes a single word of the answer you actually see.

This is test-time compute, and it is the biggest shift in how frontier AI models get better since the transformer itself arrived. Not a bigger model. Not more training data. More thinking, on demand, at the moment you ask.

🔗 Foundation posts

This post builds on What is a Large Language Model (LLM)? and How Generative AI Works — Tokens, Embeddings and the Transformer . If token-by-token generation is unfamiliar, start there — a reasoning model is still doing exactly that, just with a lot more thinking wedged in before the final answer.

One Pass vs Many — What Makes a Reasoning Model Different

A standard LLM call is a single forward pass. Read the prompt, predict the next token, repeat until the response is done. The compute spent is fixed by the model’s size — a trivial question and a brutal one get roughly the same treatment.

A reasoning model breaks that assumption. Before it writes the answer you see, it generates a stream of intermediate tokens — reasoning, checking, sometimes backtracking — that never appear in the final response but shape it completely.

I think of it as the difference between answering a client question on the spot and taking it away to work through properly. Same person, same knowledge. Wildly different quality of answer.

📌 Key Takeaway

Test-time compute is not a bigger model. It’s the same model, given permission to spend more tokens — and more time — before it commits to an answer.

Chain-of-Thought — What the “Thinking” Tokens Actually Are

Chain-of-thought is the mechanism behind that pause. The model generates a reasoning trace — breaking a problem into steps, trying an approach, noticing when something doesn’t add up, trying again.

This is not the same as a verbose answer. A verbose answer is padding. A reasoning trace is the model actually doing the work — scratch paper for a maths exam, except the scratch paper can run to thousands of tokens and you sometimes never see it at all.

OpenAI hides o1 and o3’s raw reasoning by design and shows only a summary. Claude’s extended thinking leans more transparent by default, but even here you typically see a summarised version of the reasoning rather than the raw token stream — Anthropic treats that transparency as useful for trust and debugging, while still keeping the full raw chain internal.

📝 Note

Neither approach is fully open. Hidden or summarised reasoning protects a lab’s training methods and stops people copying the exact reasoning chain. What you see is a window into the process, not the unedited process itself — and that’s true across every major reasoning model today, not just OpenAI’s.

⚠️ Warning

A visible reasoning trace is not proof of truth. Research has repeatedly found cases where a model’s chain-of-thought does not fully reflect why it actually reached an answer. Treat the trace as a helpful diagnostic — not a guarantee.

How Labs Train a Model to Reason

None of this happens by asking nicely. A base LLM does not spontaneously decide to reason for thousands of tokens — it has to be trained to do it, and that training is reinforcement learning, not the supervised fine-tuning most people associate with ChatGPT-style models.

The idea is simple to state and hard to execute. Give the model a problem with a checkable answer — a maths problem, a piece of code that either passes its tests or doesn’t — and reward it when the final answer is correct. Over enough iterations, the model learns which reasoning patterns actually lead to correct answers, and generates more of them.

OpenAI’s o1-preview, released in September 2024, proved this worked at scale. On the 2024 AIME maths exam, GPT-4o solved about 12% of problems. o1 solved around 74% on a single attempt — not a small tuning gain, a different category of capability.

DeepSeek forced the rest of the industry to pay attention in January 2025. DeepSeek-R1 matched o1’s reasoning performance using large-scale reinforcement learning as its primary training method, and released the model weights openly under an MIT licence — including smaller distilled versions anyone could run locally.

💡 Practical Tip

If a vendor tells you their model “reasons” but can’t explain whether that came from reinforcement learning on verifiable tasks or just longer prompting, ask directly. The distinction determines whether the reasoning is actually reliable or just longer.

Beyond One Chain — Self-Consistency, Best-of-N and Search

A single reasoning trace is powerful, but it is still one attempt. Stack more techniques on top and you get further gains — at the cost of even more compute.

Correct reasoning tends to converge on the same result across multiple attempts; wrong reasoning tends to scatter in different directions. That single fact is what makes all three of the following techniques work.

Technique	What it does	Compute cost
Self-consistency	Samples the same question multiple times with different reasoning paths, takes the majority answer.	Moderate — scales with sample count
Best-of-N	Generates several full candidate answers, uses a separate verifier model to pick the strongest one.	High — full generations plus scoring
Tree search	Explores multiple reasoning branches at once, prunes unpromising ones before they run to completion.	Highest — most flexible, most expensive

OpenAI showed this stacking effect directly with o1. A single AIME attempt scored around 74%. Allowing 64 samples with majority voting pushed that to 83%. Re-ranking 1,000 samples with a learned scoring function reached 93% — same underlying model, more compute spent searching.

Thinking Budgets — Making Test-Time Compute Controllable

Early reasoning models made an all-or-nothing choice — reasoning mode on, or off. That’s wasteful. A simple lookup does not need thousands of tokens of internal deliberation.

The fix is a controllable thinking budget. Anthropic introduced this with Claude 3.7 Sonnet in early 2025 — a setting that lets you specify roughly how many tokens Claude is allowed to spend thinking before it answers.

The direction since then has moved toward adaptive thinking: letting the model itself judge how much a given question needs, rather than a developer guessing a fixed number up front, with a simple effort-level control as the human override.

💡 Practical Tip

If you’re building on an API with a thinking budget or effort control, don’t set it once and forget it. Start low, measure whether the extra reasoning actually changes the answer for your specific task, and only raise it where it demonstrably earns its cost.

The Economics — Training Compute vs Test-Time Compute

For most of the last decade, the scaling law was simple: bigger model, more training data, better results. That’s train-time compute, and the cost is paid once, by the lab, before the model ever ships.

Test-time compute is a second, independent lever. You can take the same trained model and get meaningfully better answers just by letting it think longer at the moment of the question — no retraining required.

The catch is who pays, and when. Training cost is fixed and amortised across every user forever. Test-time compute is paid on every single query, by whoever is running it — which is why reasoning-mode responses cost more and take longer than a standard chat reply.

This is not a minor implementation detail. It’s a genuine shift in where AI infrastructure spend goes — from one-off training runs toward inference cost that scales directly with how much people actually use the thing.

When It’s Actually Worth Paying for a Model to Think

This is the question that actually matters day to day, and it has a reasonably clean answer. Reasoning models earn their cost on problems that are multi-step and verifiable — where a wrong intermediate step genuinely wrecks the final answer.

Maths, code, multi-step planning, an exception analysis where you need to trace cause through several dependent conditions — reasoning mode helps here, consistently. I have watched a standard model confidently skip a step in a multi-condition eligibility check and get the wrong answer fast. A reasoning model catches that step because it has to write it down.

Where reasoning mode is wasted: summarising an email, rewriting a paragraph, answering a factual lookup, most conversational chat. These tasks don’t have a chain of dependent logic to get wrong. You’re just paying for latency and tokens with nothing to show for it.

The same logic applies inside SAP. Ask an AI assistant for a quick lookup and a fast model is fine. Ask it to reconcile a multi-step exception across several linked postings, and that’s exactly the kind of task reasoning mode was built for.

✅ Best Practice

Route by task, not by default. Use a fast model for lookups, rewrites and simple chat. Reserve reasoning mode for problems where a wrong intermediate step would actually break the answer — multi-step analysis, code, maths, planning.

At a Glance — The Mental Model

Concept	One-line summary
Test-time compute	Spending extra compute at inference time, not training time, to improve an answer.
Reasoning model	An LLM trained to generate an internal reasoning trace before producing its final answer.
Chain-of-thought	The intermediate reasoning tokens a model generates while working through a problem.
RL training for reasoning	Rewarding a model for reaching correct, verifiable answers so it learns which reasoning patterns work.
Self-consistency	Sampling multiple reasoning paths and taking the most common final answer.
Best-of-N	Generating several full candidate answers and using a verifier to pick the strongest.
Thinking budget	A controllable limit — fixed or adaptive — on how many tokens a model spends reasoning.
Train-time compute	Compute spent once, during training, paid by the lab and amortised across every user.
Test-time compute (cost)	Compute spent on every query, paid by whoever runs it, scaling directly with usage.

What to Take Away

For twenty years, “better AI” meant a bigger model. That era isn’t over, but it’s no longer the only lever. Test-time compute proved that the same model can get measurably smarter just by being allowed to think longer about your specific question.

That reframes how you should think about AI capability. It isn’t just a property of the model you chose. It’s a property of how much thinking time you’re willing to pay for, on this particular problem, right now.

The skill worth building isn’t “how do reasoning models work” — you have that now. It’s knowing which of your problems actually need one.

🔗 Related posts on this site

How LLMs Are Trained — Pretraining, Fine-Tuning and RLHF — the reinforcement learning covered briefly here goes much deeper into how post-training actually shapes a model’s behaviour.
Temperature and Top-p — Controlling LLM Output — the sampling controls that sit alongside thinking budgets whenever you’re tuning inference behaviour.
AI Evaluation — Why Benchmarks Lie and What to Measure Instead — a healthy dose of scepticism for the AIME-style benchmark jumps quoted in this post.
How AI Agents Actually Think — ReAct, Planning and Reflection — reasoning models and agent planning loops solve overlapping problems in different ways.

Published on rakeshnarayan.com — Articles

URL: https://rakeshnarayan.com/articles/reasoning-models-how-test-time-compute-works/