Inference-Time Compute Is a New Scaling Axis

For the last few years, the dominant scaling story in AI has been easy to summarise: larger models, more training data, more training compute.

OpenAI's o1 release adds another axis that deserves separate attention. OpenAI says o1 improves not only when more compute is spent during training, but when the model is allowed to spend more time working on a problem at inference. That means capability is no longer determined only by what happened before deployment. Part of it can be purchased per problem, at the moment the problem is being solved.

I think this changes how we should think about model size, latency and even product architecture.

Bigger models are not the only way to spend compute

Traditional scaling happens mostly before the user sends a prompt. A company trains a larger model, spends more compute on the training run, and ships the resulting checkpoint. At inference, the interaction is relatively fixed: prompt in, tokens out.

o1 makes that boundary less rigid.

OpenAI says the model was trained with reinforcement learning to use a chain of thought, refine strategies, recognise mistakes and try another approach when the current one fails. The company reports that performance rises with both train-time compute and test-time compute.

The benchmark results make the effect visible. On the 2024 AIME problems, OpenAI reports that its full o1 model averaged 74% with one sample, 83% when taking consensus over 64 samples, and 93% when re-ranking 1,000 samples with a learned scoring function.

That is the part I find more interesting than the leaderboard position.

The same model can produce different levels of performance depending on how much computation we are willing to spend after the prompt arrives.

This idea did not begin with o1

There have been signals for a while.

Self-consistency showed that instead of accepting one chain-of-thought sample, a system could generate several reasoning paths and select the answer they converge on. Google researchers reported large gains on several arithmetic and reasoning benchmarks from that change in decoding alone.

Tree of Thoughts pushed the idea further by letting a language model explore and evaluate multiple intermediate paths rather than committing to one left-to-right completion. On the Game of 24 task, the paper reports 74% success with its search method versus 4% for GPT-4 with ordinary chain-of-thought prompting.

Verifier-based work points in the same direction. Instead of trusting the first answer, generate candidates, score their reasoning, and spend extra computation deciding which path deserves to survive.

What o1 changes is that this is no longer only an external prompting or search trick around a general model. OpenAI is training a model family around the idea that more inference work should improve the answer.

That feels like a different scaling regime.

The August papers make the trade-off clearer

Two papers published last month make the point in a more explicit way.

Wu and colleagues study what they call inference scaling laws: how model size, generated tokens, voting and search methods trade against one another under a compute budget. Their experiments show cases where a smaller Llemma-7B model paired with a stronger inference strategy outperforms Llemma-34B using a simpler strategy on MATH.

Snell and colleagues reach a related result. Their paper studies ways to allocate test-time compute and finds that the best method depends on problem difficulty. With a compute-optimal strategy, they report more than a fourfold efficiency improvement over a basic best-of-N approach. In one FLOPs-matched setting, extra test-time computation lets a smaller model outperform a model 14 times larger on problems where the smaller model already has some chance of solving the task.

That last condition matters. More thinking is not magic. If the underlying model has no useful representation of the problem, sampling it repeatedly may only produce more wrong answers.

The useful idea is not "compute longer and everything improves." It is that model size and inference budget can trade against each other in some regimes.

My takeaway: capability becomes partly dynamic

This changes the mental model I use for inference.

Today we usually choose a model before a request arrives. Cheap model for simple work. Expensive model for hard work. The selected model then runs with roughly the same inference pattern every time.

I think that will become more dynamic.

A system should be able to inspect a task and decide how much computation it deserves. A trivial extraction request should not receive the same reasoning budget as a difficult algorithm problem. A localisation lookup should return quickly. A complicated build failure, shader bug or codebase refactor may justify a slower pass with search, verification or multiple candidate solutions.

Compute becomes something the application allocates, not only something the model provider spent during training.

That is a meaningful architectural change.

Production systems will need compute budgets

For developers, the obvious cost is latency.

A model that thinks longer before answering will be slower. If the extra work involves multiple candidate generations, search or verification, it consumes more inference resources as well. The product therefore needs a policy for when that trade is worth making.

I can imagine this being useful in production pipelines I already work with.

A Blender assistant changing one material parameter does not need an expensive reasoning loop. A system trying to diagnose why an automated render pipeline produces the wrong result across several scenes might. The second task can involve logs, scripts, render settings and dependencies. Spending more compute to compare hypotheses and verify a proposed fix makes sense if the failure is costly.

Game development has the same split. Renaming an asset is cheap. Investigating an intermittent build error across code, configuration and platform state is not.

This suggests that AI products may need something similar to quality settings in rendering: not one fixed mode, but a budget chosen for the task.

Fast answers when speed matters. More inference work when correctness is worth waiting for.

Evaluation changes with it

Once inference compute becomes variable, model evaluation gets harder.

A benchmark score without a compute budget is incomplete. One sample and 64 samples are not the same result, a verifier changes what the number means, and the count of candidate paths explored and the time the model received both belong next to the score.

OpenAI's own o1 results make this obvious. AIME performance changes sharply between single-sample evaluation, consensus and re-ranking at much larger sample counts.

For production teams, I think this means evaluation needs a cost axis next to accuracy.

The useful question is not simply, "Which model gets the highest score?"

It is closer to: how much quality can I buy for a given latency and inference budget on my actual task distribution?

That is a much more practical metric for software.

Where I think this goes

My expectation is that inference-time compute will become a first-class part of model serving over the next few years.

Models may learn to spend little computation on obvious requests and much more on tasks where uncertainty remains high. Systems around them may route work between direct generation, multiple candidates, verification and search depending on difficulty.

This could weaken the assumption that the strongest system always needs the largest base model. In some workloads, a smaller model with a well-designed inference process may be a better cost-performance choice than a much larger model used in a single pass.

There are limits. Search can waste compute. Verifiers can be wrong. Longer reasoning can follow a bad path for longer. Hard tasks will still benefit from stronger base models.

But o1 changes the scaling discussion for me.

The question is no longer only how much computation we can spend to create a model.

It is how much computation the system should spend on this problem, right now.

Sources

1. OpenAI, Learning to reason with LLMs, 12 September 2024.
2. Snell et al., Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, 6 August 2024.
3. Wu et al., Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models, 1 August 2024.
4. Wang et al., Self-Consistency Improves Chain of Thought Reasoning in Language Models, ICLR 2023.
5. Yao et al., Tree of Thoughts: Deliberate Problem Solving with Large Language Models, NeurIPS 2023.