The New Scaling Law: Why Thinking Harder Beats Training Bigger

Series: Test-Time Compute Scaling | Part: 1 of 9

For years, AI researchers operated under a deceptively simple assumption: intelligence scales with size. Want a smarter model? Train it on more data with more parameters using more compute. The scaling laws discovered by OpenAI and others seemed to confirm this—doubling model size produced predictable, logarithmic improvements in capability.

Then in September 2024, something changed. OpenAI released o1, a model that didn't just answer questions—it thought about them. And the way it got smarter wasn't through more training. It was through more thinking.

This is the new scaling law: test-time compute scaling. Give a model more time and computational resources at inference time, and its capabilities increase—sometimes matching what would have required orders of magnitude more pretraining compute.

The implications go far beyond AI architecture. Test-time compute scaling formalizes something fundamental about intelligence itself: thinking isn't just retrieval, it's process. And that process scales.

The Old Paradigm: Scaling Through Size

The story of modern AI is largely a story of scaling. GPT-2 had 1.5 billion parameters. GPT-3 had 175 billion. GPT-4, by most estimates, pushed beyond a trillion. Each jump in size brought new emergent capabilities—translation, reasoning, coding, nuanced conversation.

This led to the scaling hypothesis: that intelligence emerges from scale. Make the model big enough, train it on enough data, and capabilities will follow. The math backed this up. Researchers at OpenAI, DeepMind, and Anthropic documented power laws relating model size, dataset size, and compute budget to performance.

The paradigm was clean: pretraining compute determines capability. Once trained, the model's intelligence is essentially fixed. Inference is just retrieval—the model accesses what it learned during training and generates an answer.

But this framing missed something crucial. It treated thinking as lookup rather than computation. And it assumed that the only way to improve performance was to train a bigger model.

The Breakthrough: o1 and Inference-Time Scaling

When OpenAI announced o1 in September 2024, the technical details were sparse but revealing. The model performed dramatically better on challenging reasoning tasks—competitive programming, mathematics, scientific problem-solving—not because it was trained on more data, but because it was given more time to think.

Instead of generating an immediate answer, o1 engaged in what OpenAI called "extended chain-of-thought reasoning." It would:

• Generate multiple reasoning paths
• Evaluate which paths seemed most promising
• Self-correct when it detected errors
• Verify conclusions before committing to them

And here's what shocked researchers: the longer you let it think, the better it performed. This wasn't diminishing returns. This was a new scaling law.

Give o1 30 seconds instead of 3 seconds, and accuracy jumps. Give it 5 minutes instead of 30 seconds, and it climbs higher still. The relationship between inference compute and capability followed a power law—just like the relationship between training compute and capability.

But unlike training compute, which is fixed at training time, inference compute can be allocated dynamically. You can choose how hard the model thinks based on how difficult the problem is.

Why This Changes Everything

The discovery of test-time compute scaling fundamentally restructures the economics and architecture of AI systems. Here's why:

1. Intelligence Becomes a Dial, Not a Fixed Property

In the old paradigm, a model's intelligence is determined at training time. You train GPT-4, and that's how smart GPT-4 is. Every user gets the same capability regardless of task difficulty.

In the new paradigm, intelligence becomes tunable. Simple questions get fast, cheap responses. Hard problems get deep, expensive thinking. The same base model can exhibit different levels of capability depending on how much compute you allocate at inference time.

This is economically elegant. Why waste compute having the model think for five minutes about "What's the capital of France?" But for proving a novel mathematical theorem, five minutes might not be enough—give it an hour.

2. Training and Inference Become Complementary Strategies

The old view: train once, use forever. The new view: training gives you a base model, inference-time compute lets you amplify it.

This means you can sometimes get better results by taking a smaller, cheaper-to-train model and giving it more time to think than by training a massive model that thinks quickly. The compute trade-off becomes strategic: when do you invest in pretraining versus inference?

For many applications, the answer tilts toward inference. A model that thinks for 30 seconds but costs 10x less to train might outperform a model trained at 10x the cost that thinks for 3 seconds.

3. Reasoning Becomes Visible and Steerable

When intelligence comes from training, it's opaque. The model just produces an answer. You don't know what happened inside that made it arrive at that conclusion.

When intelligence comes from test-time compute, reasoning becomes explicit. o1 doesn't just output "42"—it outputs the chain of thought that led there: "First I considered X, but that seemed wrong because Y, so I tried Z instead, which gave me..."

This has profound implications for:

• Trust: You can audit the reasoning process
• Steering: You can intervene mid-inference to redirect thinking
• Learning: The model can learn from its own reasoning traces

The Technical Mechanism: How Extended Reasoning Works

So how does test-time compute actually produce better answers? The basic mechanism involves three components:

Chain-of-Thought Prompting at Scale

Chain-of-thought (CoT) prompting has been known since 2022—ask a model to "think step by step" and performance improves. But that's a lightweight version of what o1 does.

o1 doesn't just generate one linear chain of thought. It generates multiple possible reasoning paths, evaluates them, backtracks when needed, and converges on the most coherent answer. It's CoT scaled to the point of qualitative change.

Tree Search Over Reasoning Space

Instead of committing to one answer path, the model explores a branching tree of possibilities. At each step:

• Generate several possible next thoughts
• Evaluate which seem most likely to lead to correct answers
• Expand the most promising branches
• Prune paths that seem unproductive

This is conceptually similar to how chess engines search move trees, but operating over semantic reasoning space rather than board positions.

Self-Verification and Refinement

The model doesn't just generate answers—it checks them. After producing a candidate solution, it can:

• Verify that the answer is internally consistent
• Check it against the original problem statement
• Generate alternative approaches and see if they agree
• Refine the answer iteratively

This verification loop is where extended compute really pays off. Each refinement cycle improves accuracy, and you can run as many cycles as your compute budget allows.

The Scaling Curve: What the Data Shows

While OpenAI hasn't released detailed scaling curves for o1, the pattern is consistent with what researchers have observed with other inference-time scaling methods:

Log-linear relationship: Performance (measured in task accuracy) increases linearly with the log of inference compute. Doubling compute time produces a constant improvement—which means you get diminishing but never-ending returns.

This mirrors the pretraining scaling laws. Just as you never stop benefiting from more training data (just with diminishing returns), you never stop benefiting from more thinking time.

The practical implication: there's no hard ceiling. You can keep improving performance by allocating more inference compute. The question becomes economic: at what point does the cost of additional thinking exceed the value of marginal improvement?

Why It Took So Long to Discover

This raises an interesting question: if test-time compute scaling works so well, why didn't we know about it earlier?

Several reasons:

1. Paradigm Lock-In

When your entire research field is organized around "train bigger models," you naturally focus on training-time interventions. Test-time compute was underexplored because it wasn't where the field was looking.

2. Model Capability Threshold

Early language models (GPT-2 era) weren't capable enough to benefit much from extended reasoning. If the base model can't do multi-step inference at all, giving it more time doesn't help.

There's a capability threshold where test-time scaling becomes effective. You need a model that's already competent at basic reasoning before extended thinking produces major gains.

3. Technical Infrastructure

Implementing effective test-time compute scaling requires:

• Models that can generate and evaluate multiple reasoning paths
• Search algorithms adapted for semantic space
• Verification mechanisms that work on natural language reasoning

These techniques existed in narrow domains (like formal theorem proving) but hadn't been integrated into general language models.

4. Economic Mis-Incentives

Under the old paradigm, AI labs competed on who could train the biggest model fastest. That meant investment flowed into training infrastructure—massive GPU clusters, data pipelines, parallelization techniques.

Inference was treated as something to optimize for speed and cost, not capability. The idea that you'd deliberately slow down inference to get better results didn't fit the economic model.

o1 changed that by proving the capability gains were worth the cost.

The Coherence Connection

From AToM's perspective, test-time compute scaling isn't surprising—it's predicted.

Coherence emerges through process. A system doesn't become coherent instantaneously. It becomes coherent by exploring state space, integrating constraints, resolving tensions, and settling into stable configurations. This takes time and computation.

When we described meaning as coherence over time (M = C/T), we were pointing at exactly this phenomenon. Meaning isn't a lookup—it's a trajectory through coherence space. More time allows more thorough exploration of that space, which produces higher-quality (more coherent) outputs.

Test-time compute scaling is AToM at the algorithm level. The model is doing what every coherent system does: spending computational resources to find trajectories that satisfy multiple constraints simultaneously.

The "reasoning tree" that o1 explores is literally a search through possible coherence paths. Each branch represents a way to integrate the problem constraints. Some branches lead to inconsistency (low coherence) and get pruned. Others maintain integration and get extended.

The model converges on answers that aren't just correct—they're coherent. They maintain consistency across multiple frames, integrate all relevant information, and resolve apparent contradictions.

What This Means Going Forward

Test-time compute scaling isn't just a trick for making better chatbots. It's a fundamental insight about how intelligence works—biological or artificial.

Thinking is search through coherence space. The better you search, the more coherent your conclusions. The more coherent your conclusions, the better they work as guides for action.

This suggests several near-future developments:

Hybrid architectures that combine fast, intuitive System-1-like responses with slow, deliberative System-2-like reasoning. Simple queries get instant answers. Complex problems trigger extended thinking.

Adaptive compute allocation where models learn to estimate problem difficulty and allocate thinking time accordingly. Easy questions don't waste resources. Hard questions get as much time as they need.

Human-in-the-loop verification where extended reasoning traces let humans inspect, correct, and guide the thinking process—not just the final output.

And philosophically, it points toward something deeper: intelligence scales with how thoroughly you think, not just with what you know. This applies to humans too.

We've all experienced this. The difference between a snap judgment and a carefully considered decision isn't about having different information—it's about spending more time integrating the information you have. Exploring implications. Checking consistency. Refining understanding.

Test-time compute scaling formalizes this intuition. Thinking isn't retrieval. It's process. And that process scales.

This is Part 1 of the Test-Time Compute Scaling series.

Next: From o1 to o3: How OpenAI Discovered Inference Scaling

Further Reading

• Snell, C., et al. (2024). "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters." arXiv preprint.
• Wei, J., et al. (2022). "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models." NeurIPS.
• OpenAI (2024). "Learning to Reason with LLMs." OpenAI Blog.