Inference Scaling: The Next Frontier in AI Governance

Frontier artificial intelligence is entering a new phase where progress may rely less on building ever-larger models and more on running those models with vastly more compute at inference time. This shift, often called inference scaling, could quietly overturn many assumptions that policymakers, companies, and the public currently rely on when thinking about how to govern advanced AI systems.

Drawing inspiration from Toby Ord’s recent work on inference scaling and AI governance, this article explains why this shift matters, how it might change industry dynamics and global power, and what options societies have to steer the transition responsibly.

From Training Scaling to Inference Scaling

For roughly a decade, AI progress has followed a straightforward recipe: collect more data, stack more compute into training, and scale up model size. Capabilities improved predictably as compute used in training increased, leading to large language models and multimodal systems that now shape search, coding, and productivity tools.

Inference scaling describes a different path to more capable systems:

• Training scaling: Invest massive compute once to train a model, then run it cheaply at deployment.
• Inference scaling: Keep the base model fixed or modestly improved, but spend huge amounts of compute each time you run it to solve harder problems, coordinate many copies, or iteratively refine outputs.

In practice, the frontier may combine both approaches. But if inference scaling becomes the main driver of capability gains, many existing governance tools built around training compute thresholds will need substantial revision.

Two Flavors of Inference Scaling

Inference compute can be scaled in at least two distinct settings, with sharply different implications for governance and risk.

Scaling inference at deployment

Inference-at-deployment means dramatically increasing the compute used for each external use of a model. Examples include:

• Running thousands of copies of a model in parallel on a single complex task.
• Using multi-step deliberation or tool use that multiplies the compute per query.
• Coordinating large AI “swarms” that search over strategies, plans, or proofs.

This is attractive to firms and states because it can be applied to already-trained systems without waiting for the next training run. It also tends to be easier to scale elastically, adjusting the amount of compute according to the value of the task.

Scaling inference during training and post-training

A second pathway uses substantial inference compute inside the development process:

• Running powerful models to generate synthetic data, feedback, or curriculum for weaker models.
• Using inference-heavy techniques like iterated distillation and amplification to gradually build more general systems.
• Deploying long-horizon planning or search within training loops to find better policies or architectures.

In this setup, the public may only ever see intermediate or compressed models, while the most capable systems remain internal to labs or governments until late in development. That raises profound transparency and accountability concerns for regulators and the broader public.

Why Inference Scaling Challenges Existing Governance Models

Governments have begun experimenting with compute thresholds as a central regulatory tool: systems that exceed specified training compute levels trigger mandatory reporting, safety evaluations, or licensing requirements. The EU AI Act, for example, includes specific compute-based criteria for general-purpose AI models that meet frontier-scale thresholds.

Inference scaling undermines that strategy in several ways:

• Training compute no longer tracks danger: A modestly trained model paired with massive inference compute might be more capable and risky than a larger model run cheaply at deployment.
• Danger depends on use, not just object: Whether a system is risky may depend on how much inference compute is applied in a specific application, making it harder to regulate the model as a single “object.”
• Thresholds become dynamic and context-dependent: The same base model could be benign in low-compute settings and extremely powerful in high-compute orchestration, complicating enforcement and monitoring.

Attempting to extend thresholds to cover both training and inference would force regulators to reason about specific deployments and workloads, rather than just model scale. That pushes governance toward monitoring uses and operations rather than just training events and model releases.

Impacts on Open Models, Security, and Business Models

Inference scaling does not just reshape regulation; it also alters incentives around openness, security, and competition in the AI industry.

Reduced value of securing model weights

In the current paradigm, model weights are a core strategic asset. If a frontier model leaks, many downstream risks increase: uncontrolled proliferation, misuse for cyber operations, or unregulated commercial deployments. As a result, labs invest heavily in weight security and access control.

If, however, the most powerful behavior of a system depends on enormous inference budgets that only a few actors can afford, the raw weights become less dangerous in isolation. A stolen copy run at ordinary consumer-scale hardware may not reproduce the capabilities obtained with extreme inference scaling in hyperscale clusters. This does not eliminate the need for security, but it may:

• Shift focus toward securing compute infrastructure and inference orchestration tools.
• Reduce some arguments in favor of universal weight access restrictions.
• Change threat models from “model leaks” to “compute-rich adversaries.”

Changing the debate over open-weight models

Inference scaling also affects how we think about open-weight frontier models:

• If the most dangerous capabilities require enormous inference budgets, then many open-weight copies may be relatively harmless in everyday settings.
• At the same time, well-resourced malicious actors could combine open weights with rented or covert compute to reach highly capable regimes.

This creates a more nuanced risk landscape than the simple dichotomy of open versus closed models. Governance may need to consider graduated policies, where weight openness, allowed inference budgets, and deployment contexts are all regulated together.

New business models for frontier AI

Traditional SaaS-style AI services assume relatively cheap inference per query, enabling mass-market assistants and APIs. Inference scaling opens new, more capital-intensive business models:

• Ultra-premium compute services for governments, large corporations, and critical infrastructure providers.
• Project-based inference contracts where companies spend enormous compute on specific scientific, military, or strategic goals.
• Specialized “AI supercomputing” utilities that sell inference capacity more like energy or satellite imagery than consumer apps.

These models could increase the concentration of power among a small number of firms and states that control cutting-edge compute and energy infrastructure, amplifying longstanding concerns about digital monopolies and geopolitical asymmetries.

Infrastructure, Energy, and Geopolitics

Running frontier models with huge inference budgets places new demands on physical infrastructure: data centers, energy systems, cooling, and even grid stability.

Emerging research suggests that data centers and AI workloads are already a significant and growing share of electricity demand in advanced economies, with projections of further growth as models and usage expand. In an inference-scaling world, governments may face difficult trade-offs between:

• Supporting national AI competitiveness via favorable energy and infrastructure policies.
• Maintaining climate goals and grid resilience.
• Applying export controls or investment restrictions on AI-relevant compute and infrastructure.

At the same time, inference-heavy systems may no longer require single, monolithic mega–data centers. Instead, firms could distribute workloads across many smaller sites, complicating efforts to monitor frontier activity purely through the largest facilities.

Opacity, Timelines, and Preparedness

One of the most concerning implications of inference scaling inside training and post-training is a loss of visibility into the true state of the art. If labs use massive inference internally to train, evaluate, and refine systems, but only release diluted or smaller variants publicly, then:

• Regulators and external researchers may underestimate how close we are to highly transformative AI systems.
• The public will not experience the most capable systems early, limiting the political space for proactive regulation.
• Policy responses may lag behind rapid capability jumps, increasing the chance of abrupt societal shocks.

Several AI safety researchers, including Toby Ord, have emphasized the risk of short timelines to transformative or even existentially dangerous AI if capability curves steepen unexpectedly. Inference scaling magnifies this risk by enabling internal, opaque jumps in capability that are not easily tracked by training compute alone.

Policy Options for an Inference-Scaling World

Despite these challenges, governments and civil society are not powerless. A number of policy approaches could help adapt AI governance to the realities of inference scaling.

Mandatory capability and deployment disclosures

Instead of relying solely on compute thresholds, regulators could require large AI developers to disclose:

• The highest capability systems they currently operate internally, even if not deployed.
• The maximum inference budgets used for any deployment, including internal projects.
• Forthcoming plans for significantly scaling inference or expanding into new high-risk domains (e.g., cyber operations, bioengineering).

Such requirements would not eliminate uncertainty, but they could reduce the risk of regulators and the public being blindsided by sudden, large capability jumps.

Licensing high-risk uses of extreme inference compute

Inference scaling suggests regulating not only models but also certain uses of compute. One approach would be to create a licensing regime for:

• Deployments above specified inference-compute thresholds.
• Applications in domains with high systemic or catastrophic risk (e.g., nuclear command and control, biological design tools, critical infrastructure management).
• Long-horizon autonomous systems with substantial real-world authority.

Licenses could require robust evaluation, red-teaming, incident reporting, and auditability, similar to safety regimes in nuclear energy or aviation.

Monitoring compute, not just data centers or weights

Because inference workloads may be spread across many facilities and involve both on-premises and cloud resources, oversight may need to track:

• Sales and deployment of frontier AI accelerators.
• Large, sustained cloud inference contracts for sensitive clients.
• Unusual demand patterns that signal high-end inference swarms.

This could involve a mix of export controls, reporting obligations for cloud providers, and international coordination on hardware and infrastructure standards.

Public engagement and democratic choice over AI futures

Polling in multiple countries suggests that many citizens are deeply uneasy about rapid, unregulated AI development and favor stronger oversight. However, public debate rarely addresses the structural choices around inference scaling, concentration of power, or the distribution of AI assistance.

Societies face real options, such as:

• Restricting certain extreme-inference uses by default, unless a strong societal case is made.
• Prioritizing broadly accessible AI assistance over narrow, highly concentrated capabilities.
• Experimenting with international norms or moratoria on crossing particular capability thresholds until safety evidence improves.

These are not purely technical decisions. They are political and moral choices about how much risk, inequality of power, and automation of high-stakes decisions we are willing to accept.

Strategic Scenarios: Concentration vs. Distribution of AI Power

Inference scaling interacts with deeper questions about who should wield advanced AI capabilities and on what terms. Several strategic scenarios illustrate the range of possibilities.

• Highly concentrated AI power
  A small set of firms and states control most extreme-inference infrastructure. They can run strategic planning, cyber operations, or scientific discovery at scales far beyond others. This scenario raises acute concerns about authoritarian misuse, coercive economic leverage, and accidental escalation in crises.
• Strictly limited high-end capabilities
  Strong global norms or regulations tightly constrain the use of extreme inference budgets, especially in domains with catastrophic risks. Frontier capabilities develop more slowly but in a more controlled environment, somewhat analogous to nuclear arms control.
• Broadly distributed advanced assistance
  Instead of maximizing the raw capability of a few systems, societies prioritize providing relatively equal, powerful AI assistance to many individuals and organizations. Inference budgets are used to uplift general productivity rather than to deepen strategic asymmetries. This may reduce certain concentration-of-power risks but complicate monitoring for misuse.

Which of these paths we follow will depend less on technical inevitabilities than on regulatory design, civic organization, and the willingness of experts and institutions to articulate and defend alternative futures.

Frequently Asked Questions (FAQs)

Q: What exactly is inference scaling in AI?

Inference scaling is the practice of dramatically increasing the compute used when running a trained model, rather than only scaling compute during training. This can involve many parallel copies, long deliberation chains, or complex orchestration that multiplies the effective power of a fixed model.

Q: Why does inference scaling matter for AI safety and governance?

It matters because many current regulatory tools focus on training compute or model size. If powerful capabilities arise mainly from inference scaling, then dangerous behavior may be enabled by how a system is used rather than how it was trained, making it harder to identify and regulate high-risk systems using training-only thresholds.

Q: Does inference scaling make open-source AI safer or more dangerous?

Inference scaling has mixed effects. If extreme capabilities require huge inference budgets, then many open models may be relatively safe in ordinary consumer settings. However, well-resourced adversaries could combine open weights with large compute to reach dangerous capability regimes. Policy may need to address both weight access and access to high-end compute.

Q: Can traditional compute thresholds still be useful?

Yes, but they are unlikely to be sufficient on their own. Thresholds for training compute can still help flag particularly large, general-purpose models. However, additional mechanisms—such as licensing extreme-inference deployments, monitoring hardware, and requiring capability disclosures—will be needed to capture the risks unique to inference scaling.

Q: What should policymakers prioritize in the near term?

Near-term priorities include building institutional capacity to understand technical trends, implementing basic reporting requirements for large training runs and high-end inference deployments, supporting independent evaluation and red-teaming, and engaging the public in explicit discussions about acceptable uses and risks of frontier AI systems.

Written by

Medha Deb

Medha Deb is an editor with a master's degree in Applied Linguistics from the University of Hyderabad. She believes that her qualification has helped her develop a deep understanding of language and its application in various contexts. Medha specializes in the areas of beauty, health, sports, and wellness and is committed to ensuring that the content on the website is…