California SB 1047 Explained: Frontier AI Safety and Regulation

California’s Senate Bill 1047, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, aimed to become one of the first comprehensive state-level frameworks for governing powerful AI models. Although ultimately vetoed, it reshaped how policymakers, labs, and advocates think about regulating frontier AI development.

1. Why Frontier AI Attracted Lawmakers’ Attention

SB 1047 focused on a narrow slice of AI: very large, highly capable frontier models whose misuse or loss of control could plausibly lead to catastrophic harms.

Legislators were particularly worried about models that could:

• Help design or disseminate chemical, biological, radiological, or nuclear (CBRN) weapons
• Conduct or automate large-scale cyberattacks on critical infrastructure, potentially causing hundreds of millions of dollars in damage
• Autonomously carry out conduct that would be criminal if done by a human, including sophisticated fraud or coordinated attacks
• Escape human control or resist shutdown in ways that complicate attribution and accountability

These risks overlap with emerging federal and international concerns. The U.S. government, for example, has cited similar risks in export controls on advanced chips and in voluntary commitments negotiated with major AI labs, emphasizing model misuse for CBRN, cyber offense, and other high-consequence threats.

2. What SB 1047 Tried to Regulate

SB 1047 did not attempt to regulate all AI. It targeted “covered models”—large, general-purpose frontier models that meet strict compute and cost thresholds.

2.1 Defining “Covered Models”

Instead of assessing every AI model on the market, SB 1047 tied its obligations to objective, easily verifiable thresholds:

• Compute cost: Models whose training required compute on the order of USD 100 million or more count as covered.
• Fine-tuning cost: Models significantly upgraded or specialized via fine-tuning at costs exceeding USD 10 million are included.
• Derivatives: Models derived from covered models could also fall under key safety obligations, especially when they preserve or enhance hazardous capabilities.

This approach recognized that as hardware improves, powerful models may become cheaper to train, so the bill combined compute and monetary metrics to keep the definition aligned with real-world capabilities.

2.2 Catastrophic Risk and “Critical Harm”

SB 1047 aimed at preventing a narrow band of worst-case outcomes, such as:

• Mass casualties from weapons of mass destruction enabled or substantially facilitated by AI guidance
• Cyberattacks causing at least USD 500 million in damage to critical infrastructure
• Serious economic or physical harms caused by models acting autonomously in ways that would violate criminal law if done by humans

This design intentionally left out most everyday harms—like spam, minor fraud, or biased recommendations—in order to keep the legal regime targeted, and avoid overwhelming regulators and developers.

3. Core Responsibilities for AI Developers

For developers of covered models, SB 1047 required a structured safety process rather than mandating specific technical designs. The central concept was a safety and security protocol (SSP)—a documented plan showing how the developer will identify and manage catastrophic risks.

3.1 Safety and Security Protocols

Under the bill, developers had to exercise “reasonable care” in creating and following an SSP, including:

• Documenting anticipated hazardous capabilities and relevant use cases
• Designing and running tests that probe for those capabilities under realistic conditions, including adversarial fine-tuning by experts
• Establishing clear criteria for whether a model is safe enough to deploy, and under what operational constraints
• Updating safety procedures in light of new evidence, incidents, or post-deployment findings

This aligns with broader risk-management norms in safety-critical industries, such as aviation or nuclear energy, where documentation, testing, and ongoing monitoring are central tools.

3.2 Testing for Hazardous Capabilities

SB 1047 emphasized rigorous testing as a precondition for high-risk deployment. Developers were expected to:

• Design tests that can reliably detect whether the model can enable catastrophic misuse
• Include scenarios where third-party experts try to unlock dangerous behavior via fine-tuning or prompt engineering
• Account for post-training modifications and downstream derivative models in their safety analysis
• Show that test results reasonably exclude the possibility of catastrophic capabilities, with a margin for error

These ideas resonate with recent federal policy explorations on “red teaming” frontier models and integrating adversarial testing into AI system evaluations.

3.3 Safeguards, Kill Switches, and Attribution

Beyond testing, SB 1047 required concrete safeguards to limit damage if a model were misused or behaved unexpectedly. Key elements included:

• Use controls to prevent individuals from leveraging hazardous capabilities for critical harms
• Restrictions on creating or releasing derivative models that preserve dangerous capabilities
• Technical means—sometimes described as a “kill switch” or “circuit breaker”—to rapidly disable or constrain a model in an emergency
• Mechanisms to ensure that actions leading to critical harm can be accurately attributed to particular models and users

Attribution was particularly important from a legal perspective: if a catastrophic incident occurred, policymakers wanted to trace responsibility through both the technical stack and the chain of human decisions.

4. Obligations for Compute Providers

SB 1047 did not only target labs training models; it also addressed companies operating large computing clusters used to train frontier models.

Operators of such clusters would be required to:

• Adopt written policies and procedures for runs that could train a covered model
• Evaluate whether a customer intends to use compute to deploy or develop a covered model
• Implement safeguards to detect or deter high-risk activities that might bypass model-level regulation

This reflected a broader strategic insight: on paths where AI capabilities are strongly constrained by access to advanced compute, regulating compute infrastructure can complement model-level obligations.

5. Enforcement, Liability, and Political Compromises

One of the most contested aspects of SB 1047 involved how—and when—the state could enforce the rules.

5.1 Role of the Attorney General

Earlier drafts permitted California’s Attorney General to take action based on negligent safety practices even before any harm occurred. Later amendments, influenced by industry feedback, narrowed this authority so that the Attorney General could only sue when critical harm was imminent or had already occurred.

This change significantly weakened the bill’s preventative power but addressed concerns that over-aggressive enforcement could chill innovation or penalize good-faith experimentation.

5.2 Audits and Reporting

SB 1047 contemplated a regime of:

• Regular third-party audits of covered models and their derivatives, starting in the mid-2020s
• Mandatory incident reporting for AI safety events involving covered models, submitted to a designated state entity
• Legal consequences for false or misleading reporting, potentially including expanded perjury exposure in some drafts

Many of these auditing and reporting ideas later reappeared—sometimes in softened form—in subsequent California AI frameworks such as the Transparency in Frontier Artificial Intelligence Act (TFAIA).

5.3 Whistleblower Protections

Recognizing that internal employees often have the best visibility into safety issues, SB 1047 included protections for workers who reported safety failures or attempted cover-ups.

Later bills like SB 53, which focused more narrowly on whistleblower rights and governance inside AI labs, built on this idea, emphasizing the importance of internal accountability alongside external oversight.

6. CalCompute and Access to Safe AI Infrastructure

An innovative component of SB 1047 was the proposal for CalCompute, a publicly supported compute cluster associated with the University of California.

CalCompute was designed to:

• Provide large-scale compute access to startups, academic researchers, and civil-society groups that could not afford top-tier infrastructure
• Offer an environment where research on safety, alignment, and robust evaluations could be conducted under consistent rules
• Reduce the concentration of AI capabilities inside a handful of private companies, while still maintaining strong safeguards

This idea influenced subsequent proposals to create public or semi-public AI research infrastructure at the state and federal levels, with similar goals of democratizing access while structuring oversight.

7. Veto, Aftermath, and the Shift to TFAIA

Despite passing the legislature with broad support, SB 1047 was vetoed by California’s governor in 2024 over concerns that it might be too restrictive and could push AI development out of the state.

The veto did not end the policy conversation. Instead it triggered:

• A governor-commissioned report from AI researchers recommending alternative approaches and refinements
• Renewed legislative efforts that led to SB 53 and ultimately the Transparency in Frontier Artificial Intelligence Act (TFAIA), enacted in 2025
• Wider national and international attention to California’s approach as a template, warning, or both

TFAIA retained the focus on highly capable frontier models and catastrophic risk but emphasized transparency, reporting, and whistleblower protection more than direct pre-harm liability. It also created new institutions and incident reporting channels, echoing some of SB 1047’s core design ideas.

8. Lessons from the SB 1047 Debate

SB 1047’s journey offers a set of practical lessons for AI governance.

8.1 Targeting the Right Layer

The bill’s focus on developers of very large models recognized that most catastrophic risk is likely to flow from a small number of highly capable systems, rather than the long tail of everyday applications.

By aiming at the model and compute layers, SB 1047 sought to influence the design and deployment of the core systems that downstream developers build on. This has since become a dominant theme in frontier AI policy globally.

8.2 Balancing Innovation and Safety

The debate around SB 1047 crystallized a core tension:

• Proponents argued the bill only touched a handful of firms and models, and addressed extreme scenarios like WMD assistance or billion-dollar cyberattacks, not everyday experimentation.
• Critics worried that compliance burdens, ambiguous standards, and potential liability could discourage investment or experimentation in California’s AI ecosystem.

Subsequent laws like TFAIA attempted to strike a new balance: retain focus on catastrophic risk while leaning more on transparency, reporting, and iterative adjustment instead of aggressive pre-harm enforcement.

8.3 The Role of State-Level Leadership

Even though SB 1047 did not become law, it:

• Helped define key concepts—”frontier models,” “critical harm,” “hazardous capabilities”—now used in broader policy debates
• Influenced both federal discussions and private-sector governance frameworks that reference compute thresholds and catastrophic risk criteria
• Showed that states can meaningfully shape global AI governance debates, especially when they host major AI labs and technology clusters

9. Frequently Asked Questions (FAQs)

Q1: Did SB 1047 regulate all AI systems?

No. SB 1047 applied only to covered models—very large frontier AI systems trained or fine-tuned at extremely high cost and compute levels. Everyday applications, small models, and typical business uses were outside its scope.

Q2: What kinds of harms was SB 1047 trying to prevent?

The bill focused on catastrophic risks, including mass casualties from WMD-related misuse, extremely costly cyberattacks on critical infrastructure, and serious harms caused by models acting autonomously in ways that would be criminal if done by people.

Q3: Why was SB 1047 vetoed if it had legislative support?

The governor vetoed the bill after expressing concern that it might be overly restrictive, risk driving AI development out of California, and benefit from a more incremental, transparency-focused approach. This led to follow-on legislation such as SB 53 and TFAIA.

Q4: How did SB 1047 relate to later laws like TFAIA?

Later laws built on SB 1047’s core concepts—frontier models, catastrophic risk, whistleblower protections, and incident reporting—but shifted emphasis toward transparency, reporting obligations, and flexible updating mechanisms rather than strong pre-harm liability.

Q5: What can other jurisdictions learn from SB 1047?

Other governments can draw on SB 1047’s attempts to narrowly target high-risk models, integrate safety testing and incident reporting, and balance innovation against catastrophic risk—while also learning from its political challenges and the adjustments reflected in newer laws like TFAIA.

Written by

Medha Deb

Medha Deb is an editor with a master's degree in Applied Linguistics from the University of Hyderabad. She believes that her qualification has helped her develop a deep understanding of language and its application in various contexts. Medha specializes in the areas of beauty, health, sports, and wellness and is committed to ensuring that the content on the website is…