Last reviewed
Sources
No citations yet
Review status
Needs citations
Revision
v5 · 4,383 words
Add missing citations, update stale details, or suggest a clearer explanation.
SB 1047, officially the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, was a 2024 California bill that would have established the first comprehensive United States framework for regulating the safety of large-scale frontier artificial intelligence systems. Authored by State Senator Scott Wiener (D-San Francisco), the bill was introduced on February 7, 2024, passed both chambers of the California Legislature (the State Senate 32-1 and the State Assembly 48-16), and was vetoed by Governor Gavin Newsom on September 29, 2024.[1][2][3] It remains the most consequential attempt at frontier-AI regulation in any U.S. jurisdiction to date, and its core ideas were carried into Wiener's successor bill, California SB 53, which Newsom signed exactly one year after the SB 1047 veto.[11][12]
The measure would have applied to a narrow class of "covered models" (those trained using more than 10^26 integer or floating-point operations (FLOPs) at a cost exceeding $100 million) and would have required their developers to write and follow a written safety and security protocol, implement a "full shutdown" capability, retain third-party auditors, report safety incidents to the state, and provide statutory whistleblower protections for employees who raised safety concerns.[2][4][5] The bill defined "critical harms" to include mass-casualty events involving chemical, biological, radiological, or nuclear weapons, cyberattacks on critical infrastructure causing at least $500 million in damage, and large-scale autonomous criminal conduct.[2][4]
SB 1047 became one of the most heavily contested pieces of technology legislation in California history. It drew support from prominent AI researchers including Geoffrey Hinton and Yoshua Bengio, the Center for AI Safety, SAG-AFTRA, dozens of Hollywood figures, and, in a striking moment of public lobbying, both Elon Musk and, conditionally, Anthropic. It was opposed by OpenAI, Meta, Google, Andreessen Horowitz, Y Combinator, Stanford's Fei-Fei Li and Andrew Ng, the Yann LeCun-aligned open-source community, and eight members of California's U.S. congressional delegation, including former Speaker Nancy Pelosi.[1][6][7][8][9] Despite passing the State Senate 32-1 and the Assembly 48-16, the bill was vetoed by Newsom, who argued that its scope, keyed to training compute and cost rather than deployment context, could create a "false sense of security" while leaving smaller systems unregulated.[3][10]
Although SB 1047 did not become law, it is widely regarded as the most consequential attempt at frontier-AI legislation in any U.S. jurisdiction to date. Its conceptual vocabulary (covered models, safety and security protocols, critical harms, full shutdown) was carried into subsequent state and federal proposals, into the European Union's AI Act implementation discussions, and into Senator Wiener's successor bill SB 53, the Transparency in Frontier Artificial Intelligence Act, which was signed into law by Governor Newsom on September 29, 2025, exactly one year after the SB 1047 veto.[11][12]
| Author | State Senator Scott Wiener (D-San Francisco) |
| Principal coauthors | Senators Richard Roth, Susan Rubio, Henry Stern[4] |
| Introduced | February 7, 2024 |
| Senate first passage | May 21, 2024 (32-1)[1] |
| Assembly passage | August 28, 2024 (48-16)[1] |
| Senate concurrence | August 29, 2024 (30-9)[1] |
| Enrolled | September 3, 2024[4] |
| Status | Vetoed by Governor Newsom, September 29, 2024 |
| Subject | Safety regulation of frontier AI models above defined compute/cost thresholds |
| Compute threshold | >10^26 integer or floating-point operations AND >$100 million training cost[2] |
| Sponsors | Center for AI Safety Action Fund, Economic Security California, Encode |
| Successor | SB 53 (Transparency in Frontier Artificial Intelligence Act, signed September 29, 2025)[12] |
By early 2024, the rapid commercial release of large foundation models, including OpenAI's GPT-4, Anthropic's Claude 3, and Google's Gemini, had intensified public-policy debate over whether the most capable AI systems posed catastrophic or systemic risks that required statutory oversight. President Joe Biden had issued Executive Order 14110 on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence in October 2023, and the United Kingdom had convened the AI Safety Summit at Bletchley Park the following month; both initiatives focused on the "frontier" tier of AI systems but relied on voluntary commitments rather than enforceable law.[13] The European Union's AI Act, a comprehensive risk-tiered regulation, was approved by the European Parliament in March 2024 but did not specifically target catastrophic risk from frontier models.[14]
In the United States, Congress had held numerous hearings but had not enacted a federal AI safety statute. Several U.S. states had begun moving on narrower AI questions such as algorithmic discrimination, deepfakes, and generative-AI transparency, but none had attempted to address the catastrophic-risk concerns articulated by researchers such as Geoffrey Hinton, who left Google in 2023 to speak publicly about existential risk from AI, and Yoshua Bengio, who began chairing the International Scientific Report on the Safety of Advanced AI.[15]
Senator Wiener, who chaired the California State Senate's Budget and Fiscal Review Committee and represented San Francisco, the geographic center of the U.S. AI industry, and announced an "Intent to Legislate" version of the bill in September 2023 and introduced the operative text in February 2024.[16] The bill was co-sponsored by the Center for AI Safety Action Fund (associated with researcher Dan Hendrycks), Economic Security California, and the youth advocacy organization Encode.[1]
SB 1047 applied not to AI broadly but to a narrowly defined category of "covered models." Through January 1, 2027, the bill defined a covered model as one trained using either (a) a quantity of computing power greater than 10^26 integer or floating-point operations and costing more than $100 million to develop (measured by the average market price of cloud compute at the time of training), or (b) created by fine-tuning a covered model using three times 10^25 integer or FLOP of compute at a cost greater than $10 million.[2][17] After January 1, 2027, the thresholds were to be adjusted by the Government Operations Agency based on technological developments.[2]
These thresholds were calibrated to exclude all then-existing publicly released models (even GPT-4 was widely estimated to fall below the 10^26 FLOP threshold) and were intended to apply only to a future generation of "frontier" systems trained at unprecedented scale.[17]
Developers of covered models were required, before commencing training, to write a "safety and security protocol" (SSP) describing the technical and procedural measures the developer would take to ensure that the model would not cause or materially enable "critical harms." Developers had to publish a redacted version of the SSP and retain an unredacted version for the Attorney General. Before commercial or public deployment, the developer was required to submit a written statement of compliance attesting that the developer had taken "reasonable care" to prevent the model from posing an "unreasonable risk" of causing or materially enabling critical harms.[2][4][5]
The bill required annual independent third-party audits of compliance with the SSP, beginning January 1, 2026.[4]
The bill defined "critical harm" as harm caused or materially enabled by a covered model that fell into one of four categories: (1) the creation or use of a chemical, biological, radiological, or nuclear (CBRN) weapon causing mass casualties; (2) mass casualties or at least $500 million in damages resulting from cyberattacks on critical infrastructure; (3) mass casualties or at least $500 million in damages resulting from a covered model engaging in conduct that would, if committed by a human, constitute a crime requiring intent, recklessness, or gross negligence; or (4) other grave harms to public safety and security of comparable severity.[2][17]
Developers were required to implement a "full shutdown" capability (informally referred to in press coverage as a "kill switch"), defined as the ability to enact a "prompt enactment of a full shutdown" of the model and all of its derivatives within the developer's control. The provision did not require the ability to shut down third-party fine-tuned or open-weights derivatives outside the developer's possession.[2][18]
The bill prohibited covered developers and their contractors from retaliating against employees who disclosed information they reasonably believed indicated noncompliance with the bill or that the developer's activities posed a "specific and substantial danger to the public health or safety" arising from a critical harm. Developers were required to provide a reasonable internal process for anonymous disclosures and to publish an annual notice to employees describing their rights.[2][19]
The original bill would have created a Frontier Model Division within the California Government Operations Agency to oversee implementation. In response to industry concerns, and particularly to amendments suggested by Anthropic, Wiener replaced the Division in the August 15, 2024 amendments with a nine-member Board of Frontier Models within the Government Operations Agency, intended to update compute thresholds, issue guidance, and review safety incidents.[20][21]
The bill directed the development of CalCompute, a proposed public cloud computing cluster within the University of California system, designed to provide compute resources to academic researchers, startups, and community organizations who lacked access to industrial-scale training infrastructure. CalCompute was widely seen as an attempt to mitigate the bill's disproportionate impact on well-resourced firms by lowering the cost of compliant research for the broader community.[1][17]
The bill authorized the California Attorney General to bring civil actions to enforce its provisions, with civil penalties tied to the cost of the covered model. The August 2024 amendments removed the original bill's authority for the Attorney General to sue pre-harm based on negligent practices alone; under the amended version, most enforcement required either a critical harm to have occurred or an imminent threat thereof.[20][22]
Senator Wiener formally introduced SB 1047 on February 7, 2024. The bill cleared the Senate Judiciary and Appropriations Committees over the spring, with substantial amendments tightening the covered-model definitions and clarifying the SSP requirements. On May 21, 2024, the California State Senate passed the bill by a vote of 32-1.[1][4]
During the summer of 2024, the bill became the focus of an unusually high-profile lobbying campaign. Opposition was led by OpenAI Chief Strategy Officer Jason Kwon, who in an August 21, 2024 letter to Senator Wiener argued that "SB 1047 would threaten that growth, slow the pace of innovation, and lead California's world-class engineers and entrepreneurs to leave the state in search of greater opportunity elsewhere."[7] Kwon urged that AI safety be addressed at the federal level. The trade-association arms of Meta and Google also opposed the bill, as did the venture firms Andreessen Horowitz and Y Combinator.[6][9]
Anthropic took a distinct position. In a July 2024 letter from CEO Dario Amodei, the company circulated specific suggested amendments, known colloquially as the "support if amended" letter, aimed at narrowing the legal standard, removing pre-harm enforcement, eliminating the Frontier Model Division, and changing the legal standard to align more closely with the company's own Responsible Scaling Policy framework.[20][21]
On August 15, 2024, Senator Wiener accepted a substantial portion of Anthropic's suggested amendments. The most significant change shifted the developer's required compliance standard from "reasonable assurance" to "reasonable care," the latter being the centuries-old common-law tort standard. The amendments also removed the Frontier Model Division (replacing it with the smaller Board of Frontier Models), eliminated perjury penalties from the certification requirements, narrowed pre-harm enforcement authority, and clarified that liability would generally arise from a developer's failure to take reasonable care rather than from any harm in itself.[20][21]
On August 21, 2024, Amodei sent a letter to Governor Newsom stating that "in our assessment the new SB 1047 is substantially improved to the point where we believe its benefits likely outweigh its costs," while noting that the company was "not certain of this" and that some aspects of the bill remained "concerning or ambiguous."[23] On August 26, 2024, Elon Musk, whose AI company xAI had not previously taken a public position, posted on X that "all things considered, I think California should probably pass the SB 1047 AI safety bill," noting that "for over 20 years, I have been an advocate for AI regulation."[8]
The amended bill passed the State Assembly on August 28, 2024, by 48-16. The Senate concurred in the Assembly amendments on August 29, 2024, by 30-9. The bill was enrolled and presented to Governor Newsom on September 3, 2024.[1][4]
A coalition of AI researchers, civil-society organizations, labor unions, and a handful of industry voices supported the bill. Turing Award laureates Geoffrey Hinton and Yoshua Bengio, together with Harvard Law School professor Lawrence Lessig and Berkeley computer-science professor Stuart Russell, co-signed an August 7, 2024 letter to California state leadership characterizing the bill as "the bare minimum for effective regulation of this technology" and warning of "severe risks" if next-generation AI systems were "developed without sufficient care and oversight."[24] The same letter described SB 1047 as "a remarkably light-touch piece of legislation," noting that it imposed no licensing regime, required no government permission before training or deploying a model, and did not hold companies strictly liable if a catastrophe occurred.[24]
A subsequent letter signed by 113 current and former employees of OpenAI, Google DeepMind, Anthropic, Meta, and xAI was released on September 9, 2024 in support of the bill; signatories included former OpenAI safety researchers Daniel Kokotajlo, William Saunders, and Jan Leike, all of whom had publicly resigned from the company over safety disagreements during 2024.[1][25] Anthropic (conditionally), Elon Musk and xAI, Vitalik Buterin, Max Tegmark, and OpenAI cofounder Scott Aaronson also endorsed the measure.[1][8][23]
Beyond the AI community, SB 1047 was supported by SAG-AFTRA, the National Organization for Women, the Los Angeles Times editorial board, and more than 120 actors and entertainment-industry figures including Mark Hamill, Jane Fonda, J.J. Abrams, Mark Ruffalo, Sean Astin, and Rosie Perez.[1] Sponsoring organizations included the Center for AI Safety Action Fund, Economic Security California, and Encode.[1]
The bill was opposed by the major U.S. AI labs other than Anthropic, most prominently OpenAI, whose Chief Strategy Officer Jason Kwon wrote publicly against it on August 21, 2024, as well as Meta (whose Chief AI Scientist Yann LeCun was among the bill's most vocal critics) and Google.[6][7] The venture firms Andreessen Horowitz (through general partner Anjney Midha and co-founder Marc Andreessen) and Y Combinator (under president Garry Tan) campaigned actively against the bill.[9][26]
Within academia, Stanford University's Fei-Fei Li, co-director of the Stanford Institute for Human-Centered AI and widely known as a "godmother of AI," published a Fortune op-ed on August 6, 2024 arguing that "if passed into law, SB-1047 will harm our budding AI ecosystem, especially the parts of it that are already at a disadvantage to today's tech giants: the public sector, academia, and 'little tech.'"[26] Andrew Ng, founder of DeepLearning.AI and former head of Google Brain, characterized the bill as "anti-open source" and "anti-innovation."[26] Other academic critics included Ion Stoica (Berkeley), Jeremy Howard (fast.ai), and Russell Wald (Stanford HAI). The AI Alliance, a consortium led by IBM and Meta that advocates for open-source AI, formally opposed the bill.[27]
In an unusual intervention, eight Democratic members of the U.S. House of Representatives from California, including former Speaker Nancy Pelosi (San Francisco), Zoe Lofgren (San Jose), Anna Eshoo (Palo Alto), Ro Khanna (Silicon Valley), Scott Peters (San Diego), Tony Cárdenas (Los Angeles), Ami Bera (Sacramento), and Lou Correa (Orange County), sent letters to Governor Newsom opposing the bill on the grounds that AI safety regulation should be a federal rather than state matter.[28] Pelosi separately issued a statement on August 15, 2024 endorsing Fei-Fei Li's position and calling SB 1047 "well-intentioned but ill-informed."[28]
Governor Gavin Newsom vetoed SB 1047 on September 29, 2024, the final day of his consideration period.[3][10] In his veto message, Newsom praised the bill's intent and Senator Wiener's leadership but identified four principal concerns.
First, Newsom argued that the bill's threshold-based approach, keyed solely to a model's training compute and cost, was not "informed by an empirical trajectory analysis of AI systems and capabilities" and did not "take into account whether an AI system is deployed in high-risk environments, involves critical decision-making or the use of sensitive data."[10][29]
Second, he argued that "the bill applies stringent standards to even the most basic functions, so long as a large system deploys it." In his most-quoted line, he wrote that SB 1047 "establishes a regulatory framework that could give the public a false sense of security about controlling this fast-moving technology. Smaller, specialized models may emerge as equally or even more dangerous than the models targeted by SB 1047."[10][29]
Third, Newsom expressed concern about the bill's effects on the open-source AI ecosystem and on California's competitive position in AI development.[10]
Fourth, despite the veto, Newsom emphasized continued commitment to AI regulation, writing that "safety protocols must be adopted. Proactive guardrails should be implemented, and severe consequences for bad actors must be clear and enforceable." He announced that the State of California would work with the Carnegie Endowment for International Peace, the Stanford Institute for Human-Centered AI (including Fei-Fei Li), and computer-science professor Tino Cuéllar to develop "an evidence-based approach" to frontier-AI regulation, and that California "cannot afford to wait for a major catastrophe to occur before taking action to protect the public."[10][29]
In response, Senator Wiener called the veto "a setback for everyone who believes in oversight of massive corporations" and stated that the veto would "leave us all less safe from the risks posed by these models."[30] He pledged to return with new legislation in the following session.
The deadline to override the veto, requiring a two-thirds vote of both chambers, was November 30, 2024; no override attempt was made.[31]
In early 2025, Senator Wiener introduced SB 53, the Transparency in Frontier Artificial Intelligence Act (TFAIA), a slimmer successor to SB 1047 that retained several of its core ideas while abandoning others.[11][12] The bill required large frontier developers (defined in part by a 10^26 FLOP training threshold, and a $500 million annual revenue threshold for the "large developer" category) to publish a written framework describing how the developer identifies, mitigates, and governs catastrophic risks; to publish a transparency report before deploying a new or substantially modified frontier model; to report critical safety incidents to the California Office of Emergency Services within fifteen days of discovery; and to provide protected internal channels for employee whistleblower disclosures.[12]
Civil penalties for violations were set at up to $1 million per violation, enforced by the Attorney General. Most key provisions take effect January 1, 2026.[12] Governor Newsom signed California SB 53 on September 29, 2025, one year to the day after the SB 1047 veto, calling it a law that places "California at the forefront of responsible AI governance."[11]
Compared to SB 1047, SB 53 omitted the contested "full shutdown" requirement, the pre-deployment audit mandate, and the broad civil-action authority that had drawn industry opposition; instead, it emphasized transparency, incident reporting, and whistleblower protection. Anthropic, OpenAI, Google, and Meta either supported SB 53 outright or did not publicly oppose it.[12]
Several elements first proposed in SB 1047 reappeared in subsequent federal proposals and in voluntary industry commitments. The "safety and security protocol" concept aligns conceptually with the Responsible Scaling Policy frameworks adopted by Anthropic, OpenAI, and Google DeepMind during 2023-2024, and with the AI Safety Framework commitments made at the Seoul AI Summit in May 2024.[32] Frontier-model risk reporting and SSP-style documentation are reflected in the Frontier Model Forum's working-group outputs and in the European Union's draft General-Purpose AI Code of Practice released under the EU AI Act.[14]
Several U.S. states, including New York, Massachusetts, Colorado, and Illinois, introduced bills during 2025 modeled in whole or in part on SB 1047, focused on transparency, incident reporting, and whistleblower protection for frontier models.[33] The U.S. Office of Management and Budget cited the policy debate around SB 1047 in subsequent guidance documents.
Following the veto, several leading laboratories, including Anthropic, OpenAI, Google DeepMind, and Meta, voluntarily published or strengthened their internal frontier-safety frameworks. Anthropic released updated versions of its Responsible Scaling Policy, Google DeepMind published a Frontier Safety Framework, and OpenAI published its Preparedness Framework. Researchers including Stuart Russell noted that these voluntary measures broadly tracked the substantive obligations that SB 1047 would have made legally binding.[34]
Commentary on SB 1047 reflected sharply different views about whether the bill represented prudent risk management or premature regulation of a nascent technology.
Supporters argued that the bill represented a narrowly tailored, evidence-informed approach: it applied only to a small number of unreleased, frontier-scale models; it required process compliance (publishing and following an SSP) rather than substantive design mandates; and it adopted the long-standing common-law standard of "reasonable care" as the threshold for liability. Brookings Institution analyst Chris Meserole and others argued that much of the public criticism rested on misrepresentations of the bill's text, particularly the false claim that it imposed criminal penalties on individual developers, and that the bill's actual provisions were considerably more limited than opponents suggested.[35]
Critics argued that the compute-and-cost threshold was a poor proxy for actual risk; that liability for downstream misuse would chill open-source release and academic research; that the "reasonable care" standard, while well-established in tort, would be unpredictable when applied to a rapidly evolving technology with poorly understood capabilities; and that state-by-state AI regulation would generate compliance overhead without addressing the cross-jurisdictional nature of AI risk. The Andreessen Horowitz-Y Combinator coalition argued additionally that the bill would advantage well-resourced incumbents (which could afford compliance) over startups and open-source projects.[9][26]
The veto itself drew mixed reactions. Civil-society and AI-safety organizations broadly criticized Newsom's stated reliance on the absence of "empirical evidence" of harm as a standard that, by definition, could not be met before a catastrophic event. Industry voices and academic critics of the bill generally praised the veto. Lawfare's Kevin Frazier characterized the episode as "the most consequential AI policy debate in American history to date," noting that despite the veto, SB 1047's vocabulary had largely been adopted by subsequent regulatory drafts.[36]
The episode was widely interpreted as illustrating the political difficulty of regulating frontier AI in a jurisdiction whose economy depends heavily on the industry being regulated. It also marked the first time that a serious legislative attempt to address catastrophic AI risk, long a topic in academic and effective-altruist circles, was taken seriously by a major legislative body, and the first time that AI laboratories took differing public positions on a single piece of legislation.