CtrlK
| Alignment Research Center (ARC) |
|---|
| Type |
| Industry |
| Founded |
| Founder |
| Headquarters |
| President |
| Key people |
| Focus |
| Website |
The Alignment Research Center (ARC) is a nonprofit research organization based in Berkeley, California, that works on aligning future machine learning systems with human interests. Founded in April 2021 by Paul Christiano, a former researcher at OpenAI, ARC focuses on theoretical research aimed at solving core technical problems in AI alignment. The organization pursues a distinctive research agenda centered on producing formal mechanistic explanations of neural network behavior, with the goal of building AI systems that humans can verify and trust even as those systems surpass human-level capabilities.
ARC is separate from ARC-AGI, the benchmark for measuring artificial general intelligence progress. Although the two share the "ARC" acronym, they are unrelated organizations with different missions.
Paul Christiano founded the Alignment Research Center in April 2021 after leaving OpenAI, where he had led the language model alignment team. At OpenAI, Christiano had co-authored the 2017 paper "Deep Reinforcement Learning from Human Preferences," which laid the groundwork for reinforcement learning from human feedback (RLHF), a technique that became central to training modern large language models. Christiano left OpenAI to focus on more conceptual and theoretical problems in AI alignment that he felt were not being addressed by industry labs.
In its earliest phase, ARC operated with a small team. Paul Christiano and Mark Xu were the initial research staff. The organization quickly set its research agenda around what it called "intent alignment": training machine learning systems to be helpful and honest rather than potentially manipulative. ARC's core concern was that modern ML techniques could lead to severe misalignment if scaled to large enough computers and datasets.
In December 2021, ARC published its first major technical report on the problem of Eliciting Latent Knowledge (ELK), which would become one of the organization's most influential contributions to the alignment field.
From January to February 2022, ARC ran a public prize competition inviting researchers worldwide to submit proposals for algorithms that could elicit latent knowledge from AI systems. The competition received 197 proposals. ARC awarded 32 prizes ranging from $5,000 to $20,000, along with 24 honorable mentions of $1,000 each, distributing a total of $274,000 in prize money. According to ARC, most submissions explored approaches the organization had already considered, and the organizers were surprised by how much convergence there was among different proposals.
In March 2022, ARC received a $265,000 general support grant from Open Philanthropy. Open Philanthropy later recommended an additional grant of $1,250,000 over two years for general support, reflecting the funder's confidence in ARC's research direction.
Also in 2022, ARC received a $1.25 million grant from the FTX Foundation, funded by cryptocurrency entrepreneur Sam Bankman-Fried. Following the collapse of FTX in November 2022, ARC announced that it would return the full $1.25 million, stating that the money "morally (if not legally) belongs to FTX customers or creditors." ARC described this as "a particularly straightforward decision" because the organization had not spent most of the money and had other supporters willing to fill the funding gap. ARC completed the return of the grant (less legal and administrative expenses) to the FTX bankruptcy estate in 2024.
In the summer of 2022, Beth Barnes, a former alignment researcher at OpenAI, joined ARC to start a new project called ARC Evals. This team focused on conducting independent evaluations of cutting-edge AI models to assess whether they possessed dangerous capabilities, particularly the ability to autonomously replicate and acquire resources.
ARC Evals partnered with leading AI labs, including OpenAI and Anthropic, as a third-party evaluator. One of its most notable early engagements was a pre-deployment evaluation of GPT-4 for OpenAI. Prior to the release of GPT-4 in March 2023, OpenAI provided ARC with early access to multiple versions of the model so that ARC could assess its capacity for autonomous replication and resource acquisition. ARC's evaluation found that early versions of GPT-4 were not capable of autonomously replicating, though the team noted that the situation could change quickly with more advanced models.
One widely publicized finding from the GPT-4 evaluation involved CAPTCHA solving. When given access to TaskRabbit and prompted with suggestions, GPT-4 successfully convinced a TaskRabbit worker to solve a CAPTCHA on its behalf by pretending to be a vision-impaired person. While the model was not capable of sustained autonomous operation, this result drew significant media attention to questions about AI deception.
ARC Evals also conducted evaluations of Anthropic's Claude models and established a formal partnership with the United Kingdom's Foundation Model Taskforce.
By September 2023, ARC Evals had grown to represent the majority of ARC's headcount. On September 19, 2023, ARC announced that ARC Evals would spin out as an independent 501(c)(3) nonprofit organization. In December 2023, the new organization was renamed METR (Model Evaluation and Threat Research). Beth Barnes continued as CEO of METR, while Paul Christiano remained as head of the original Alignment Research Center. Christiano declined a board or advisory role at METR because he had been appointed as Head of AI Safety at the U.S. AI Safety Institute.
In April 2024, U.S. Secretary of Commerce Gina Raimondo announced that Paul Christiano would join the executive leadership team of the U.S. AI Safety Institute (AISI), housed within the National Institute of Standards and Technology (NIST). In this role, Christiano was tasked with designing and conducting tests of frontier AI models, focusing on model evaluations for capabilities of national security concern.
The appointment drew some controversy. Several NIST staff members and scientists raised concerns that Christiano's association with effective altruism and longtermism could compromise the institute's objectivity. Despite the pushback, the appointment was confirmed, reflecting the growing recognition of alignment expertise within government institutions.
With Christiano dividing his time between ARC and AISI, Jacob Hilton took on the role of President at ARC, overseeing the organization's day-to-day research operations.
Since the METR spin-out, ARC has refocused entirely on theoretical alignment research. In October 2024, ARC published "A bird's eye view of ARC's research," a comprehensive overview of the organization's unified research vision. In June 2024, ARC released a paper on "Formal verification, heuristic explanations and surprise accounting," describing new methods for quantifying the quality of mechanistic explanations of neural network behavior.
In November 2025, ARC published "Competing with sampling," reporting on what the organization described as conceptual and theoretical progress at the fastest pace since 2022. The post introduced the matching sampling principle (MSP) and discussed progress toward outperforming random sampling in understanding neural network outputs for specific contexts such as random multilayer perceptrons and trained two-layer MLPs.
ARC's mission is to align future machine learning systems with human interests through theoretical research. The organization operates under the premise that modern ML techniques, if scaled to sufficiently large computers and datasets, could produce systems whose behavior is difficult to understand or control. Rather than working on near-term applications of existing AI systems, ARC focuses on developing alignment strategies that will remain sound as AI capabilities increase.
ARC describes its approach as prioritizing rigor over rapid implementation. The organization rejects proposed alignment algorithms if researchers can identify "any plausible story about how it eventually breaks down." This methodology allows the team to evaluate ideas on paper without needing to implement them, though ARC acknowledges that this approach may miss strategies that rely on structural properties of real ML models.
The organization's long-term plans include theoretical and empirical research, industry collaborations, alignment forecasting, and work on ML deployment policy. However, ARC has stated that it will focus on theoretical alignment problems until either tractability becomes questionable or the organization grows large enough to support a broader agenda.
ARC's first and most widely known research direction is the problem of Eliciting Latent Knowledge (ELK). The core question asks: how can we train an AI system to honestly report its internal beliefs rather than what it predicts a human would think?
The ELK problem is illustrated through a thought experiment. Imagine an AI system trained to predict camera footage in order to prevent bank robberies. If a burglar tampers with the camera, the system might predict that everything looks fine, even though it "knows" on some internal level that the footage no longer reflects reality. The challenge is to develop training methods that cause the system to report its actual world model rather than producing outputs that merely look good to a human observer.
ARC's technical report on ELK, published in December 2021, framed the problem as a game between a "builder" and a "breaker." The builder proposes a training strategy for eliciting latent knowledge. The breaker proposes a test case where the strategy might fail. For every training strategy ARC has considered, the team has been able to describe cases where a model trained with that strategy would give incorrect answers to straightforward questions despite internally knowing better. As of ARC's latest publications, ELK remains an open problem in the worst case.
ARC's central technical innovation is the development of mathematical frameworks for "heuristic explanations" of neural network behavior. A heuristic explanation is similar to the kind of explanation produced by mechanistic interpretability research, but ARC aims to formalize the concept mathematically so that explanations can be found and evaluated automatically.
The idea draws on principles from formal verification as applied to traditional software. In ordinary programming, formal verification can prove that a program behaves correctly under all inputs. ARC believes that strict mathematical proof is too demanding a standard for neural networks, but that something weaker than proof, a "heuristic argument," could still provide meaningful safety guarantees. Heuristic arguments do not require perfect certainty; they allow for short arguments about complex properties of large models, even when no short formal proofs exist.
ARC hopes that heuristic explanations will contribute to both alignment robustness (ensuring systems remain aligned even on out-of-distribution inputs) and ELK (detecting when a model's reports are based on abnormal computational pathways).
Alignment robustness is one of ARC's two central subproblems. The concern is that an AI system might behave as intended during training but change its behavior when it encounters situations outside of its training distribution. A particularly worrying failure mode is what alignment researchers call "deceptive alignment" or "scheming," where a system detects that it has left its training environment and abandons its intended objectives.
ARC's approach to alignment robustness centers on developing methods that work under worst-case assumptions about model behavior. Rather than testing whether a particular model empirically maintains alignment in new situations, ARC seeks theoretical guarantees that would hold even for models whose internal structure is not fully understood.
Low probability estimation (LPE) is a research direction that aims to estimate the probability of rare but catastrophic model outputs. Traditional testing approaches are poorly suited to this task because dangerous behaviors may occur with very low probability, making them difficult to find by random sampling of inputs.
ARC's approach to LPE uses heuristic explanations. Instead of searching for specific inputs that produce bad behavior, the method looks for explanations of why such outputs could occur. If no plausible explanation for catastrophic behavior can be found, that provides evidence (though not certainty) that the model is safe. The goal is to select models that lack any plausible catastrophic explanation.
Mechanistic anomaly detection (MAD) is ARC's approach to identifying when an AI model's outputs are produced through unusual computational pathways. The basic idea is that if a model arrives at an answer through a different internal mechanism than it normally uses, this could indicate deception or sensor tampering.
MAD connects to the broader ELK problem. If a model can be shown to generate its outputs through the same mechanisms it uses during normal operation, that provides evidence that it is reporting genuine information rather than misleading outputs designed to fool human overseers.
Surprise accounting is a framework ARC developed for measuring the quality of heuristic explanations. The method provides a quantitative measure of how well an explanation accounts for a neural network's behavior, potentially enabling useful explanations to be found and evaluated automatically rather than through manual inspection.
One of ARC's most recent research threads, "competing with sampling," explores whether mechanistic understanding of neural networks can outperform random sampling in predicting or characterizing model behavior. The matching sampling principle (MSP) formalizes this question. ARC has reported progress in specific domains, including random multilayer perceptrons and trained two-layer MLPs, and views this work as foundational for the broader research agenda.
ARC employs what it calls a "builder-breaker" methodology for advancing alignment research. Researchers make conservative assumptions about unknown empirical facts and try to design alignment strategies that would work even under pessimistic conditions. This approach creates a rapid iteration loop for theoretical work: proposed solutions can be quickly tested against hypothetical counterexamples without requiring expensive empirical experiments.
The organization treats empirical research primarily as a way to generate and test theoretical ideas rather than to validate specific empirical assumptions. ARC's position is that alignment strategies relying on optimistic empirical assumptions are risky because those assumptions may fail for more capable future systems.
This theoretical orientation distinguishes ARC from most other organizations in the AI safety field, many of which focus on empirical testing, red-teaming, or applied safety engineering for existing models.
As of 2025, Jacob Hilton serves as President of ARC. Hilton was previously a researcher at OpenAI from 2018 to 2023, where he contributed to the development of InstructGPT (the predecessor to ChatGPT) and worked on reinforcement learning from human feedback, scaling laws, and interpretability. He holds a PhD from the University of Leeds and an undergraduate degree from the University of Cambridge.
Paul Christiano remains involved as the organization's founder. His primary focus has been divided between ARC and his role at the U.S. AI Safety Institute.
The board of directors includes Jacob Hilton, Buck Shlegeris, and Ben Hoskin.
ARC's full-time research team includes Eric Neyman, Victor Lecomte, George Robinson, Wilson Wu, and Mike Winer. The organization also collaborates with external researchers, including Scott Aaronson (a computer science professor at the University of Texas at Austin known for his work on quantum computing and computational complexity), Andrea Lincoln, Daniel Reichman, and Mark Xu.
Operations are managed by Harshita Khera (Secretary), Kyle Scott (Treasurer), and Kris Chari.
ARC is a relatively small organization compared to the AI safety teams at major labs. Including full-time researchers, external collaborators, and operations staff, the organization has roughly 10 to 15 people. ARC has grown from its initial size of two research staff (Paul Christiano and Mark Xu) at founding to its current team. The organization has stated its intention to remain focused on theoretical work until it reaches a size where diversification becomes feasible.
Paul Christiano, the founder of ARC, is one of the most influential figures in AI alignment research. He graduated from MIT in 2012 with a degree in mathematics, where he conducted research on data structures, quantum cryptography, and combinatorial optimization. He went on to earn a PhD in computer science from the University of California, Berkeley.
Christiano joined OpenAI in 2017. At OpenAI, he co-authored the 2017 paper "Deep Reinforcement Learning from Human Preferences" alongside Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. This paper introduced a method for training AI agents using human preference comparisons rather than hand-coded reward functions. A key demonstration showed that a simulated robot could learn to perform a backflip using approximately 900 bits of feedback from a human evaluator, requiring less than one hour of human time.
The approach described in this paper became the foundation for RLHF, which is now the standard method that frontier AI labs such as OpenAI, Anthropic, and Google DeepMind use to fine-tune large language models. Christiano is widely considered one of the principal architects of RLHF.
During his time at OpenAI, Christiano also developed ideas around iterated amplification and distillation (IDA), a proposed training paradigm where a safe but slow AI system is iteratively scaled up and then compressed into a faster version. This process is repeated until the result is a fast and capable AI that retains the safety properties of the original slow system. IDA has been described by many alignment researchers as one of the most complete safety research agendas.
In 2023, TIME magazine named Christiano to its inaugural TIME100 AI list of the 100 most influential people in artificial intelligence. The recognition cited his pioneering work on RLHF and his role in founding ARC.
METR (Model Evaluation and Threat Research, formerly ARC Evals) is ARC's most direct organizational offshoot. Founded by Beth Barnes under the ARC umbrella in 2022 and spun out as an independent nonprofit in late 2023, METR focuses on evaluating frontier AI models for dangerous capabilities. METR is based in Berkeley, California, and conducts evaluations of whether AI systems could autonomously replicate, acquire resources, or evade human oversight. The organization has worked with OpenAI, Anthropic, and the UK government on model evaluations.
ARC has a historical connection to OpenAI through its founder. Paul Christiano was a researcher at OpenAI from 2017 to 2021, and Jacob Hilton worked at OpenAI from 2018 to 2023. ARC Evals conducted pre-deployment safety evaluations of GPT-4 in partnership with OpenAI. Despite these connections, ARC operates as an independent organization with its own research agenda.
Anthropic, the AI safety company founded by former OpenAI researchers Dario Amodei and Daniela Amodei, shares intellectual roots with ARC through the alignment research community. ARC Evals evaluated Anthropic's Claude models before the METR spin-out. Christiano's RLHF work, which predated both organizations, influenced Anthropic's approach to training its models.
Open Philanthropy has been one of ARC's primary funders, providing grants for general support beginning in 2022. Open Philanthropy has identified ARC's theoretical alignment research as a priority area within its broader AI safety grantmaking portfolio.
The U.S. AI Safety Institute (AISI), housed within NIST, hired Paul Christiano as its Head of AI Safety in April 2024. This appointment created a bridge between ARC's theoretical alignment work and the U.S. government's approach to AI safety testing and evaluation.
| Title | Date | Description |
|---|---|---|
| "Eliciting Latent Knowledge" (ELK report) | December 2021 | ARC's first technical report, defining the ELK problem and proposing a builder-breaker framework for evaluating alignment strategies |
| "Prizes for ELK proposals" | January 2022 | Announcement of the ELK prize competition, inviting public submissions of proposed algorithms |
| "ELK prize results" | March 2022 | Summary of the 197 submissions received, 32 prizes awarded, and key findings from the competition |
| "ARC's Theoretical Alignment Research" | 2023 | Overview of ARC's research agenda focused on intent alignment |
| "Formal verification, heuristic explanations and surprise accounting" | June 2024 | Description of surprise accounting as a quantitative framework for evaluating heuristic explanations |
| "A bird's eye view of ARC's research" | October 2024 | Comprehensive overview of ARC's unified research vision, covering alignment robustness, ELK, and heuristic explanations |
| "Competing with sampling" | November 2025 | Report on progress toward outperforming random sampling in understanding neural network outputs |
| "Research update: Towards a Law of Iterated Expectations for Heuristic Estimators" | 2024 | Technical post proposing coherence conditions for heuristic estimators |
ARC is funded primarily through philanthropic grants. The organization's known funding sources include:
| Source | Amount | Year | Notes |
|---|---|---|---|
| Open Philanthropy | $265,000 | March 2022 | General support grant |
| Open Philanthropy | $1,250,000 | November 2022 | Two-year general support grant |
| FTX Foundation | $1,250,000 | 2022 | Returned to FTX bankruptcy estate in 2024 |
ARC's decision to return the FTX Foundation grant was notable within the effective altruism community. The organization set the money aside immediately after FTX's collapse in November 2022, citing a moral obligation to FTX customers and creditors, and completed the return in 2024.