Our Theory of Change
What day-to-day processes should we follow? How do decisions that we make today work towards our ultimate goals?
At Convergence, we’ve developed a Theory of Change detailing an outcome-based, high-level strategic plan on how to mitigate existential risk from transformative AI.
Understand
How we conduct research to investigate the societal implications of transformative AI
Holistic Strategy Research
Conduct research into meta-level strategies to maximize AI safety
Scenario Research
Publishing Scenario Reports
Release a research corpus on key AI scenarios and threat models
Governance Research
Publishing Governance Reports
Release a research corpus on key governance strategies and recommendations
Act
How we deploy insights from our research to inform key parties
AI Safety Consensus
Build convergence within the AI safety community on scenarios & governance
Public Awareness
Raise public awareness of key AI scenarios
Policymaker Advocacy
Advise government actors on key governance recommendations
Outcomes
How our work leads to more effective decision making in AI safety
Increased Alignment
Policymakers and the public are more focused on key scenarios and governance strategies
Better AI Governance
Government actors establish recommendations through legislation, executive action, and international agreement
More Effective Funding
Grantmakers support more projects contributing to key governance strategies
Better Coordination
AI safety organizations improve coordination as a result of greater consensus
Ultimate Goal
Mitigation of existential risk from AI
Our Key Beliefs
Existential risk research is critical and neglected.
It's clear to us that the accelerating development of AI technology leads to a high risk of critically dangerous outcomes on a global scale if not managed properly, up to and including the extinction of the human race.
It's also obvious that the number of people working to identify, categorize, and minimize these risks are orders of magnitude too small.
Because there aren't clear financial incentives and the potential risks are universal and long-term, existential risk research for AI is a "tragedy of the commons," akin to where climate change was 20 years ago: not worth investing in until it's too late. This must be changed.
Technical AI alignment research is important but not enough
The vast majority (roughly 75%) of individuals working in AI safety are focused on technical alignment - the problem of aligning frontier AI models with the intended goals, preferences, and ethical principles of the humans designing them.
Though technical alignment is critical, it's only one component of the broader effort to mitigate risks from AI technologies. It’s clear to us that successfully designing well-aligned AI models still does not prevent the development and consequent misuse of such models by malicious human parties.
As the costs to develop high-powered AI systems rapidly lower, more and more organizations will gain the ability to develop these AI models. As we’ve found with any military technology throughout history, human parties have wildly varying priorities and goals, many of which can and will cause catastrophic harm even if implemented by a perfectly aligned AI.
As a result, we must have global systems in place such that society is resilient to the effects of transformative AI, no matter whether AI models are misaligned or their human developers have malicious goals.
Minimal foundational research exists on AI scenarios or specific AI governance proposals
Though many one-off papers have been written about potential AI futures, there doesn’t exist any comprehensive reports modeling the most likely AI scenarios, the parameters that might lead a scenario to happen, or the societal threats that these AI scenarios might entail. We don’t have answers to questions such as:
What are the possible paths of development for AI technology, and how likely is each path?
What are the most likely set of negative societal outcomes to result from the acceleration of AI capabilities?
A similar problem exists in AI governance. Though there are dozens of public calls to action around key governance proposals, there is very little detailed, systematic analysis of these proposals to answer questions such as:
Is a governance proposal feasible to be implemented in today’s geopolitical climate, and what would the details of such a proposal look like?
What are the negative externalities of implementing such a proposal?
Because we don’t have this foundational research, interested parties (including industries, researchers, policymakers) must currently consider dozens of scattered resources over many weeks before arriving at an informed position.
Even after gaining expertise in one domain, interested parties have no tools to help compare the likelihood or risk profile of other scenarios & interventions. They’re likely to develop blinders towards their area of focus and overstate its importance.
Our Methodology
Step 1
Understand
How we conduct research to investigate the societal implications of transformative AI
Holistic Strategy Research
We spend at least 20% of our time conducting meta-level strategic investigation and reflection to determine the best course of action within AI Safety - both for ourselves, and for the community as a whole. This branch of research guides the overall direction of Convergence, including our two current research agendas, and allows us to remain flexible and opportunistic in the face of a rapidly shifting environment.
Projects in this category include creating effective theories of change, advocating for strategy research, and developing new conceptual frameworks such as risk modeling, information hazards, and downside risks.
Scenario Research
Threat Model
Descriptions of and proximal pathways to existential catastrophes. They are sometimes also referred to as “hazards,” “failure modes,” “existential risks,” or “x-risks.”
Example: Societal outcomes of the misuse of a frontier AI system capable of designing biological weapons.
At Convergence, we’re launching a set of foundational technical reports on AI scenarios early in 2024. These technical reports will evaluate upcoming likely AI scenarios according to two key dimensions: strategic parameters and threat models.
Roughly, these two forms of research can be mapped to:
Strategic Parameters: Describing the inputs that impact how AI technology progresses
Threat Models: Describing the real-world outputs and outcomes of AI technological advancement
Combined, these two approaches should give us a comprehensive overview of potential future states of the world in response to the development of AI technologies.
Governance Research
Theory of Victory
A high-level course of action for responding to a scenario or set of scenarios that describes how humanity successfully navigates the transition to a world with advanced AI.
Example: A multilateral international monitoring system emerges and prevents unsafe AI development.
Our scenario research will inform our work identifying and analyzing the best governance strategies that can be conducted in response to key scenarios.
At Convergence, we plan to launch a detailed report on effective AI governance strategies, and potential Theories of Victories, early in 2024. This will be closely aligned with our scenario research.
Simultaneously, we are also developing a research agenda providing a systematic framework to evaluate and consider specific governance recommendations. We will be conducting a deep analysis into key recommendations such as:
What does the political landscape look like for the U.S. to require the registration and transfer reporting of key AI chips?
What types of emergency powers could the U.S. government be given control of to stop the distribution or training of a dangerous AI model?
What domain-specific safety assessments are critical to identify dangerous capabilities of AI models?
Step 2
Act
How we deploy insights from our research to inform key parties
The AI Safety Community
Within the AI safety community, our goal is to build consensus on the most critical AI scenarios, and the optimal governance interventions necessary to improve them. Though we’re focused primarily on reducing existential risk, we’re also identifying ways in which we can most effectively support scenarios that result in humanity flourishing.
To achieve this goal, we’re looking into coordinating working groups, seminars, and conferences aimed at bringing together top AI safety researchers, aligning on key research, and presenting those topics externally.
Policymakers & Thought Leaders
For key individuals who will have a massive impact on the success of AI safety, our work is intended to accessibly summarize complex domains of information. In particular, we're developing the critical reference hub necessary to bring parties up to speed rapidly. Our work allows policymakers to compare and contrast scenarios and interventions, and learn about the AI safety consensus on important issues.
To achieve this goal, we’re actively writing policy briefs for key governance proposals, performing reach-outs to leading governance bodies, and advising several private organizations. In particular, we’re advocating for a small number of critical & neglected interventions.
The General Public
We recognize that in order to effectively enact governance interventions, we need the endorsement and concern of the constituents behind policymakers. As a result, a high priority of ours is to determine how best to distribute and popularize the lessons gained from our research. Our focus is building awareness and immediacy to the existential risk of AI.
To achieve this goal, we’re currently launching a number of initiatives, including a book on AI futures and a podcast with key public individuals on AI outcomes.
Step 3
Outcomes
How our work leads to more effective decision making in AI safety
Increased Alignment
We expect that our work will help the community:
Share a common vocabulary and knowledge base when discussing key scenarios and recommendations
Unite around certain governance recommendations and propose effective strategies to execute them
Align resources onto key projects that best address our most neglected scenarios.
Better AI Governance
Convergence’s research will tremendously assist organizations seeking to create or influence governmental policies. We expect to see benefits for groups:
Proposing and writing the design for effective policies, governmental bodies, or task forces.
Providing advisory and research support to organizations working on passing policies.
Passing legislation via lobbying, political campaigning, or coalition-building.
More Effective Project Funding
From distributing our research, we expect to see:
Better identification of impactful projects for external funders
More effective funding-matching ecosystems pairing funders to projects
More funding for projects focused on around key interventions
Better Coordination
We expect to see improved coordination among AI safety researchers leveraging our foundational research. Our impact could include the following:
More effective communication with the general public and policymakers about key AI scenarios to avoid
Providing advisory and research support to organizations working on passing policies.
More workshops and conferences around key governance strategies
Ultimate Goal
Mitigation of existential risk from AI.
We are actively forging a future where AI is harnessed safely and ethically. Through our systematic research blended with proactive advocacy and community engagement, we hope to move the needle on the likelihood of global existential risk.
Through our research, we endeavor to demonstrate a clear, logical path from action to impact, underlining our commitment to collaboratively shaping a flourishing future with AI.