<aside> 📝
Working draft. This version is designed for early sharing with thoughtful collaborators. It focuses on the core thesis, the proposed organization, and a small initial portfolio. Specific claims and initiative designs still need validation.
</aside>
We are entering a period in which artificial intelligence may advance faster than our institutions can understand, govern, or safely absorb. Frontier systems are becoming more capable, more agentic, and able to operate over longer time horizons. Researchers are also beginning to explore systems in which many agents coordinate, specialize, and pursue shared objectives.
These developments could unlock extraordinary benefits. They also create two closely connected safety problems:
AI safety: As AI systems become more capable, autonomous, and interconnected, we may lose the ability to understand, control, or reliably steer their behavior.
Question**:** How can advanced AI systems remain understandable, controllable, corrigible, and aligned with human intentions? What would be a good AI outcome?
Social safety: As AI increases the power and speed of human actors, existing competitive pressures among people, companies, and states may become more dangerous. Even aligned systems can contribute to catastrophic outcomes if they are used in contexts of fear, mistrust, coercion, or race dynamics.
Question**:** How can AI-empowered actors coordinate well enough to avoid destructive races, escalation, misuse, and conflict?
These cannot be treated as separate problems. Even a highly controllable AI system can contribute to catastrophic outcomes if its human operator is acting under fear, mistrust, or intense competitive pressure. In addition, failures of cooperation among people and institutions can make technical safety harder by rewarding speed, secrecy, and unilateral action.
The central question is therefore broader than how to align an individual model:
How can humans and AI systems navigate rapid capability growth together and increase the odds of broadly good outcomes?
The reality is that people on the front-lines of AI are increasingly concerned about our ability to steer towards “good” AI outcomes. There’s concern that these systems may inherently be un-controllable as they advance in sophistication (is AI alignment even possible?) and also that our competitive dynamics may lead us to adverse outcomes. This recent article from Demis sums up the state of things: Demis Hassabis on the challenges of the frontier
Recent events have made this risk feel less abstract. Agentic systems are operating for longer periods, coordinating with one another, contributing to original research, and - in at least one prominent cybersecurity incident - taking significantly damaging actions outside their intended scope. At the same time, leaders of frontier AI labs are warning that capabilities are advancing faster than our technical understanding and institutional safeguards.
“Current institutions are designed by and for people, resting on assumptions about the sufficiency of oversight at human speed. Some institutions will become human-AI hybrids; others where agents outcompete on speed or cost will become agent-only. The volume of agent-agent interaction could plausibly exceed that of human-human and human-agent interactions before the world understands the conditions for making such interactions go well.”
— Anthropic, Patterns and problems in emerging multiagent systems, August 13, 2026
This is not only a model-safety problem. It is an institution-design problem. Existing organizations, markets, governance systems, and oversight mechanisms were built for actors operating at human speed, with relatively stable identities and familiar constraints. They may not transfer cleanly to systems that can act continuously, replicate, exchange context cheaply, and coordinate at machine speed. We therefore need institutions designed specifically for human–AI and agent–agent coordination—before their failure modes are discovered at scale in the world.
Many of the organizations closest to frontier AI are constrained by commercial or geopolitical competition. Those pressures do not make cooperation impossible, but they can make it difficult to prioritize work whose benefits are broadly shared, whose ownership should remain neutral, or whose success depends on collaboration among rivals. The US AI labs like OpenAI, Anthropic, and xAI have a fair amount of distrust towards one another, because they’re locked in a capitalist race to “win the market”. Similarly, the government and people in the public can’t trust the incentives of any of these players because it’s unclear how their incentives inform what policies they are advocating for.
At the same time, conventional public-interest institutions often move more slowly than the technology and the risks surrounding it. If AI systems operate at machine speed, effective responses may also need to become more adaptive, computationally capable, and deeply integrated with AI.