These are the factions debating how to make AI safe
It's more complex than doomers vs accelerationists.
Hello,
Before we crack into this week’s piece, I wanted to let you know I’m 38007 ft above the southern tip of Japan right now, en route to the UK. I’m moving to London to accelerate my journalism career as part of the Tarbell Fellowship. It isn’t the first time I’ve left New Zealand. But it is the first time I have departed without a return ticket.
This weekend, I’ll attend Effective Altruism Global, a conference that brings together a wide network of people trying to do good, including many AI safety researchers. Next week, I will (hopefully) start my placement at a publication. A few weeks later, I’ll head to Oxford for the Tarbell Summit on AI journalism. What I’m trying to say is this. A lot of cool stuff is happening very soon. Subscribe to follow along.
These are the factions debating how to make AI safe
Last week, at an international summit on AI in Seoul, the US announced the creation of a global network that will connect the US’s AI safety institute with counterparts in the UK, Japan, Canada, and other allies. They’re encouraging other nations to join. One would be forgiven for thinking there’s a consensus emerging on how to make artificial intelligence safe.
Look closer, however, and you will find deeply divided factions that disagree on the nature of the risks posed by AI —and what we should do about them. What were once philosophical disputes between niche communities on X (formerly known as Twitter) now have real consequences.
The US Senate has reportedly started to split over governing AI, with some lawmakers believing they need more information before they take decisive action. Meanwhile, the number of groups lobbying the US federal government on AI has tripled since 2022. When OpenAI’s CEO Sam Altman was fired and rehired last November, many speculated that differing views over safety had caused the rupture.
The debate over AI is sometimes framed as a struggle between two factions: Accelerationists, on one side, who think AI will usher in a utopia-like society, and, on the other, doomers, who fear AI could wipe out humanity. In reality, the debate is far more complex.
Part of the challenge is that AI is not a single thing but rather the catch-all term for a collection of technologies that are poised to touch many domains, including law enforcement, information, and scientific discovery. That means that what each faction says we should or shouldn’t do about AI doesn’t just carry implications for Silicon Valley. It will have flow-on effects for millions of Americans and billions of people around the world.
The doomers
Doomers believe that, short of a technical miracle, an AI system with greater-than-human intelligence will spell the literal end for humanity. How? Rather than a system hell-bent on world domination for its own sake, doomers fear that a powerful AI system chasing a poorly specified or poorly understood goal will lead to destructive yet unintended consequences. Imagine, for example, tasking a future AI agent with reducing our carbon emissions to zero. After identifying that human activity accounts for the main sources of carbon dioxide, the system begins eliminating humanity.
Trying to capture what we really want in a way AI systems understand is known as the alignment problem. Leading AI developers like OpenAI, DeepMind and Anthropic all say they are researching ways to align their models, but doomers claim that even if these approaches work on today’s systems, they’ll likely fail as AI surpasses human intelligence and outsmarts its safety mechanisms. Many doomers believe such a system is years, not decades, away.
The best way to prevent AI from killing humanity, according to doomers, is not to build it. In the build-up to last week’s AI Summit in Seoul, the advocacy group PauseAI organised a wave of protests in thirteen countries with the goal of persuading decision-makers to implement a global halt on training large AI systems. In an op-ed for TIME published last year, the prominent doomer Eliezer Yudkowsky, went further, writing that nations should be willing to use airstrikes or even nuclear weapons to enforce a global ban on AI training. Even the horrors of war seem insignificant if you believe we’re years away from building AI systems that will almost certainly destroy humanity. However, both the level of risk and the time it will take to develop smarter-than-human AI agents are disputed by critics, including many experts.
The accelerationists
Accelerationists (also known as effective accelerationists or e/accs) start with the observation that all the wonders modern life offers—medicine, lighting, instant communication, travel—are the result of technological innovation. AI, they say, will take this to another level by further speeding up progress.
In an influential blog post published last October, venture capitalist and accelerationist Marc Andreessen wrote that we should view AI as a “universal problem solver” because it will help humanity cure disease, stop global warming, and more. With AI’s potential to save so many lives, Andreessen says we have a moral duty to build it as fast as possible. Andreessen’s firm is in talks with the Saudi government over the creation of a 40 billion dollar deal to fund AI development. If the deal goes ahead, the resulting partnership would be the world’s largest investor in AI.
Rather than AI replacing or killing humans, accelerationists believe that AI will augment human intelligence. Some believe this will eventually entail humans merging with AI. They see those who want to slow down AI development as delusional, at best. At worst, those who say we should slow down, or “decels”, pay lip service to the risks of AI with the hope that doing so will usher in regulation that locks out potential competitors for their own financial gain. The best way to ensure we continue on our trajectory to AI is to avoid regulation and allow the market to do its thing. To that end, Andreessen has said he’ll support any politician who opposes regulation that could slow innovation.
AI safety
Those in the AI safety camp believe that future AI systems could help drive scientific and technological breakthroughs, but they also believe that AI poses great risks. Unlike doomers and accelerationists, they deny that either outcome is inevitable. They believe that we can balance the risks and benefits through a mix of technical solutions and regulation. Because what we do now could determine whether AI turns out to be extremely good or bad, people working in AI safety often think it’s the highest priority issue in the world.
AI safety-ists worry about the threat of AI going rogue but see this as just one threat in a broader framework of risks. AI could magnify other long-running societal-level threats, for example, by making it easier for bad actors to acquire chemical or biological weapons or for authoritarian regimes to create propaganda and track dissenters.
Whereas accelerationists see markets as the best way to drive innovation, those in AI safety fear unfettered competition—between private companies or states—will result in safety being compromised in favor of building more powerful systems. To prevent such a race, those in AI safety want both national and international governance of AI, which could include licencing and mandatory testing of AI systems for dangerous capabilities before deployment.
Key ideas emanating from AI safety circles, such as requiring companies to evaluate whether their systems could aid in the creation of weapons of mass destruction, featured prominently in Biden’s AI AI executive order. At least publically, leading AI developers like OpenAI and Anthropic have all invited the kind of regulation those in AI safety want. All of this suggests the AI safety faction wields a lot of power.
However, according to TIME, in private, those same companies show less enthusiasm for mandatory guardrails. Earlier this month OpenAI disbanded a key safety-focused research team after the team’s two leaders, Ilya Sutskever and Jan Leike, resigned. In the months prior, a string of other safety-conscious employees either quit or were pushed out. OpenAI is not the only company struggling to keep safety-conscious staff. Last year, the man often touted as the “godfather of AI”, Geoffrey Hinton, quit his role at Google so he could speak freely about the risks the technology poses.
AI ethicists
AI ethicists say that focusing on doomsday scenarios diverts attention from the societal harms AI is already causing. Deepfake pornography has ruined the lives of young women, while biased algorithms have already led to the wrongful arrest of one man in Michigan. Similar biases in training data could be used to deny people medical insurance, housing, or welfare—without the person ever knowing why.
AI ethicists reject the notion that we’re on the cusp of building AI that comes anywhere near human intelligence. They say, large language models (LLMs)—the tech behind chatbots like ChatGPT—mimic humans rather than demonstrate a genuine understanding of language. Because LLMs lack comprehension, they’re unlikely to take over the world, but it does give them a nasty habit of making stuff up. Some fear that AI-generated misinformation, coupled with the automation of jobs or the use of facial recognition to surveil political enemies, could have destabilizing effects on democracy. AI ethicists also draw attention to the ethically questionable practices involved in creating today’s AI models, such as their reliance on copyright material and feedback from people who, TIME reports, are paid less than $2 per day. By ignoring these immediate harms in favor of existential threats, we play into the hands of the big tech companies.
AI ethicists want to see mechanisms for holding developers accountable for bias or discriminatory systems. Many believe that rooting out bias will also require more systemic changes, such as increasing diversity among those building AI. AI ethicists also want the use of AI banned in specific contexts, such as the use of facial recognition in law enforcement or using algorithms to determine criminal sentencing.
AI ethicists have courted the attention of lawmakers. The White House’s AI executive order acknowledged the potential for AI to deepen inequities and pledged to “address algorithmic discrimination.” Several bills introduced to Congress this term seek to address the harms caused by deepfakes. The EU AI act, which passed its final vote earlier this month, restricts the use of real-time facial recognition technology in public spaces.
Leading AI developers have been slower to take AI ethicist’s concerns seriously. Back in 2020, Timnit Gebru, who co-led the AI ethics team at Google, was fired after she refused to withdraw a paper she co-authored examining the ethical pitfalls of LLMs. Since then, Google has restructured Gebru’s former team, while Meta has disbanded its responsible AI team. OpenAI, which has published reports examining other risks has been comparatively silent on discrimination. Anthropic is a notable outlier. In December, the company published a paper exploring ways to mitigate discrimination in LLMs.
Thank you to Tom Ough for his feedback on an earlier draft of this piece.
Stories I’m following
It was a pretty scandalous fortnight at OpenAI:
People quickly noticed that OpenAI’s new voice, “Sky”, sounds a lot like Scarlett Johansson’s character from the 2013 film Her. Johansson said she was “shocked” and “angered” because OpenAI had contacted her about using her voice, and she declined. (Verge)
OpenAI might not be the straight-forward bad guy here. It looks like the voice actress behind Sky was hired months before the company reached out to Scarlett Johansson (Washington Post).]
As mentioned in this week’s article, OpenAI has shut down one of its safety-focused teams after its two leaders resigned. (Vox)
OpenAI has reportedly gagged staff who left the company by forcing them to choose between signing an NDA or forfitting their equity in the company. (Vox)
I’m publishing this newsletter in the hope that we can learn together. That means I hope you take something away from each edition. But it also means that you can teach me something.
If you’ve got thoughts on how I can do better next time, or if something resonated with you and you want to let me know, leave a comment. I read each one.