An Anthropic researcher has resigned over fears that the race to develop increasingly powerful AI could produce self-improving systems that humans can no longer control, The Wall Street Journal reported today.
Jacob Coxon, a 27-year-old researcher specializing in model pretraining, told the Journal he was leaving the AI industry rather than participate in what he sees as a race by Anthropic, OpenAI and other developers toward self-improving AI. Coxon previously worked at OpenAI before joining Anthropic, in part because of the company’s emphasis on AI safety.
“We’re on track for a lot of the most aggressive of these scenarios where by the end of next year things could be out of control already,” Coxon told the Journal.
His departure is notable because Anthropic has positioned itself as one of the frontier AI industry’s strongest proponents of safety measures. Coxon said he believes the company’s safety efforts are sincere, but that competitive pressures make trade-offs inevitable as U.S. developers race one another and Chinese rivals.
The concerns extend beyond Coxon. Anthropic scientist Evan Hubinger responded to the resignation by saying he believes there is a greater than 10% chance AI could kill all humans within the next decade. He also said Anthropic does not yet have a plan for aligning superintelligent AI and is not clearly on track to develop one.
Coxon’s resignation comes amid increasing debate within the AI industry about systems capable of improving their own capabilities. Anthropic CEO Dario Amodei and other AI leaders have warned about the potential dangers of increasingly autonomous models while continuing to develop more capable systems.
Coxon said he now believes no individual company can responsibly develop AI that surpasses humans across a broad range of tasks without government intervention or a coordinated slowdown across the industry.