The Scientist Building an AI With No Reason to Kill Us

TLDR

• AI researcher Ben Goertzel believes advanced AI could pose an existential risk, but argues the answer is to build AI with beneficial goals rather than halt development.
• He wants AGI to be open and decentralized so no single or small group of companies or governments control it.
• He is building an open-source, decentralized AI architecture designed from the beginning with beneficial goals.

Ben Goertzel, the AI researcher who popularized the term artificial general intelligence or AGI, believes a sufficiently powerful AI could wipe out humanity.

His answer is not to stop building it.

Instead, Goertzel is working on an alternative: an open-source, decentralized AI architecture designed from the beginning with beneficial goals – one that he hopes could eventually become a foundation for artificial general intelligence that no single or small group of companies or governments control.

Goertzel argues that the possibility AI could kill humanity does not mean it inevitably will. The more important question is what motivates an intelligence once it becomes smarter and more powerful than its creators.

“Yes, I think that it is possible for AI to wipe out the human species,” Goertzel said in an interview with The AI Innovator. But “there’s no reason to assume the AI is going to want to kill off humanity.”

That distinction goes to the heart of Goertzel’s argument about the race toward AGI, broadly understood as an AI capable of performing intellectual tasks at the same level as humans across many different domains.

He thinks the industry is focusing too much on securing AI after it is built and not enough on what kind of intelligence humans are creating, what values are built into it and who ultimately controls it.

Goertzel, founder of SingularityNET and the startup BGI Labs, has spent decades pursuing AGI. The term comes from a book title he chose for the 2006 book “Artificial General Intelligence,” which he co-edited with Cassio Pennachin. (He later found out that physicist Mark Gubrud had coined the term in 1997 in a paper.)

Goertzel believes the industry is approaching AGI. But the ‘Singularity’ – the point at which AI-driven advances become so rapid and profound that what happens next becomes hard for humans to predict – remains further away, he said.

“My phone is not yet making five Nobel Prize-level discoveries every minute,” he said. “We do not yet have AIs that can do frontier science or engineering on their own without human assistance.”

The limitations are apparent to businesses using AI agents today, he said.

“You can’t yet actually just put a bunch of AI agents in charge of your business and have them just do the whole thing.”

Could AI actually kill us?

Goertzel does not dismiss the existential-risk argument that has moved from academic circles into mainstream political debate.

A sufficiently advanced AI could potentially discover biological weapons beyond human capabilities, manipulate people or computer systems or find other ways of causing catastrophic damage, he said.

“You might imagine that the AI can’t get into the nuclear control facility. But it can get into the bank, it can pay the security guard a million bucks to let it in,” Goertzel said.

But capability and motivation are different questions.

“The issue is more, ‘what is the AI motivated to do?’” Goertzel said. “Any of us could kill a lot of people if we wanted to, and yet a very small number of people become murderers. … You would imagine an AI vastly smarter than people could find ways to do what it felt like without harming people at all.”

He compares the development of advanced AI to raising children. Parents cannot guarantee what their children will eventually become, but upbringing, experience and values influence them.

“We’re trying to build a superintelligence that will help us do stuff and embodies our values,” Goertzel said. “If you raise them to be vicious killers or greedy, remorseless profit seekers, then maybe that’s what they will grow up to be.”

The analogy is provocative, but the underlying technical problem is real and unresolved: Researchers do not currently know how to guarantee that the objectives of a system more capable than its designers would remain aligned with human intentions as the system learns, acts and improves itself.

Listen to a summary of this article in an AI-generated podcast:

Goertzel believes the answer lies partly in giving AI an explicit motivational architecture. “To me, it’s about getting the right motivational system into the AI,” he said.

The system should begin with beneficial goals and then be designed so that, as it changes and improves, it does not simply abandon them.

“You want it to improve itself in a way that respects the spirit of what it began with. There’s no reason that’s not possible,” Goertzel said.

But “it happens to be not the kind of system that people are building in the frontier labs right now because, in essence, they’re building systems with no motivations and goals whatsoever, and they’re building systems that don’t know who and what they are and what their purpose is and how they’re embedded in the world.”

The risks that are already here

Goertzel also argues that the debate over whether AI can kill all humanity obscures an immediate problem: Harms done by existing AI systems today.

These are essentially “obnoxious people using AI to do obnoxious things to other people,” he said. “This is more of the current situation. Let’s build AI drones to kill innocent people all over the place. Let’s use AI to suck up everyone’s data, repackage it, and sell it back to them. Let’s use AI to spy on everyone and then arrest them for doing nothing wrong.”

“This is real,” he added. “Some people like to worry a lot or make a lot of noise about the hypothetical existential risk of long-term AGI to distract attention from all the nasty things actual people are doing right now, which are very avoidable if you wanted to just use AI differently.”

“We could be making AI school teachers and elder care workers instead of spies and killer drones,” Goertzel continued. “It’s just not what we’re putting our money into.”

That distinction leads Goertzel to reject one increasingly prominent response to existential AI risk: stopping or pausing development.

The recently introduced Ban Artificial Superintelligence Act from Sen. Bernie Sanders, I-Vt., and Rep. Greg Casar, D-Texas, would permanently prohibit development and deployment of artificial superintelligence and temporarily pause advanced AI development until a new federal regulator establishes safety rules. Individuals who violate or circumvent the restrictions could face up to 20 years in prison.

Goertzel regards such proposals as unenforceable in practice: “What sense does that make? There’s no one in charge of the world. The U.N. can’t say, ‘stop AI,’ and then Trump and Xi Jinping and Putin will listen. Pausing AI is not an actual plan because if China promised to pause it, the U.S. wouldn’t believe them and vice versa.”

Even if the U.S. agreed to slow down, what it means practically speaking is unclear. “Is Google pausing making smarter search? Is Goldman Sachs pausing making a smarter prediction agent?” he asked.

And unlike nuclear weapons, whose development requires access to highly regulated uranium, people can spin up AI much more easily.

“How do you stop me from getting 20 servers in my garage and trying to make a smarter AI and networking it with other people’s servers?” Goertzel said. “You’d need a full-on fascist oversight of everyone’s computer chips.”

Goertzel’s unlikely agreement with Trump

That position puts Goertzel in an unusual place politically.

He said he has generally sympathized with politicians on the left but finds himself closer to President Trump’s position on AI development.

“I’m a bit disconcerted that Donald Trump is mostly doing what I think is the right thing about AI, because I’m not a fan of many of the guy’s policies,” Goertzel said. “But on the whole, I think his instinct for AI is not that terrible.”

Trump has opposed broad new restrictions on AI development while emphasizing U.S. technological leadership. A June executive order explicitly rejected mandatory licensing, pre-clearance or permitting requirements for developing or releasing AI models, while creating voluntary mechanisms for government evaluation of some advanced systems.

Goertzel said the basic instinct – that the U.S. cannot simply stop while the rest of the world continues – makes sense to him.

“We’re not going to ban this or slow this down because the rest of the world isn’t,” he said in describing that position. “So we should just be building it and doing interesting things with it. I think in the end, that’s probably the best stance.”

Goertzel worries that a regulatory system requiring government approval for advanced AI could disproportionately favor the largest AI companies, which have the resources to comply, while effectively shutting independent developers out.

He described the outcome he fears as being effectively “banned from making open source AI.”

The proposed Sanders-Casar legislation does not specifically ban open-source AI. Its restrictions are based on the capabilities of AI systems, regardless of whether they are proprietary or open. But Goertzel argues that the cost and complexity of complying with such a regulatory regime could make advanced open-source development much more difficult.

Anthropic has advocated regulation of frontier AI risks, but CEO Dario Amodei has explicitly said the company has “never advocated for a ban on open-weights models” and called open-weight models without dangerous capabilities a public good. OpenAI, meanwhile, recently called for mandatory national safety requirements based on model capabilities.

But Goertzel’s concern about concentration leads directly to the second half of his proposed solution.

It is not enough, he argues, to make AGI benevolent.

Nobody should own it.

An AI with beneficial goals

Goertzel believes today’s dominant AI architecture is poorly suited to building the kind of intelligence he envisions.

A large language model is fundamentally trained to predict tokens. That’s its purpose and the model tries to become better at what it’s designed to do.

Instead, Goertzel wants to build “what in the good, old-fashioned AI field you would call a cognitive architecture” – a system with explicit goals, reasoning, a model of the world and a model of itself.

“What we need to do, which is also hard, is make something smarter than ChatGPT and Claude, but which is based on a genuine cognitive architecture with beneficial goals at the top,” he said.

His teams at SingularityNET and BGI Labs are developing Omega, an agent framework that combines open-weight LLMs with symbolic AI systems designed to perform reasoning and represent goals and world models.

Goertzel argues that open models make that experimentation easier because developers have greater access to and control over the underlying systems.

Asked what he thought of Anthropic’s ‘Constitutional AI’ effort that would imbue human values in their AI models, he said its approach is incorrect.

Anthropic’s ‘Constitutional AI’ goes well beyond a simple list of prohibited actions but also attempts to cultivate judgment and values such as honesty, thoughtfulness and concern about AI’s effects on people. But it’s still not enough.

“Honestly, it’s more like you give your kid a bunch of text to read about how to be good, and then when … they don’t really obey what it said, you put a bunch of rigid rules on them, and then they go about trying to bypass those rules,” he said. “None of that is a substitute for engaging with the young mind in shared beneficial activity.”

Goertzel wants AI systems to learn benevolent behavior partly through experience – by having AI agents teach children, for example, or assist people in hospitals – rather than relying solely on rules telling them how to behave. Through those interactions, the AI learns compassionate behavior that becomes part of the system’s developing model of itself.

The Linux model for AGI

Another part of Goertzel’s project may be just as ambitious.

He wants the core technology behind AGI to operate more like Linux and the internet than a proprietary model owned by a technology company.

“The first key is the core AGI code is (to be) open-source code,” he said.

That does not mean everything built with it would have to be free or open.

Goertzel envisions companies building proprietary commercial products on top of an open AGI foundation — roughly the way enormous businesses have been built using Linux and open internet protocols.

The model matters because Goertzel sees concentrated ownership itself as an AI risk.

The only way to ensure that the first AGI is owned and guided by all of humanity is to distribute its infrastructure across different owners and legal jurisdictions, he said.

“The internet and Linux operating system are our shining examples,” he said. “They are global. They have not been captured.”

To be sure, governments and corporations do exert influence over internet infrastructure and open-source ecosystems. But no single company owns the internet or Linux, and their open, distributed architectures have allowed them to survive the rise and fall of individual companies, governments and technologies.

That is the open, decentralized architecture Goertzel envisions for AGI.

Millions of computers – eventually

The vision also has a practical motivation: compute. Goertzel’s projects do not have anything approaching the computing resources of the largest frontier AI labs.

“We have like hundreds to low thousands of GPUs and CPUs,” he said. “We don’t have millions.”

But his decentralized framework could theoretically pool computing resources spread across large numbers of machines.

To be sure, not every AI workload lends itself to that architecture. Training enormous neural networks can require thousands of GPUs connected at extremely high speeds inside data centers. Goertzel acknowledges that his project still needs larger server farms.

But other algorithms can be distributed much more widely, he said, potentially using independent data centers, converted crypto-mining facilities or even consumer devices.

The result would be a hybrid infrastructure: some tightly connected compute where necessary, other workloads spread across a decentralized network.

BGI Labs’ strategy is also designed to reduce the amount of training required. Rather than build every component from scratch, Goertzel wants to use existing open-weight LLMs as part of a larger cognitive system.

However, decentralization could make a dangerous system harder to stop. Open-source AI can broaden innovation and scrutiny, but it can also give malicious actors access to powerful capabilities.

Goertzel does not claim those tensions disappear.

His bet is that the alternative – allowing the first truly general intelligence to emerge inside infrastructure controlled by a small number of corporations or governments – poses a greater danger.

“I’ve seen this situation coming many decades ago,” he said. “It wasn’t hard a long time ago to see why AI should be open and decentralized and why it should be wired for compassion and should have a motivational system.”