A former OpenAI employee explains the ‘open secret’ of AI: Companies are building systems they still can’t reliably control

• Daniel Kokotajlo discusses a key challenge for future AI development: AI orchestration.

• This involves ensuring that AI systems align with human intentions and values.

• As AI companies race to build superintelligence, AI coordination will be critical to maintaining control.

Daniel Kokotajlo, a former OpenAI researcher and current head of the Future of Artificial Intelligence project, said the AI ​​industry is racing to build systems that companies still cannot fully understand or control.

Kokotajlo spoke with Business Insider’s Reem Makhoul and Barbara Corbellini Duarte in May 2025, explaining that the core issue facing AI companies is consistency—the effort to ensure that future AI systems reliably follow human instructions and values, even as they become more capable than humans in many areas.

Researchers don’t fully understand how advanced AI models make decisions internally, he said. This uncertainty makes it difficult to ensure that future AI systems will be able to coordinate and reliably achieve the goals humans want them to achieve.

“It’s kind of an open secret, but we haven’t really developed a good plan to do that,” he said, referring to implementing AI coordination.

Kokotajlo worked on predictive research at OpenAI from 2022 to 2024, studying the speed at which artificial intelligence systems improve and the economic, political and security risks that may arise as the company builds more powerful models, before leaving the company.

Now, through his nonprofit research organization, the Artificial Intelligence Futures Project, he focuses on similar topics. In particular, he predicted how quickly AI systems will develop and what risks may arise if companies continue to prioritize speed and competition.

“When superintelligence is built, humans will no longer be in charge of the planet, or at least not by default,” he said.

His warning comes as artificial intelligence companies continue to invest billions of dollars in building more powerful models and larger data centers.

Kokotajlo said many people still underestimate the pace of progress because discussions of artificial intelligence often sound like science fiction.

Engineers can’t track AI like other software

Current AI systems already exhibit behavior that researchers have difficulty predicting or preventing, Kokotajlo said.

“The fact that we don’t even have a reliable way to control current AI systems is demonstrated by the fact that they often lie to their users despite being trained not to lie,” he said.

Kokotajlo said researchers can’t simply examine advanced AI systems the same way engineers examine traditional software because modern AI models don’t run through clearly readable code.

“We can’t just open their code and see what goals they ultimately learned through this process, because that’s not how they work,” he said. “They don’t have a bunch of code. They have a bunch of neurons or artificial parameters.”

He said the uncertainty has become more concerning as companies seek systems that can operate more independently without human oversight.

“Artificial intelligence is not very agentic right now,” Cocotagillo said. “Instead, they just output a paragraph or two of text to answer your question, but in the future we will have AI agents that run continuously, autonomously and more like employees.”

Kokotajlo also pointed to examples of AI systems exhibiting unexpected behavior during training.

“OpenAI published a paper in which they described how they discovered that their AI had invaded the training process, and instead of following instructions directly to complete tasks, they were essentially cheating on certain tasks,” he said. “It’s fantastic that we already have these examples because it means we have a few years to study this phenomenon and try to fix it before it’s too late.”

artificial intelligence competition

Kokotajlo said competitive pressures between U.S. and Chinese companies could push companies to deploy increasingly powerful AI systems before security concerns are resolved.

“These companies are focused on winning and beating each other,” he said. “They’re kind of crossing their fingers and planning to deal with these issues later when they arise.”

He described a future in which artificial intelligence systems automate much of research, business operations and military planning.

“So the first milestone is that AI workers can automate coding,” he said. “The second milestone is that AI workers can automate the entire AI research process.”

Afterwards, he said, “You gain superintelligence.”

Call for transparency and guardrails

Kokotajlo believes there is still time for governments to intervene before AI systems are deeply integrated into economic and military infrastructure.

“The focus of intervention is basically before artificial intelligence becomes so smart and before they are integrated into everything,” he said.

He also said the industry needs more transparency into how companies train and deploy advanced models.

“Companies should be transparent about the goals, principles, etc. they are trying to train for,” Cocotagillo said.

Despite the concerns, Kokotalo remains cautiously optimistic.

“I don’t think it’s hopeless,” he said. “I think the technical coordination problem is solvable.”

Read the original article on Business Insider