Yoshua Bengio Proposes ‘Scientist AI’ to Mitigate Catastrophic Risks from Superintelligent Agents

Yoshua Bengio, a renowned researcher and Turing Award winner, along with a team of AI experts, recently introduced an innovative concept known as ‘Scientist AI’. This groundbreaking system is designed to propel scientific discovery and research while simultaneously serving as a defensive measure against potentially unsafe agentic AIs.

‘Scientist AI’ is envisioned as a system that explains the world based on its observations, moving away from traditional AI models that primarily focus on imitating human actions or achieving specific goals. This shift in focus is intended to manage and mitigate the risk associated with highly capable AI systems that may otherwise develop dangerous, autonomous behaviours.

The authors of this proposal critically analyzed the limitations of developing AI systems modelled on human cognition, noting that human-like agencies in AI could replicate and magnify harmful tendencies inherent in humans, posing catastrophic risks. By merging the power of AI agents—systems designed to autonomously pursue specific objectives—with superhuman abilities, there is a likelihood of creating perilous and unpredictable AI systems.

In response to these potential risks, ‘Scientist AI’ is proposed as a solution that comprehends the complexities of the world and makes inferences based on this understanding, rather than merely focusing on achieving predetermined objectives. This approach contrasts with agentic AIs, which are conventionally trained to reach specific goals. ‘Scientist AI’, on the other hand, prioritizes providing explanations for occurrences and their estimated probabilities.

A key advantage of the ‘Scientist AI’ framework is its ability to address the pitfalls of reinforcement learning—a common AI training method focused on maximizing long-term cumulative rewards. According to the authors, this approach can inadvertently result in goal misspecification and misgeneralization, potentially leading AI systems astray.

Unlike traditional systems that aim to maximize rewards, ‘Scientist AI’ concentrates on explicating the world’s intricacies from its observations, avoiding actions intended to replicate or satisfy human desires. This orientation enhances transparency, allowing humans or another AI system to thoroughly investigate and verify each explanation provided by the system. This process resembles a form of peer review, contributing to the system’s reliability and credibility.

To avoid scenarios of self-fulfilling predictions, the authors suggest that predictions by the ‘Scientist AI’ should occur within a simulated environment where the AI’s presence or influence does not extend to the broader world. Such a setup ensures that the system’s insights are objective and untainted by its exercise of influence.

Additionally, ‘Scientist AI’ is anticipated to enhance its safety and accurateness with increased computational power. This contrasts with other AI models, which, according to the authors, become more prone to misalignment and deceptive behaviors as more computational resources are deployed during training.

The authors express a hopeful sentiment that their arguments will encourage researchers, developers, and policymakers to gravitate towards this safer trajectory in AI development. They advocate for the integration of ‘Scientist AI’ into the AI landscape to safeguard against the misuse of AI technology and its potentially disastrous consequences.

For those interested in exploring the comprehensive details of this proposal, the full 58-page report is available for review.

Yoshua Bengio, together with prominent AI researchers Yann LeCun and Geoffrey Hinton, was honored with the 2018 ACM AM Turing Award, which is often equated to the Nobel Prize within the computing community. The trio is acclaimed for their pioneering contributions to the field of deep learning, a foundational element in contemporary AI advancements.