OpenAI safety worker quits company amid AI governance concerns
Daniel Kokotajlo, a safety worker at OpenAI, has resigned from his position due to concerns over the company's approach to artificial general intelligence (AGI). As a philosophy PhD student working in OpenAI's governance team, Kokotajlo expressed his disillusionment in an online forum. He stated that he had lost confidence in the company's ability to "behave responsibly around the time of [artificial general intelligence]."
Growing calls for pause on AGI research
Kokotajlo's resignation comes amid a rising debate over the need to halt research that could lead to the creation of AGI, an AI that can outperform humans. This discussion has been spurred by warnings from experts about the potential risks of an AI surpassing human cognitive abilities. Last year, over 1,100 AI experts, CEOs, and researchers signed an open letter advocating for a six-month moratorium on AI experiments.
Kokotajlo's stance on 'selective pause' in AI research
In his posts on the forum LessWrong, Kokotajlo clarified his position. He stated that most people advocating for a pause are pushing against a 'selective pause,' and for an actual halt that would apply to major labs leading the progress. Kokotajlo argued that such a 'selective pause' would not affect the "big corporations that most need to pause." He concluded by saying this disillusionment was part of why he left OpenAI.
Superalignment team faces resignations
Kokotajlo's resignation follows that of research engineer William Saunders, who left two months prior. Saunders was part of OpenAI's Superalignment team for three years, a group tasked with ensuring that "AI systems much smarter than humans follow human intent," according to OpenAI's website. The company acknowledges the potential dangers of superintelligence, stating that "the vast power of superintelligence could also be very dangerous, and could lead to the disempowerment of humanity or even human extinction."
OpenAI's approach to superintelligent AI control
OpenAI, instead of having a solution for controlling superintelligent AI, hopes that "scientific and technical breakthroughs" could lead to an alignment tool that could keep systems in check. However, not everyone on the Superalignment team seems aligned with the company's approach. This disagreement may have contributed to the firing and rehiring of CEO Sam Altman. Ilya Sutskever, who earlier sat on the original board of OpenAI's non-profit entity, reportedly disagreed with Altman on this topic, before the latter was ousted.
