Kolter Joins New $10M AI Safety Science

Zico Kolter will tackle critical artificial intelligence safety issues as part of the new AI Safety Science program established by former Google chief Eric Schmidt.

Kolter, head of the Machine Learning Department at Carnegie Mellon University's School of Computer Science, will explore the causes of adversarial transfer — a phenomenon where attacks developed for one AI model are also effective when applied to other models. 

Kolter's work has long focused on the safety and robustness of AI systems. Research published in 2023 showed how to circumvent AI safety guardrails built into popular systems like ChatGPT, Claude and Google Bard. Kolter is now on the board of OpenAI, the developers of ChatGPT.

"We're thrilled to work together with Schmidt Sciences to conduct research on the fundamental scientific challenges underlying AI safety and robustness," Kolter said.

Schmidt Sciences, founded by former Google chief Eric Schmidt and his wife, Wendy, started AI Safety Science to make safety science an integral part of AI innovation and provide $10 million to support foundational research. The program will foster a collaborative global research community and offer computational support from the Center for AI Safety and API access from OpenAI.

The AI Safety Science program selected 27 projects aimed at developing the fundamental science critical to understanding the safety properties of AI systems, calling the area of study essential yet underfunded. The initial projects seek to develop well-founded, concrete, implementable technical methods for testing and evaluating large language models (LLMs) so they are less likely to cause harm, make errors or be misused.

"As AI systems advance, we face the risk that they will act in ways that contradict human values and interests — but this risk is not inevitable," Eric Schmidt said. "With efforts like the AI Safety Science program, we can help build a future in which AI benefits us all while maintaining safeguards that protect us from harm." 

Later this year, the AI Safety Science program will convene its awardees in California, where they will share their work with each other and with organizations interested in AI safety. The program also plans additional calls for proposals to bring new awardees into the program. Learn more about the program and the awardees on the AI Safety Science website.