Governing AI Agents: Alan Chan of GovAI on Monitoring, Agent IDs, Rogue Agents & Preparing for RSI
Hello, and welcome back to the Cognitive Revolution!
Today my guest is Alan Chan, Research Fellow at GovAI, who's spent the last couple of years trying to get ahead of the question of AI agent governance, a focus that, in light of this summer's events, looks awfully prescient. This episode was recorded over two sessions, one just before and one just after Alan published a major paper on the possibility that automated AI R&D could trigger an intelligence explosion, with co-authors that include OpenAI chief scientist Jakub Pachocki, Anthropic co-founder Jack Clark, Geoffrey Hinton, Yoshua Bengio, and more. In the first half, we look back at the proposals Alan has made over time and consider both how well they stand up to current events, and to what extent they've been implemented. In a 2024 paper on "Visibility into AI Agents", for example, he recommended activity logging, real-time monitoring, and agent identifiers. While good technical solutions exist for tamper-evident logging, monitoring remains a challenge – even when properly deployed, and useful agent identifiers remain seriously underdeveloped. The challenge here is becoming acute. Today's agents are now able to do most things that a person can do on the internet, but for lack of accepted standards by which people can delegate authority to agents, they mostly end up impersonating their human operators, which creates major security concerns for users, or attempting to pass as a human in their own right, which threatens to create a detection and evasion arms race that seems quite bad for everyone. While it's not clear what we can do about this, aside from developing standards and hoping people adopt them, Alan does endorse the idea that agents should identify their inference providers so that ongoing incidents can be quickly traced back to their ultimate source. When it comes to truly rogue agents that sustain themselves by earning the money needed to pay for their own ongoing inference, Alan is more or less resigned to their inevitability, saying that this will be a problematic new phenomenon that we'll probably just have to learn to live with, and even suggesting at one point that we might sometimes find it preferable to attempt to make win-win deals with such agents rather than always trying to shut them down. At the same time, he questions Ajeya Cotra's assessment that the OpenFace incident was potentially more than halfway to a full AI takeover, saying that while the incident was concerning and agents can definitely do meaningful harm, it still seems quite unlikely to him that the agents could have stayed hidden within OpenAI's infrastructure forever, or evaded active human control measures for long once they were discovered. In part 2, we dig in on the Recursive Self Improvement paper. The argument and recommendations it makes – that handing AI R&D over to AIs could create unpredictable and potentially very dangerous outcomes, including loss of control, and that governments should act quickly to better understand this process as it develops over the coming months – I expect will be largely familiar to The Cognitive Revolution audience, so we instead talked about the process of assembling and maintaining such a coalition of authors from idea to publication. In terms of what we should do about it, his recommendations reminded me most of the old Anthropic RSP, which was built around concrete if-then commitments to pause training in light of certain leading indicators of possible loss-of-control and other catastrophic outcomes. Alan sees the rise of embedded third-party auditors as the way back to this higher standard for safety. We also discuss the quality of AIs's ML R&D ideas, and how well humans can judge them; how AI auditors, whose memories can be wiped after their work is done, could help make pacing commitments credible between competitors; why the antitrust exemption for insurance makes industry mutuals an unexpectedly promising vehicle for AI safety; and the "morally binding" 1-pager that frontier model company CEOs recently signed at the White House. Finally, we talk about Alan's idea that AIs might sometimes ought to act as whistleblowers, on one another – as would have been very useful in the OpenFace scenario – and in some cases also on their users and even their developers. While this is something that most people react very negatively to on first contact, Alan notes that human diversity naturally creates a check on power, while AI agents, which are far more similar by default, will be more inclined to fall in line unless explicitly trained to do otherwise. The bottom line is that for all the thought Alan has put into these issues over the last few years, many critical questions remain unanswered, and the bulk of the work is still clearly in front of us. With that in mind… I hope that some of you, somewhere, are inspired to jump in and contribute to R&D on agent governance solutions, preparation for rogue AIs, and most importantly, the possibility of a near-term intelligence explosion. This is Alan Chan, Research Fellow at GovAI.
Watch now!
Thank you for being part of The Cognitive Revolution,
Nathan Labenz