Full Transcript

Nathan Labenz: (0:00) Hello, and welcome back to the Cognitive Revolution. Today, I'm speaking with Helen Toner, director of strategy and foundational research grants at CSET, the Center for Security and Emerging Technology, and author of a new substack called Rising Tide. Helen is best known to the general public for her role as an OpenAI board member responsible for temporarily firing Sam Altman in late 2023. But she's been feeling the AGI, or at least the need for society to invest in preparation for the possibility of transformative AI, since way back in 2016 when she started working on AI policy full time. That's a full 5 years before joining the OpenAI board in 2021 when, it's worth noting, OpenAI had already launched GPT-3 as an API product, already taken $1,000,000,000 in investment from Microsoft, and was increasingly recognized by those in the know as a leader in the generative AI wave. Certainly by that time, OpenAI had plenty of access to super talented candidates for its board. With that context in mind, and again remembering that her blog is called Rising Tide, despite what you might have heard elsewhere, it probably should not surprise you to learn that Helen is definitely not an AI decel or even especially hawkish on most AI safety issues. On the contrary, she argues in early posts on her substack that nonproliferation is the wrong approach to AI misuse and instead promotes the concept of adaptation buffers, the notion that society broadly has a critical window of opportunity to adapt to new AI capabilities between the time when they're first demonstrated, typically at high cost in terms of both R&D and compute, and when they later become widely accessible, typically at much lower cost, as we've recently seen with companies like DeepSeek dropping the cost of frontier reasoning capabilities. While her focus today is on other things, I couldn't resist asking Helen some OpenAI related questions. And I appreciate her willingness to engage despite having addressed these issues in multiple forums already, including especially an episode of the TED AI show, which we'll link to in the show notes. The only truly new detail that you'll hear in this conversation is her assertion that media reports suggesting that some sort of Q Star breakthrough in reasoning had led to the board's decision were, quote, unquote, totally false. But nevertheless, I think it's important that Helen and other former OpenAI team members continue to speak candidly about their experiences with the company and its leadership. As Helen notes in another of her first blog posts, everyone's timelines are dramatically shorter than they used to be. What passes for long timelines in AI circles today would have been quite short not many years ago. And given this new short timeline's consensus, scenarios like former OpenAI researcher Daniel Kokotajlo and team's recent AI 2027 reflect not just one of the shorter timeline forecasts, but if I'm reading between the lines effectively, a warning about how OpenAI leadership might fail to act responsibly around the time of AGI by abandoning its principle of iterative deployment, keeping the best models for its own internal use plus maybe that of the US government, and aiming for a sort of AI takeoff via the automation of AI research. That's a warning, by the way, that's become a bit more credible this week with the news that OpenAI has indeed announced that GPT 4.5 will be deprecated from the API. All that's enough for me to feel strongly that it's important for Helen to use appearances like this to continue to remind Washington decision makers that OpenAI's CEO was not consistently candid with its board. And also for me to applaud moves like the amicus brief recently filed to the Elon Musk first OpenAI lawsuit by 12 former OpenAI team members who argue that nonprofit promises were central to OpenAI's early hiring success and that the nonprofit should not cede control of the company at any price, a development that happened after I recorded with Helen and on which I hope to do a full episode soon. Of course, the stakes are only rising from here. With OpenAI and other AI companies seeking Pentagon contracts and special legal protections, Helen's latest research out of CSET with Rhodes scholar and former Navy Aegis operator Emilia Probasco on AI for military decision making is super important and a sober attempt to map out how AI systems have been and are likely to be used and how that may diverge from how they actually should be used given their current limitations. Among many other interesting details, I was amazed to learn that some nations, including most prominently Russia, currently have published military doctrines about AI, which seem to be fundamentally out of touch with current AI systems' lack of reliability and total lack of adversarial robustness. This too is something that Washington decision makers probably can't be reminded of often enough as they seek to develop autonomous killer robots. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, read a review on Apple Podcasts or Spotify, or just drop us a comment on YouTube. Of course, we welcome your feedback too as regular listeners will know. While I believe that there's probably some nontrivial and irreducible risk associated with developing advanced AI at all, it's my sense that much of the extreme AI risk we face today, in fact, exists because key decision makers, under intense and growing pressure, seem fairly likely to make some very bad mistakes. If this show can do anything to contribute to a positive future, I hope that it can help people start thinking about those critical but avoidable failure modes sooner and better so that we can minimize the extreme downside risk and get to live in that age of AI provided abundance that we've been promised. If you think I can be doing a better job, I encourage you to reach out either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy this conversation, looking back on OpenAI and looking ahead to adaptation buffers and military use cases, all amidst shorter and shorter timelines to AGI, with Helen Toner from the Center for Security and Emerging Technology and author of the new blog, Rising Tide.

Nathan Labenz: (6:04) Helen Toner, director of strategy and foundational research grants at CSET, the Center for Security and Emerging Technology, and author of a new substack, Rising Tide. Welcome to the cognitive revolution.

Helen Toner: (6:15) Thanks. Great to be here.

Nathan Labenz: (6:17) Yeah. I'm excited for this conversation. We got a lot of ground to cover. I think, you know, we all have crosses to bear in this life, and one of yours is you're gonna go on and do a ton of things in the AI space, and yet people are always gonna come back and ask you questions about your tenure on the board of OpenAI and of course, you know, everybody is at least somewhat familiar with how that ended. So I'm not gonna be an exception to that entirely, but I do wanna make sure we have time for a bunch of different things. To set the stage, one question I actually don't know the answer to at all and I'm really kind of curious about is how did you get involved with OpenAI in the first place? I mean, this goes back years to a time when, like, there was no powerful AI. Most people would, you know, dismiss the notion as fanciful. And, you know, very few people that were taking the whole topic seriously in any real way, but you obviously were. So maybe just kinda share, like, your backstory with respect to AI and, you know, some of the enthusiasm that you must have had to get into that position in the first place.

Helen Toner: (7:14) Yeah. Absolutely. I mean, so I joined the board in 2021, but I had been familiar with the company and with many of the folks working there since they were founded. So they were set up in, you know, San Francisco in kinda 2015, 2016 kind of time frame. And at that point, I was working in San Francisco and that was right around when I was starting to work on AI issues. So it's really interesting reflecting at the time, it felt like kind of being behind the game because this was, you know, 2012, you had AlexNet, the deep learning revolution was really in full swing by 2015, 2016. And so by the time I kind of came around to the view that, okay, this is gonna be a really big deal, there's a lot of work to be done here, I wanna make this, you know, really real focus for my work, AI and policy and national security. Yet at the time, it felt like that was sort of coming late to the party. But it's been fun to see since then kind of a couple more waves, you know, I think around sort of 2018, 2019 or so, wanna say, people started paying a little more attention. And then obviously, ChatGPT in 2022, there was this huge new burst of interest. So looking back, I no longer feel like I was as late to the party as I kind of felt at the time. But, yeah, I think it's also easy to underestimate how weird and how kind of against the grain it was for them to found a company to build AGI at the time. That was really not the kind of thing that you talked about in Polite Society, including in Polite Machine Learning Society. It was really Google DeepMind or sorry, just DeepMind at the time was sort of the only game in town of sort of serious researchers who were talking about AGI. And so I think that was also when I look back on why I was invited to join the board, think certainly part of it was being in the AI policy space, having a few years of experience at that when not many people did, having spent time in China, having, you know, that sort of China expertise and national security expertise, which I think was valuable for the board. I think it was also having taken the idea of AGI and taken their mission seriously for multiple years by the time I joined the board was really unusual. And it's sort of funny to look back on that now in 2025 because obviously AGI is on everyone's lips. OpenAI is such a famous company that the situation looks kind of different. But I think for the community was really small. The set of people who had been thinking about these topics and who had actually informed perspectives was was really small. And so it was super interesting to get to be sort of familiar with the company from the very early stages.

Nathan Labenz: (9:33) Were you always a relatively short timelines person? One of the blog posts that you shared the draft with me of just I think rightly just kind of reminds everybody that, like, even, you know, what people are now passing off as long timelines are actually quite short. Yeah. But where were you, you know, 5 years ago in terms of your expectations?

Helen Toner: (9:54) Yeah. No. I wasn't. I still don't know if I am. The standards have changed so much. And, yeah, hopefully, the post you're talking about that current draft title is long timelines to advance AI have gotten crazy short. And, by the time this comes out, it's published, and hopefully, the substack is launched and everyone who listens to this episode will subscribe. But, yeah, when I got into the I think so when I got into the space, I think I counted as short timelines for the time, which was, as I described in the post, like, this seems it seems plausible. It seems likely enough to be worth preparing for that we build very advanced systems in the next couple decades, you know, in the next in our lifetime. This seems like a potential development. And if it happens, it would need a ton of societal preparation to almost know what is thinking about it. So that would be, you know, worthwhile to spend time on. I think that did describe my view when I got into the space. Nowadays, I don't really identify as having short timelines because that means like expecting super intelligence before, you know, the 2020s are out or something like that. And I feel I feel much more uncertain about that. I still tend to fall back to this view of, look, think this is all likely enough that it warrants quite a lot of thought, quite a lot of preparation, which is different than, you know, I think it's very likely to happen or very likely to happen, you know, in the next 5 years, the next 3 years. So, yeah, I guess the question is, you know, depending on what your standards are for short timelines, maybe yes, maybe no.

Nathan Labenz: (11:07) Yeah. I used to make a very similar argument to people when the whole notion of, like, powerful AI was fanciful and certainly any notion of, like, safety concerns related to AI was doubly fanciful. I used to just say, you know, we have a small number of people that, like, scan space to try to find asteroids so that we don't get taken out like the dinosaurs did, and that seems really good. And this, you know, kinda seems like another thing. And now it definitely feels like there is an asteroid and it's like coming at us. And we don't know if it's good or bad, but it's definitely gonna be both.

Helen Toner: (11:39) Yeah. I mean, I never to me, the sort of underlying, like, internal motivation to kinda work on this space was related to the way that I think if you look across the scope of history, huge new technologies tend to really change what society looks like for better or for worse, often for better and for worse. And so it was really coming to believe in sort of the early early to mid 2010s. Okay. This looks like gonna go through one of these transformations probably in my lifetime and that's gonna be a really huge deal and if I'm interested in trying to, you know, leave the world a better place than when I found it to the extent that I can, then maybe this is an area to go work and in shape. And so to me, it was never like, oh, this is definitely gonna kill us and so I have to get into this space to prevent it from killing us. It was much more this broader argument about seems like pretty likely we're gonna go through this massive transformation. Can I get into a line of work that can help contribute to that going better?

Nathan Labenz: (12:28) That mindset point is really interesting, and I wanna ask you about the sort of prevailing mindsets at OpenAI from your perspective. Before getting into that, I know you've, you know, given a couple different interviews about this and spoken about different aspects of it and probably are tired of it understandably. What kind of governs what you can and can't say at this point? I mean, we've all seen also the, like, you know, non disparagement clauses that were then nullified. And I don't think you ever had any, like, equity in the company. Maybe you did, but I don't think so. Right? So yeah. Like, how much is sort of external constraint on what you can say and how much is sort of just, you know, you deciding, like, how much you really wanna talk about this?

Helen Toner: (13:09) Yeah. It's two big factors at this point or maybe three depending how you count. As a board member, I'm under ongoing confidentiality obligations that don't apply, for example, to former employees. So just in terms of, like, conversations on the board, topics that we're discussing, you know, I wanna respect those obligations, I take them very seriously. And then there's also just ongoing legal processes where I could be, you know, my statements could be compared against each other for consistency and minor discrepancies could cause problems. I might need to, you know, testify under oath or otherwise, you know, say things. And then I think also there's, know, there's just a range of other stuff. So confidentiality conversations that I've had in confidence with people. A lot of this stuff as well, it's like so there's just so many details and so much you would need to go back and explain so much context and bring in like other people who really don't need to be dragged into this and like, the payoff wouldn't even be that great because none of the you know, I gave an interview. The best interview I've been able to give on this, the most detail I've been able to go into was on the TED AI show last year. If folks are interested, they should go listen to that. And, you know, it's not that there's like hidden secrets that are more shocking than what was what was there. There's just a lot more kind of, I wanna say almost like boring detail but that brings in a lot of stuff that's maybe confidential or maybe involves other people who don't need to be dragged in. It's sort so it's sort of like, I don't I don't think the payoff is there. And so certainly if people are like, oh man, there's still this big deep dark secret that Helen still hasn't spat out, that's not the case. Yeah. There are still reasons that I that, you know, that I just talked about that I am not just sharing everything totally publicly. But I think even if I were able to, it wouldn't change the overall picture in any in some kind of dramatic way.

Nathan Labenz: (14:43) Yeah. Context is that which is scarce as we say.

Helen Toner: (14:46) And actually, maybe one this is a good point to share one thing. You know, one example of something that I didn't comment on because I thought it was, I wanted to respect my obligations to the company around confidentiality was this rumor at the time about Q Star contributing to the board's decision, which was totally false. So I didn't haven't really commented on it because I didn't wanna get out ahead of OpenAI's sort of reasoning work and the what has now been released as o1 and o3. The board was aware that that research was underway, but we never we never got some letter about a breakthrough. We didn't make our decision based on a letter from employees, like that whole Reuters story was totally false. So that's one example of something that is now slightly easier to talk about because the underlying sort of confidential information around that line of research is now, you know, now out in the open.

Nathan Labenz: (15:32) Yeah. Okay. So about the mindset or the sort of motivations, mean, you said, you know, wanting to leave the world a better place, you know, as part of what motivated you to get involved and and just kind of recognizing the stakes, feeling like this is sort of a a high leverage activity. That seems to me to be a big part of how I understand what I think is kind of motivating people at OpenAI in general. And, you know, in a way that's like good. Right? I mean, everybody should wanna make a positive difference. But I do sometimes worry that it can cross over into a sort of main character mindset or kind of a hero mentality. And especially as I, like, hear more and more things about sort of guru style, you know, coaching going on and, you know, practice of, like, detachment and, you know, sort of a elite performance mindset, I'm not sure if it's, like, maybe gone too far. And I sort of you know, part of me is like, maybe we should want some amount of attachment among the people that are developing the potential, like, superhuman AI systems. So so much of this is, like, hearsay. You know, I don't even really know how pervasive some of these ideas are, but there are definitely, like, quite a few data points at this point. How would you do you say that's, like, a prevailing sense of the company that there's this, like, heroic, world changing quest that they're on, or would you put that as a more, like, minority position that's just occasionally popping up?

Helen Toner: (17:05) I mean, I think board members are generally not the best position to talk about the culture of the company because they're not kind of immersed in the employees the way that a team member would be. So I honestly don't know that I have perspective on that that you wouldn't know, this perspective I do have comes from knowing people who work there, and I know you know people who work there as well. So I don't feel like I have more to add necessarily.

Nathan Labenz: (17:25) Okay. Fair. How do I get at that? I mean, it seems it does seem like a very important question. Like, I do have this sense that sort of detachment in frontier AI development seems somehow wrong. Like, it feels like we're taking, you know, to put it in machine learning terms, it feels like we're taking something that was learned, like, out of domain. You know, we this sort of mindset that I think I'm seeing is kinda like what they tell NBA 3 point shooters to adopt. Right? Just like, keep shooting and, you know, don't worry, but, you know, trust the process over the outcome and, you know, sort of. And I don't know that that generalizes, you know, super well. Yeah. Mean, the these high stakes technology environments.

Helen Toner: (18:14) Yeah. I mean, I think the other version of it generalizing would be just this separation that we sort of implicitly have in society more broadly between kind of the people building new technologies and doing scientific work and the people who are figuring out how to apply them and how to regulate them and I think it's generally pretty reasonable for someone who's figuring out how to make you know, an airplane wing be shaped slightly more efficiently to not be thinking about like how should the FAA regulate this or how should airline seats be priced or, you know, things like that. I think often it does make sense to kinda decouple those more technical and more sort of societal questions. And so to me that's more that out of distribution is. This might be a technology where the if the technical progress outpaces society's ability to adapt, then the people who have just been doing that sort of decoupled technical work might end up having these huge societal consequences that sort of you know, only they or almost only they were able to affect or to prevent. So to me that's the way that I would think about kind of the OOD element here.

Nathan Labenz: (19:12) Yeah. That's interesting. Hey. We'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (19:19) In business, they say you can have better, cheaper, or faster, but you only get to pick 2. But what if you could have all 3 at the same time? That's exactly what Cohere, Thomson Reuters, and Specialized Bikes have since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. And better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is the cloud built for AI and all of your biggest workloads. Right now, with 0 commitment, try OCI for free. Head to oracle.com/cognitive. That's oracle.com/cognitive.

Nathan Labenz: (20:29) Being an entrepreneur, I can say from personal experience, can be an intimidating and at times lonely experience. There are so many jobs to be done and often nobody to turn to when things go wrong. That's just one of many reasons that founders absolutely must choose their technology platforms carefully. Pick the right one, and the technology can play important roles for you. Pick the wrong one, and you might find yourself fighting fires alone. In the ecommerce space, of course, there's never been a better platform than Shopify. Shopify is the commerce platform behind millions of businesses around the world and 10% of all ecommerce in The United States. From household names like Mattel and Gymshark to brands just getting started. With hundreds of ready to use templates, Shopify helps you build a beautiful online store to match your brand's style, just as if you had your own design studio. With helpful AI tools that write product descriptions, page headlines, and even enhance your product photography, it's like you have your own content team. And with the ability to easily create email and social media campaigns, you can reach your customers wherever they're scrolling or strolling, just as if you had a full marketing department behind you. Best yet, Shopify is your commerce expert with world class expertise in everything from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into cha ching with Shopify on your side. Sign up for your $1 per month trial and start selling today at shopify.com/cognitive. Visit shopify.com/cognitive. Once more, that's shopify.com/cognitive.

Nathan Labenz: (22:25) Another thing that I mean you know, actually, I think last time we actually spoke, I was still participating in the GPT-4 red team. You were still on the board, and, you know, it's been quite a journey since then. I've definitely watched the company, like, super close. And I feel like I've kind of been on this roller coaster where I've been, like, many times sort of disappointed and kind of at times even, like, outright scared by what I'm seeing. And then at other times, I'm like, well, that's, like, dramatically reassuring. And I'd say there's probably 10 episodes, you know, that and it might be, like, 5 to 5. The most recent 2 would be the publication of the obfuscated reward hacking paper, which I would put up there in the, you know, pantheon of, like, most important, you know, and clearly stated warnings about like how AI can go wrong if it's not developed with like, you know, utmost care. And then at the same, like maybe the same week or like within like 72 hours of that, there was the response to the White House request for comment on, like, what AI policy should be. And there, you know, we got a rather, yeah, I would say, escalatory, vibe, you know, certainly with respect to China. We've had you know, Altman has said things like, it's our values or their values. There's no third way, which seems again like ruling out a lot of possibility space, you know, quite prematurely. And then also asking for things like, you know, just cancel all, you know, property rights so we can just train on everything. Because, if we don't do that, you know, that China will. So it seems like there's this, like, almost schizophrenic, nature to the company, and I wonder how you understand that.

Helen Toner: (24:00) Yeah. I find it confusing as well, and I think it has gotten seems like it's gotten more pronounced over the last year or 2. One explanation could just be that there's which I think is as far as I know is the case that there's relatively, you know, a decent amount of freedom given to employees to, you know, tweet as they choose to, you know, do some research directions and maybe write about those research directions. I think there's different processes that different things go through. But again, I, you know, I haven't been close enough to those sort of on the ground decisions about what gets published when to have any kind of insider perspective on that. But I do think I agree with you that it's striking how how different some of the voices from inside the company seem to be, and I wonder if I don't know. I hope I hope that the the more technical folks there are paying attention to the kind of policy messages that are being sent out by the company given how much they they sort of contradict each other.

Nathan Labenz: (24:46) How would you advise people that are there today? I mean, if you are inside and you are and this could generalize beyond OpenAI, I think, you know, I don't see any reason to think that, like, you know, xAI won't have similar issues and potentially, you know, other companies that are generally held in higher esteem in terms of their, you know, safety practices very well could too. Yeah. It's all coming at us pretty fast. So if you are somebody inside a company and you're, like, concerned about what you're seeing, what should those people be thinking about? And maybe, like, you know, flip side of that or a complimentary question is, like, what should policymakers be thinking about in terms of protecting whistleblowers or, you know, facilitating? I think, you know, I also, like, respect the idea that, you know, these companies should be able to keep some trade secrets. For sure. But then it also seems like the level of secrecy that was requested, like, of me at one time was too much where it was like, the public sort of at some point does need to know what capabilities exist. So I don't really have a great sense for how to find that line, but maybe let's start with the policymakers. Like, what do think the rules should be? And then we can go into, like, if you're in a position where maybe the rules aren't there yet or whatever, you know, how should one as an individual think about taking responsibility themselves?

Helen Toner: (26:11) So rules specifically around whistleblowing?

Nathan Labenz: (26:14) Yeah. I mean, you go broader than that if you want, but I'm definitely interested in how do we get things that the public really needs to know Yeah. To come to light when, you know, company policy says it's a secret.

Helen Toner: (26:26) Yeah. Yes. I mean, I think the whistleblowing piece does, like, connect to other parts of the policy picture. I won't try and give a comprehensive view on policy right now, but sort of starting from the whistleblowing piece and expanding out. I mean, way whistleblowing usually works is it's for illegal behavior. And so, you know, the SEC has very clear processes for if you're seeing financial misconduct, you can go talk to them and lots of other whistleblowing processes are similar. So I think for policymakers, a big challenge I want them to have in mind here is a lot of the concerns that we're talking about or potential concerns are behavior that is actually not illegal. And so where is the line for when can you whistleblower? What kind of behavior should be protected? So I think the best way to do whistleblower protections is to pair it with some kind of disclosure or some kind of, you know, rules around what information needs to be shared because then it creates a clear standard for if the company is either not sharing information it's supposed to share or being misleading or or, you know, inaccurate in the information it's sharing. That's kind of, like, structurally a simpler way for the whistleblowing to work as opposed to trying to have this vague standard of like, if you're worried, you know, then call this hotline or something because that's just so squishy and hard both for employees and also for the company as you say, trying to think about trade secrets or other reasons that they don't want their employees just kind of blabbing. It's much more helpful to have kind of clear standard to compare against. So that's one thing I would say on the policy side is if you compare it with some kind of expectations requirements around information sharing of some kind or around processes that you have to carry out internally even if you don't share the results but then that leaves employees able to say, oh actually we didn't carry out that process. So you know, to be a little more concrete, a version of this that I think can work quite well is the kinds of ideas around creating a safety and security plan, potentially publishing that plan or sharing it with the government and then that creates an opening for whistleblowing activity if you're not sticking to that plan basically, which again is just a little crisper than, you can whistle blow if you're if you're worried, if you're concerned, if you think there's too much risk being taken. And the other thing for policymakers I think that they need to keep in mind is that these are technical folks, they're not legally sophisticated, they might be scared, they might not have that much time, they might be working really hard. And so the simpler and clearer the process can be in terms of how do you know if you're eligible, how do you know what your next step is, I think that kind of almost like, you know, UX set of questions matters a lot as well. If the user is a whistleblower then then what is their user experience? For the people in the companies, I mean, sort of very concretely, know, there's other, you know, formal whistleblowers who have gone on the record and so those are definitely people you can reach out to if you're kind of looking for advice. I think more broadly and conceptually, a really important thing to keep in mind for the people who are contributing to AGI companies work, to these frontier companies is I think they are in a very powerful position. You know, I think we've seen multiple times how powerful employees can be and I heard this point recently and I thought it was really smart that they might be in the most powerful position they're going to be in because they're actively working to replace themselves and actively working to sort of hand away their own power. So if you're in one of these companies, I think there might be a temptation to sort of sit tight and wait until things get more serious. And that might be right but I think it's worth thinking about you know, both the case that you might actually be less powerful in the future than you are now if your work is more automated. It seems like in general tech workers power is is going down right now in terms of the labor market and so on. And also there, I think it's perfectly likely that there will not be some sort of clear crisis moment in the future, but instead it might really be kind of boiling frog style, this is worrying, this is worrying, I don't like this, this seems a little bit dishonest, this seems a little bit too risky. And so just being realistic with yourself about, if you're only ever going to do something, if there is some big moment, then being realistic with yourself that that might mean that you you never do something. Maybe that's the right call, but not kinda kidding yourself about that, I guess.

Nathan Labenz: (30:11) If you were organizing a union at one of the frontier developers right now, like, do you have a sense of what your demands would be?

Helen Toner: (30:20) I haven't thought about much. I think it's an interesting line of thought. I think, in general, I think we're starting to get to the point where, you know, there's a whole world around labor organizing, worker power, and I think it really hasn't connected much with either sort of the technical AI world so far or the AI policy world so far. I think that's gonna change and I'm pretty interested to see kind of folks who have more of a background in that space and thinking about kind of how to use this kind of power and how to what kind of leverage is productive, how to represent a broad set of interests. I think we'll see more of that in the coming years, I'm pretty interested to see where it goes.

Nathan Labenz: (30:50) You mentioned the sort of disclosure requirements. I mean, we had briefly a sort of 10 to the 26th threshold where at least you had to sort of say they were doing it and I think a little bit more about sort of what tests you ran and kinda how they came out. My sense is that's now gone.

Helen Toner: (31:07) I think it's unclear. Last I heard it was unclear if it was gone because it had started to go through the official sort of notice and comment process in the government. So I think it might still be at the discretion of the agencies, so specifically commerce. I so I haven't heard that that is definitively dead. Certainly seems like it's on a wobbly footing right now.

Nathan Labenz: (31:26) Yeah. They've announced the intention to remove it at least. If there's really nothing else, right, that we sort of at an actual rule at this point?

Helen Toner: (31:35) There's the EU AI act, which is in the process of putting together their code of practice, which I think involves some transparency around I think for them, it's models over 10 to the 25 and maybe some other criteria, and I think the details of what exactly is going to be what exactly you need to be transparent about is still being hashed out. And there's also a lot of political pressure to water that down right now, so we'll see. But that is at least one other legal process that is underway.

Nathan Labenz: (31:59) Yeah. What do you think is I mean, compute thresholds are one thing. What do you think is most important for the public to know? Like, I tend to focus on just observed behaviors, but I'm also really mindful that, like, all of these things sort of have at least potential for unintended consequences, you know, that with the observed behaviors. So there's sort of the, like, well, we don't look, we don't observe, and so, you know, you know, that can fall down pretty fast.

Helen Toner: (32:26) Yeah. I think there's a lot of things that could be helpful to know more about to share, certainly trade offs in terms of what do you share with the public, what do you share with the government, how confident are you that the government will keep things you wanna keep private private. So I think lots of details to be worked out. I think I tend to think in terms of test results both for capabilities and risks being pretty important to share. So what do we think these systems are capable of? I think also just being transparent about, you know, what kind of processes and protocols are you using to make sure that things are safe? Again, just disclosing, not having the government come in with a checklist and say here's what you have to do, but just saying like, tell us how are thinking about this? And I liked ideas as well from Daniel Kokotajlo and Dean Ball had a, you know, a sort of joint piece on transparency. I thought some of the things they pointed out there like looking at you know, the model spec, what what is your model actually being trained to do? Also makes sense to me to have that kind of thing be shared publicly. Again, not because the government should be saying what it should be but because this is a very fast moving space and the way I think about it is there's this huge information gap between the companies that are developing this stuff and everyone else. And so if you can narrow that information gap a little bit, think that's I think that's decent.

Nathan Labenz: (33:36) Yeah. It seems important to me. And I mean, honestly, there's just a lot of work to be done even in educating people about what is already fully public. You know, one of my kind of mantras is, like, if people understood what is already out there today better, they would probably have a healthier fear of what might be coming down the pipe, you know, in the not too distant future. And so to some extent

Helen Toner: (33:54) Or even not a healthier fear, but a clearer understanding, clearer picture.

Nathan Labenz: (33:58) Yeah. Think a little I think a little fear is healthy personally, honestly. But, you know, mileage may vary on how much that does for different people. I'm you know, I also sometimes describe myself as a adoption accelerationist and hyperscaling poser, meaning, like, I love the tools that I have today and I, you know, absolutely am trying to use them to the maximum. And I also am like, man, you know, there's a there's a huge overhang, you know, for society broadly on what we already have. And we continue to see so much more pulled out of, you know, models of a certain kind of, you know, resource input that I do think we're, you know, potentially starting to get close to the line of where, like, certain thresholds could be crossed in ways that we just, like, can't take back and might ultimately regret. You know, I mean, obviously, the canonical one is, like, you open source a model that is later found, you know, to be able to help people make bioweapons or whatever, and, like, you've just created a new sort of Damocles that hangs over everybody, you know, kind of indefinitely. Yeah. I don't know. It seems like we're do you feel like we're not that close to that? It feels to me like we're fairly close.

Helen Toner: (35:09) I don't know. Yeah. I feel like I'm I also, I was I had 4 and a half months of parental leave over the winter, and so I feel like I'm still I just came back to work a few weeks ago, so I feel like I'm still kind of reorienting around o3, you know, DeepSeek R1, this new ChatGPT image release, Gemini 2.5, haven't had a chance to try yet, but like, you know, there's just so much I feel like my picture is changing all the time. I definitely think we're at the point to me the version of compute thresholds that makes sense is not saying, oh, models over 10 to the 26 are dangerous so we have to restrict them more, but saying, okay, we need some way to target the models that are newest and best and most capable because those are the models where the sort of potential risks of potential unknown unknown risks of like what can they do are highest. And so those models should be subject to a little bit more scrutiny. And so I do think we're at a point where it makes sense to say like, look, seems plausible that the next generation of models could be really concerning in a whole bunch of ways. Totally plausible they won't be, but probably we should be like looking a little more closely than, you know, when, you know, GPT 3 came out and that was just, I think, really unlikely to be anything worrying. And I think it was right that there were kinda no rules in place for releasing that kind of model. So, yeah, that's really the way that I'm thinking about it right now.

Nathan Labenz: (36:20) Hey. We'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (36:25) It is an interesting time for business. Tariff and trade policies are dynamic, supply chains squeezed, and cash flow tighter than ever. If your business can't adapt in real time, you are in a world of hurt. You need total visibility from global shipments to tariff impacts to real time cash flow, and that's NetSuite by Oracle, your AI powered business management suite trusted by over 42,000 businesses. NetSuite is the number 1 cloud ERP for many reasons. It brings accounting, financial management, inventory, and HR all together into 1 suite. That gives you 1 source of truth, giving you visibility and the control you need to make quick decisions. And with real time forecasting, you're peering into the future with actionable data. Plus with AI embedded throughout, you can automate a lot of those everyday tasks, letting your teams stay strategic. NetSuite helps you know what's stuck, what it's costing you, and how to pivot fast. Because in the AI era, there is nothing more important than speed of execution. It's 1 system, giving you full control and the ability to tame the chaos. That is NetSuite by Oracle. If your revenues are at least in the 7 figures, download the free ebook, Navigating Global Trade, 3 Insights for Leaders at netsuite.com/cognitive. That's netsuite.com/cognitive.

Nathan Labenz: (37:50) This may be a good opportunity to talk about your concept of adaptation buffers, which is I really like that phrase and notion. And I think it helps deconfuse or hopefully, I think it will I'm hopeful that it will help deconfuse people about the sort of apparent, you know, sort of hard to reconcile idea that, like, these AI advances that we see are really hard to achieve and, like, cost huge amounts of money. But then we also see that, like, you know, DeepSeek and otherwise, like, things are becoming dramatically cheaper. So maybe Yeah. Set up that dynamic a little bit and talk about this adaptation buffer and how you see the window of time that we have to kind of adjust to capabilities advances.

Helen Toner: (38:30) Yeah. I mean, I think an important underlying thing here is I'm generally a believer that humanity, you know, society at large is very adaptable and has adapted to a lot of things in the past. So a lot of thing you know, it's sort of a cliche that often when new technologies come in, whether it's the printing press or the television or the telephone or whatever, people cry that the sky is falling and this is a terrible thing and it's gonna ruin everything forever and then it doesn't. And so I think the starting point here is thinking that for a lot of different kinds of technology, we actually have a pretty good track record of sort of digesting them, figuring out how to incorporate them into society in a good way, how to have a set of, you know, institutions or barriers or social norms around them that make them sort of positive rather than negative or on balance positive. And I think with AI, think there are a set of questions around AI that seem less that way to me. So I think the whole thing around, are we building something that is, you know, a successor species or, you know, more intelligent than humanity by a long way, those set of questions I think are do seem potentially very different to me. But I get a little worried so on whether this this post, which again, you know, you saw a draft of, hopefully now on Substack live by the time this comes out, A place where I think people sometimes overstate how new this is is when it comes to AI misuse. So this idea you mentioned of, like, are you gonna have some open source model that can help anyone create a bioweapon or can really make it much easier to carry out really sophisticated cyber operations, hack infrastructure, so on. I sometimes see this impulse in kind of AI policy circles of, oh, that's too dangerous. That can't we can't let that happen. Can't let that out. Can't have that technology, you know, be proliferated. It's just like sort of in an absolute sense too dangerous, no go, big problem. And so what you see from that is people talking about wanting to ensure or wanting to work toward non proliferation of these systems, meaning can you prevent them from being distributed in any way? Can you prevent, you know, access from getting beyond a small number of, you know, very controlled people? And, yeah, your question about kind of DeepSeek versus the frontier giant cluster training models gets at why I think that is really problematic, which is we have this weird dual dynamic at the moment in AI development where it is both true that developing the next best model that push frontier being at the cutting edge keeps getting more and more expensive in terms of compute power. Also in terms of you know, number of the amount of expertise you need that you know, you need a really absolutely top team of researchers and engineers. That kind of amount of you know, expenses going up and accessibility less and less accessible. At the same time, every time we reach a certain point on that development curve, every time we reach a new set of capabilities, it's very expensive the first time we build it, but then it gets cheaper and cheaper and cheaper. So, you know, DeepSeek was an illustration of this dynamic where they didn't actually build, you know, a model that pushed the cutting edge and was better than anything we'd seen. What they did was they matched what some US companies had had, depending on how you count, you know, a month or 2 ago if it was the reasoning model alone, more like 6 to 9 months if we're looking at the base model. And so and they had done that at a lower price point. We can fight about what the actual price point was, but I think that's not the point for here, for this, you know, the situation. And the reason this matters is I think if you're taking, if you're trying to have a policy approach that says we're gonna prevent anyone from having access to a certain kind of model, but that model is getting cheaper and cheaper and easier and easier to get your hands on, that means that your kind of policy regime is gonna have to get more and more invasive to prevent people from having access to that. So the comparison I give in the post is we have nuclear non proliferation, it works pretty well, you know, the only about a dozen countries have nuclear weapons. That's pretty good, I think compared to what a lot of people would have expected in, you know, the fifties. But imagine how that could have worked or how that would have not worked if nuclear technology were improving over time, getting much more efficient at the same kind of crazy rate that the AI technology is, such that you needed less and less uranium to build a nuke and you needed it to be less and less enriched. So right now you need quite a lot and you need very very enriched uranium for you know, to actually be able to build a bomb. Imagine if that number is going down over time, at some point, you would need to have the IAEA coming and inspecting, you know, what you're doing in your farmhouse because you have a couple acres of land and, you know, across that couple acres of land, there's enough uranium in the soil that in theory you could build some like very efficient, you know, teeny tiny nuclear bomb. That you know, that's a totally untenable regime. That the nuclear non proliferation we have right now only works because there's a limited amount of physical material that people don't necessarily need for other purposes that needs to be enriched in these very highly specialized facilities and that's really not the case for AI. But I think if we're thinking, instead of thinking purely about how do we prevent people from getting access to this, if we're thinking more about how do we make the most of the time that we have to adapt, to build our societal resilience, to be doing things like scaling up our vaccine production infrastructure or scaling up our outbreak detection infrastructure. So, you know, more wastewater monitoring. How do we have more test kits available in more places around the world so that if someone does use a bioweapon, we can more quickly identify it, more quickly respond to it, similar things for on the hacking side of things. I think that approach of how do we how do we maximize the value that we get out of the time we have as opposed to how do we lock this down and prevent this sort of capital b bad technology from being spread is both gonna be more more productive and also less invasive. I do think it might not be enough. It might be that we're just in a really bad situation if AI progresses incredibly rapidly, but I think it's a much better approach and a much more sort of societally healthy approach.

Nathan Labenz: (43:59) Does that imply a certain pessimism about technical solutions? Like, another one of the blog posts is about the, you know, kind of fundamental challenge of just like getting AI to do what you want it to do at all. You know, I'm old enough to remember the LessWrong led discourse from years past on how these things are gonna just turn us all into paper clips. Of course, that's like was always kind of a character, but, you know, there was a, I think, a felt sense that we have no kind of a genie problem where we have no idea how to communicate our real values and real intent to a system like this and so they're just gonna be like extremely extremely unwieldy. I would say relative to that I've been very pleasantly surprised on the upside that, like, today's models do seem to have a pretty good internalization of human values broadly and sort of a general respect for norms. Although at the same time, like one always has to be situationally aware, we are now seeing many of the problems that, you know, Eliezer and all sort of predicted back in the day of like, once they have values, they also sort of seem to be inclined to try to protect them by, you know, lying to users if that's what's needed or trying to subvert a training process that they understand themselves to be going through. So, you know, this is another one of these roller coaster rides where I feel like, man, it's gone way better than I thought. But also some of the, you know, some of the doomsaying is, like, starting to be proven correct. But I guess I would be optimistic that, like, if we had a sort of adaptation buffer that was maybe a little bit more required or, you know, imposed by authorities as opposed to just kind of trusting the sort of natural sawtooth motion, because it does seem like DeepSeek might be about to challenge that or at least those sawtooth time intervals might be getting really short. Like, yeah, I would, you know, definitely love to see all the great things of wastewater monitoring and, know, I mean, the fact that we haven't done anything really about it. The last pandemic is not bode well.

Helen Toner: (45:57) Indeed.

Nathan Labenz: (45:57) But I also kind of feel like we need that time to maybe sort of figure out, like, how does one distribute a frontier model with a better sense of, like, what capability you know, making unlearning work or making, like, mixture of experts work in such a way where you can, like, distribute all but 2 of the experts or something, you know, so that certain capabilities are, like, redacted while the core utility of the overall thing can be diffused. Long question. I guess the core of it is, do you think we'll see technical solutions that could allow us to sort of square the circle and have, you know, free distribution, but also, you know, pretty confident sense that, like, what we're distributing isn't gonna come back to bite us?

Helen Toner: (46:38) I think the thing I wanna challenge is the focus on only technical solutions. Cause this does come up a lot. It's like people talk about the offense defense balance of, like, you know, AI for cybersecurity, for example. Does it help hackers more or defenders more? Same for bio. People talk about, does it help you, you know, design a new vaccine as much as it helps you design a bioweapon? And I think that's I think that's just one small part of the picture. I think the thing that I'm trying to point to with the idea of an adaptation buffer is that a lot of the ways that we specifically, we're talking about these misuse risks. So are there gonna be terrorists who build a bioweapon? Are there gonna be hackers in their basements who can suddenly bring down The US power grid? A lot of the things we can do about that don't actually relate to AI. They relate to what is going on in society. Or if they do relate to AI, it's not actually the frontier model. So it's like, can you, you know, can you use AI tools to look at, you know, large scale disease monitoring data, for example, and notice anomalies or something like For bioweapons, it's like, you know, if you talk to people who who work in biosecurity and bioterrorism, there's a lot of stuff that has nothing to do with AI, nothing to do even with biomaterials. It's things like who is the FBI tracking? You know, what is how good are they at detecting plots before they get very far? So, yeah, I think certainly as the technology advances, there will be sort of new defensive tools that become available to us and we should make use of those. But I sometimes think that the discussion here gets too focused on only those as opposed to looking at kind of broader parts of the picture. And, you know, if we are also if we're talking about AI tools, I think a huge part of the discussion needs to be about the application and dissemination of those tools. So for example, in cyber, I think it's sort of less a question of what can the absolute most advanced model do for cyber defense and more a question of how can you have well designed, ready to ship, you know, defensive tools that you can get in the hands of operators who are not very sophisticated on AI. So you're people who are running your your water treatment plants and your power grid and your chemical plants and so on. How do you have those even if they are AI related, it's not a kind of capabilities question. It's more of a dissemination application question for how you get those defenses kind of out into the real world.

Nathan Labenz: (48:41) I just talked to somebody not long ago who is applying language models to the challenge of formal verification of software, and it sounds like there is a multiple order of magnitude speed up that is becoming possible in that domain, and it does feel like if we just have enough of an adaptation buffer, then a lot of the things that people are most worried about could really be brought dramatically down in terms of the, you know, the absolute magnitude of the risk. Yeah. And so, yeah, it's just a question I think of, like, are we investing enough in that? Almost surely not. And, you know, do we have enough time before the sort of, you know, next disruptive thing hits?

Helen Toner: (49:23) For sure. And I mean, to not to sound too optimistic here. I mean, I do think sometimes I hear from folks, you know, in the cyber domain that, well, it's fine because AI is gonna help defenders more than attackers, so so it'll all be good. And I think that is also, you know, in my mind, a quite naive take if you're looking at the sort of dissemination on real world use case here where I think even if that is ultimately the case long term, for example, if you can, you know, use AI to develop formally verified code, you're still likely gonna go through this sort of dangerous transition period where it's just obviously going to be much quicker. The time lag between some model is released and some disaffected teenager in the basement can use it to carry out an attack is gonna be much shorter than the time lag between the model release and when yeah. The I don't know how many it is. Thousands of critical infrastructure providers in The US can go through their very old code bases where they have this complicated division between their IT and their OT operational technology what gets updated when. You know, it's they're not gonna be nimble. They're not gonna be agile. They're not gonna have all these defenses kind of wrapped up or built in really quickly. So I don't think the fact that these advances seem promising and could help defenders means that we'll get away from that sort of dangerous transition period. But I do think that thinking in terms of a transition period rather than, like, a permanent state of danger helps us respond much better.

Nathan Labenz: (50:42) So have you seen any regulatory proposals that you'd like? I mean, you mentioned Dean Ball, friend of the show. He recently put out a post. I thought it was quite interesting, basically proposing a regulatory market type structure where the government would essentially accredit, you know, or sort of authorize private regulators to sort of, let's say, approve the practices of AI developers. And as long as the developers were able to keep the private regulator happy, then they would get some sort of liability shield. And, you know, then, of course, that could be, like, withdrawn if they didn't do it and even, you know, the regulators, the private regulators, you know, authorization could be withdrawn if, you know, if the state found them to be out of compliance. I understand now, I haven't read the text, but I understand there is now a California bill that is kind of moving in that general direction. Interested in your takes on that or, you know, there's also, like, proposals to really embrace liability and go the other way and say, like, you know, maybe you should even be liable for, like, close calls, you know, because the close calls could be, like, so big and bad that, you know, even probabilistically if you were, you know, if it was a near miss, you know, maybe you should sort of face liability, consequences for that. So I don't know. React to those or, you know, tell me any other policy proposals that you think are particularly promising.

Helen Toner: (52:05) Yeah. So I haven't seen a policy, a sort of like proposed regulatory regime to really sort of comprehensively manage the risk that we're facing from frontier AI systems. I haven't seen one that I like. I think it's a really big problem, it's a complicated problem and it's especially difficult because we're not actually sure exactly what the problem is or when we'll face it. So to me, the policies that I have seen that I'm interested in are more kind of building blocks that put us in a better position for the future as opposed to sort of solutions, which I think is in some ways appropriate given that there's so much uncertainty about the technology, though also certainly scary given that one way the future might go is we might have very little time in which case maybe some initial building blocks right now are gonna be far from sufficient. So I think the proposal by Dean is an example of a building block that seems potentially pretty useful, seems certainly better than nothing. I think it's, you know, he's written it deliberately to be something that could be implemented at the state level which certainly seems far far far more feasible than any kind of federal legislation. I don't know that it does as much as we would need to target those kind of cutthroat financial incentives and also just like other many incentives for the frontier developers to do anything other than push ahead as fast as they can. But I, you know, I think it's an interesting idea and it does seem better than nothing. And then other other kinds of building blocks, we talked about transparency. I do think that that is the kind of thing that can doesn't in itself solve any problems but does put many actors in a much better position to help solve problems as they rise down the road. Likewise, I think there's a lot of stuff that can be done that isn't like sort of regulatory, isn't obliging anyone to do anything but things like funding, trying to trying to really boost our science of measuring AI, I think could be a target of research funding, potentially something for like a, you know, focused research organization or things like that. Similarly for interpretability of course and alignment research and lots of things like that. I think even just even like building blocks as basic as trying to get more technical capacity into governments so that they're able to kind of handle things as they arise and make better decisions as the technology progresses. These are all kind of, again, individual components that don't add up to a comprehensive solution, but that I do I think put us on better footing for the future. So that's the terms that I'm thinking of right now.

Nathan Labenz: (54:22) Just as an aside, it was Gabe Weil. I had to look up and make sure I had his name right who's arguing for this sort of embrace of liability, and I hope to do an episode with him about that. But in the meantime, he's written about it for folks who wanna go into that in more depth. You know, one thing that I thought OpenAI always had right was the idea of iterative deployment. You know, the idea that, like, if we sort of develop super intelligence in secret and then, you know, drop it on the world one day, that's gonna be, like, far more disruptive than if we sort of, you know, launch a bunch of products along the way and people can see what they're good at and get used to them and so on and so forth. That itself now seems to be sort of at risk both in the sense that like Ilya has gone off and started a company that has an explicit strategic statement that they are not gonna release anything until they achieve super intelligence, which seems crazy, but also, you know, to borrow a term like strikingly plausible that they might actually achieve it. And even at OpenAI, I was really, you know, kind of taken aback by Sam's recent statement when they released 4.5 that, you know, let us know if you like this or not because, you know, we've got a lot of other models to build and, you know, this one is pretty compute intensive. And so if it's not like really doing it for people, then we might like take it offline and, you know, focus our resources on building more models. This has me thinking like, you know, we might actually be at sort of risk of like these companies kind of closing down what they put out into the public and just kind of going for broke totally internally. Miles Brundage is also, you know, formerly of OpenAI has made some, I think, kind of cryptic comments about sort of the rising importance of internal deployment decisions. So I'm sure you'll find major flaws with this, but one idea that I've had is like, could we put a sort of speed limit in place, Not an absolute speed limit necessarily, but a relative speed limit where we might say, you can only develop a model that is so many times bigger in terms of resource inputs than the biggest one you currently have deployed. And if you wanna go bigger than that in your development, you gotta deploy something that's kind of, you know, helping us as the rest of society, like, understand where all this is going so that we don't have these, you know, sort of not exactly unilateral because we know that we've got multiple voices inside the companies, but, you know, these, like, very small kind of concentrated decision makers, without much at all in the way of visibility just going for something that they seem to believe could be a world takeover capable technology. So I guess how big of a problem do you see that possible retreat from iterative deployment being? And, you know, do you like my relative speed limit solution or have any others to address that?

Helen Toner: (57:13) Yeah. I agree with you. I think iterative deployment in general seems like a good approach with of course the caveat of at some point you should probably have some criteria in place for when you would not just iterate because you know, the idea of iterative deployment I think is to kinda put it out in the world, see what happens, adjust, try again. I think that's great as long as you're confident enough that what you're putting out in the world is not gonna have any, you know, really severe irreversible consequences. So the question is how do you how do you know when do you decide to do it differently? I'm not sure that I see a retreat from that. I think it I don't know that it's ever been like a certainly was or has been OpenAI's MO, but I don't know that it's ever been kind of an across the industry approach. Your idea, I think it's I think it's interesting. I've heard kind of similar proposals could also be things that companies adopt internally in terms of, you know, how much scale up are you going forward at a given time. It sort of feels to me like it runs into the same same problem that so many of these run into, which is one, how are gonna implement it? Like, to do that in The US, you would certainly need legislation. I think I don't see how you do it without legislation. We're not gonna get legislation. So then how do you do it? And then also, doesn't it just mean that we lose to China? It's gonna be the other, you know, the other big question. So I think conceptually things like that could work maybe or could make sense if they were implementable. I don't really see how they're implementable. And then if they were, then I would wanna come back to this question of like, okay, is this actually the right approach? For me, I tend to be pretty pessimistic about sort of solutions that involve slowing down at some kind of the input level, meaning slowing down, pausing for a certain number of months or slowing down how much, you know, how quickly you're advancing versus slowdowns that are sort of conditional, more of this kind of like if then approach of, you know, we're not gonna keep progressing until we have hit this level of understanding of our system or this level of risk mitigation, which I think a lot of the companies now have in place or, you know, have made voluntary statements that they will they will think in that way. So that would tend to be my preferred approach, I think this kind of thing I think we're in a rough situation right now for anything that will involve kind of cross industry coordination because there's so little political appetite at this moment. Maybe that'll change. Probably that'll change. But, for now, it seems hard to imagine.

Nathan Labenz: (59:20) Do you think at all at the end of the day is about the fact that the policymakers, the members of congress, whatever, just, like, just don't buy it? I mean, it seems like if they really believe what Ilya is saying that he's gonna not release anything until he has super intelligence and he thinks that's gonna happen in the not too distant future, then they wouldn't just sit back and be like, well, let us know when you got the super intelligence. Right? Like Yeah. It seems to me that they just fundamentally don't believe it, and that's, like, the biggest barrier to something. You know? And Yeah. There's a lot of questions, obviously, on, like, what that something should be, but I find the notion of, like, well, we're not gonna get legislation. To me, that still feels like a sort of education challenge. Like, again, if I think if people had a better sense of what already is deployed, you know, they might be like, yeah. I don't know that I'm comfortable with Ilya and, like, how many people how many people even work there? You know? It's like, are we talking, like, a couple dozen potentially, maybe up to, you know, low hundreds now? It can't be that big. And then we're just gonna, you know, sort of wait for them to pop up with superintelligence? Like, that seems so crazy.

Helen Toner: (1:00:25) I agree with you. I think it is in large part a question of how seriously do people take the possibility that this that AI will get as good as something like Ilya thinks it will. But I also think that US congress is really broken right now

Nathan Labenz: (1:00:38) Got it.

Helen Toner: (1:00:39) In an AI way, but just like incredibly dysfunctional. A friend of mine who really knows his way around congress has worked on Hill for years and years said that he thinks that the US house is house representatives is less functional than it's been since after the civil war. So like really really really dysfunctional separate from AI. So I agree with you. I think there will be most likely windows that open again if the technology keeps progressing and I think people will change their views and change their level of urgency around it. And also, I don't know that that will be enough to actually get thoughtful productive regulation through congress at the federal level. It might be enough to get some kind of bill. I certainly know people who are working on, you know, sort of like a Patriot Act for AI where the Patriot Act was put in place after 09/11, but it really been developed in advance. And sort of setting aside the merits of that bill, I think that approach makes sense of saying what you're gonna get some window after some crisis. I think that is a reasonable way to be thinking about AI policy right now. I think it'll be a big lift even in the wake of a crisis to get the right kind of productive useful legislation through not just kind of fighting the last war or doing a bunch of stuff that people wanted to do for other reasons. I mean, the other model for this, in a past era when it seemed like we might actually get something through congress, the other model for this that I have thought about is sort of the trying to avoid a Three Mile Island situation where Three Mile Island as a nuclear disaster seems to have basically killed The US nuclear industry because the safety regulation that was put in place afterwards was just too onerous and wasn't comparing to risk posed by other sources of energy and wasn't actually kind of commensurate to the level of risk and instead it just sort of shut down the whole industry. And I think that story in my mind should be motivation for AI developers to want to have some more safety guardrails in place earlier to prevent that kind of accident or to mean that if that kind of accident happens, you have a better answer or you know, legislators have a better answer for the public of here's all the stuff we did in advance, and this really was just a freak accident. I think right now we're on track for a massive knee jerk overreaction when something something happens, but it we may have passed the point where we can prevent that at this at this point given how unlikely regulation looks. I don't know. Maybe the states will exceed my expectations. Maybe they'll maybe they'll be kind of more useful stuff that comes up there.

Nathan Labenz: (1:02:49) Yeah. I say something similar to, like, AI developers and investors,

Helen Toner: (1:02:53) not

Nathan Labenz: (1:02:54) even so much at the frontier level, but even just, like, your rank and file app developers all the time. Like, you we are right now, the voice AI world is totally taking off. And also, I think is if they're not, you know, pretty quick to sharpen up how they handle the technology, we're headed for a world where there are gonna be some high profile things and because the voices are getting really good, and you can still go to all these products and just drop in whatever voice you want, click the checkbox, and the next thing you know, you're calling, you know, as Trump or as Taylor Swift. I did it with Biden, you know, during the election as well. Just call anyone, say anything. There are 0 guardrails on these products. And, you know, that's not gonna be good for the industry if it, you know, they're like definitely, again, playing with a certain kind of fire that I think self interest alone would dictate better governance or stewardship of such powerful technology. But that seems to fall on deaf ears. And one thing I think is maybe gets a little bit glossed over in sort of the kind of more detailed or like wonky discussions of this, people who are thinking really hard about the risks, thinking really hard about the policies, is I think there's it's quite unpredictable or quite maybe unintuitive which kinds of things will catch the public imagination or will create sort of a perceived crisis. So, you know, I think seems to me like in the after the GPT 4 release, one of the things that really caught fire was this conversation that Kevin Roose had with Sydney and, you know, was trying to get him to leave his wife or whatever and, you know, the AI people I know who read that transcript were like, look, he really led it there, like he was really kind of prompting the model in a way that got it to go there. It's not that surprising. It wasn't dangerous. Didn't actually harm his marriage at all. Probably was a huge boost his career, know. But that is what really kind of caught attention and got people worried and likewise, you know, I don't personally feel that worried about the risks from voice synthesis. We could, you know, talk about that maybe. But I agree with you that it's the kind of thing that is very vivid and very easy for people to latch onto, and so it could be the kind of thing that produces a backlash sort of disproportionate to the actual risk or harm of that specific use case.

Nathan Labenz: (1:04:57) Well, in the interest of time, let's keep moving. You alluded to maybe the one thing that can unite the congress, and that is the threat of China and, you know, certainly a growing number of my AI conversations sort of get backstopped or kind of run into this, like, final barrier of like, well, you know, but China, you know, we'll lose to China. One thing I think is very much under discussed, I'd love to hear your take on is what is the threat from China? You know, like, I don't get great answers to this usually, and I sometimes joke like, am I supposed to expect that my grandkids are gonna be speaking Chinese, you know, if we don't, like, develop AI as fast as we can? How do you understand the threat from China to The United States, the West, like, my values, my way of life?

Helen Toner: (1:05:43) Yeah. I've heard some of the conversations you've had about this and somebody that jumped out to me there coming from the world. Come from this sort national security, foreign affairs, geopolitical kind of viewpoint is you seem to be starting from a point of view of like, well, The US and China should be friends unless there's some strong reason otherwise. And I think for a lot of people with experience in international relations, defense, military history, the starting point is more, okay, we're an established power. China is a rising power. Who has power on the world stage matters a lot. And by default, if they are coming in to be rivaling us in terms of how much power they have and how much they can throw their weight around on the world stage, by default, it's going to be a more hostile relationship. And maybe you can have exceptions to that. So, know, the the classic exception that from last century was as The US was rising and kind of eclipsing Great Britain as, you know, the British empire was crumbling right around when The US was really coming into the height of its power, that was a relationship that was actually very close, actually very cooperative, and so there wasn't a huge amount of tension there. But that's really unusual. I don't know how to give a I don't wanna, you know, go off on a 20 minute tangent about sort of US China history and the specifics. I think here, there was a real effort in the eighties, nineties, aughts to try and usher China into a position in what was seen as sort of the rules based international order, which was, you know, I guess the background here is usually for much of history in many places around the world, most things operated under a might makes right framework. So whoever has the most power, whoever has the most guns gets to push around everyone else. The second half of the twentieth century was a big exception to that where it gets called the Pax Americana or other things where The US was sort of the leading power in the world. There was also The USSR for a good chunk of that. But The US was instrumental in setting up this set of institutions, this way of countries relating to each other. There's a, you know, UN charter which puts sovereignty central. It makes sovereignty of countries central so that you can't just invade other countries and do whatever you want instead. You know, borders are sacrosanct and so on. This whole system that was put in place in the second half of the twentieth century with The US leading, which was seen as a big improvement on the might makes right sort of default. And I think we're in a weird place right now where if you look back at the last few decades of US China relations, I think a lot of the hostility now comes from the failed attempt to bring China into that order. So this is, you know, of course, there was, you know, warming throughout the eighties when Deng Xiaoping was doing his reform and opening strategy to try and, you know, make China more market focused and freer. Big hit to that in 1989 with Tiananmen Square, then sort of more optimism again in the 1990s culminating in China entering the World Trade Organization. The classic phrase, I forget who used it first, was trying to make China a responsible stakeholder in this system. And then that all kind of fell apart in you can date it different ways. Certainly by the time Xi came into power in 2012, that was starting to crumble and Xi has accelerated that where China's become more illiberal again, has become more hostile in the South China Sea, more aggressive, really making it clear that they want no part in this sort of quote unquote rules based international order. And so that is the context for, I think, why they are seen in a more hostile light. The challenge now is that The US itself is retreating from that rules based international order and moving back into a might makes right kind of frame where the idea is while The US has all this power, we have, you know, the dollar as the world's reserve currency. We have the biggest best military, so we should be able to get what we want. So sort of in that light, it comes becomes confusing again, like why would we have a hostile relationship with China? But I think that is very much a kind of set of changes that are still in process and that the system hasn't quite figured out how to orient towards yet. Yeah. And I mean, I think it definitely comes down a lot to I guess there's sort of 2 big questions. One is China's position in international institutions and its relationship to sort of the whole world, and then there's also these sort of territorial military questions around is China gonna try and take Taiwan? Is China going to be aggressive around Japanese, Filipino, Korean assets? Sort of more of a like hard power military set of questions as well. Are they going to, you know, for example, come back to the sort of rules based international order? Are they gonna damage the freedom of navigation norms that The US has been so instrumental in preserving that are so good for international trade? For example, are they gonna prevent people from using their what they claim to be their waters or, you know, there's a whole kind of set of questions around who has power, what are the rules, what are the norms, where China is very clearly not wanting to cooperate with The US on that. And so The US has shifted since 2016, 2017 into a more confrontational posture.

Nathan Labenz: (1:10:36) I have to say though that, you know, all of that stuff, like, I'm generally familiar with that history. Plenty of things we can, you know, complain about China doing. You didn't even mention stealing all of our intellectual property, you know, which is definitely a rightful point to which, you know, many American business leaders and others object. So, you know, there's wrongdoing inside the Chinese nation as well that we can point to and, I think, again, like, justifiably and rightfully criticize. But I still don't quite get the flip, and I'm not sure if I should be understanding what is happening now is just, like, strategic communication where everybody's, like, trying to influence, you know, the guy that I call he who must always be named. But, like, both Altman and Dario have done a pretty dramatic flip, you know, where there's video evidence of them not that long ago saying everybody, you know, is too worried about China, like a race to China would be one of the worst things. You know, we should make our own decisions about what's right to do and not worry so much about them. And we can, you know, point to these clips. And now we've got both of them basically saying, you know, we gotta go as fast as we can or we're gonna lose. And Dario's even like saying, you know, we can't accept a multipolar world. We need to maintain a unipolar world. And I sort of am like, man, who's really being aggressive here? You know, I haven't heard China say they wanna be at the center of a unipolar world. I've only heard Americans say we want to be at the center of a unipolar world.

Helen Toner: (1:12:04) Depends who you read and how you read it. The Chinese I don't know. But, I mean, I agree with you. I think

Nathan Labenz: (1:12:10) their technology. Right? Like, they're not we're the ones that are saying, like, we need to box them out and have this unassailable lead. Meanwhile, they're just, like, open sourcing everything. It doesn't seem like they're, like, trying to dominate us as far as I can tell.

Helen Toner: (1:12:22) I mean, the open sourcing, I think, makes a lot of sense if you're the in the poll if you're, you know, the following position. I think it makes a lot of sense to be trying to show off how good you are, trying to attract talent, trying to so I don't know that that is, like, just coming from pure goodwill and lack of competitive spirit. I think that is more that makes a lot of sense if you're not leading and it's much less clear how to handle openness if you are leading. I agree with you that the position change and the rhetoric change from a lot of the top CEOs has been pretty striking and I think, honestly, I think it's just the path of least resistance at this point. There's so many different issues to handle here with AI and so little agreement on what to do about it and the one thing that people can agree on is, well, gotta beat China. And so it doesn't surprise me that the companies are leaning into that message as a way to say, look, we should get funding, should get government contracts, we should have no regulation, we should get shielded from liability. Yeah. I that's the explanation that makes most sense to me right now. And I'm sure it also depends on, you know, different individuals, how they're thinking about those specific statements.

Nathan Labenz: (1:13:22) You've got a paper coming out on decision support systems in the military. And I think this is really interesting as a, you know, okay. Yeah. We gotta beat China, whatever, but, like, we also have to confront the fact that the systems that we have have, you know, for all the upside, which, you know, I'm well on the record embracing that and use it every day, yada yada yada, also have a lot of problems in terms of, you know, their reliability, their hallucinations. Again, like now, scheming against their human users in some cases. And, you know, I for one would not wanna go into combat with an AI buddy until I was, like, quite confident that all of these sort of scheming issues were, like, well and fully resolved. So

Helen Toner: (1:14:04) Not to mention, confabulation and reliability and

Nathan Labenz: (1:14:06) Yeah. I mean, there's a lot of issues, right, that I like, if I'm taking this thing and trying to really rely on it in a genuinely life and death situation, you know, and I say this as a, you know, top tier enthusiast, I would say, don't I wouldn't wanna use it in that sort of context. Yeah. So that also seems to be, like, a big disconnect to me in terms of how the AI debate, you know, sort of seems kind of especially with respect to China, it seems like a little bit decoupled from the actual reality of the systems that we have. So I'll shut up, give you the floor, and just, you know, tell us about your work on decision support systems and, you know, what we can and probably shouldn't be relying on them for.

Helen Toner: (1:14:47) Yeah. So this paper this paper is led by a colleague of mine called Emmy Probasco, and she is a super interesting person to be working on this because she actually served in the navy and her job in the navy was operating Aegis missile defense systems onboard US Navy ships. So this is actually people really don't a lot of people don't know kind of that we actually have essentially autonomous weapon systems already. So Aegis is a system on board a ship that looks at incoming missile fire and automatically identifies and then takes out incoming threats. Not based on deep learning, developed in like sixties, seventies, employed in the eighties. So yeah, super cool to get to work with Emmy on this paper because she's just bringing such a grounded informed perspective. The paper is about, yeah, what gets called decision support systems and what I think is really important here is to move that discussions about AI in the military don't just get stuck on this autonomous weapons question because there's so many things you can use AI for in the military. And so this category, decision support, is another is a big broad category of, you know, other uses. It can mean a lot of different things. But basically, you know, it's what it sounds like. It's systems that are helping commanders or operators make decisions. And so there's a long history of different kinds of tools like this. And, recently, there's been interest in how do you kind of add AI or how do you use AI to perform some of those decision support functions or upgrade existing decision support systems. Big range of different types of things we could be talking about here. So on the simple end, could be things as simple as you look at a photograph of an area and you use AI to determine the view sheds in that photograph. So what a view shed is is basically if you're a sniper, where can you see? What, you know, where is in your field of vision and where is not in your field of vision for example or if you're, you know, not necessarily a sniper, anyone. And so that could be a computer, you know, vision sort of segmentation image that is doing some kind of processing of an image to figure out what is visible from where. That counts as, you know, in our definition decision support. Likewise, you could have a system doing medical triage. So you have you're on a battlefield. You have a bunch of wounded. How do you figure out who to treat first? Where to take them? That kind of thing. Likewise, student support or looking for looking at movements of ships and doing kind of anomaly detection. You could be using, you know, anomaly detection has gotten way better over the last 10, 15 years. You could be using kind of upgraded AI systems to do that kind of thing. And in the paper, we basically sort of look at a whole range of different sort of AI based decision support systems that are being either used or advertised. So on the that's on the sort of simple end, the ones I talked about so far. On the more complex end, you have companies like Palantir, also Scale AI, advertising, large language model based systems that are really trying to be more of what you described of this kind of all purpose battle buddy, or at least that's how some of the early marketing looked. They've changed their marketing since then to look a little bit more restricted. But this could be something that some of the early videos would have a huge range of potential functionalities kind of in these demo videos. They might include things that to me like make a good amount of sense like you're using a natural language interface to access some like clearly documented information elsewhere. So for example, you've identified some, you know, enemy movement and you wanna know where's your nearest unit, Like, geographically, where is it? So you could ask that, and then it could refer to some database or, you know, some other system and show you, okay. Here's the nearest enemy unit. Then you could say, okay. And how many, you know, missiles of such and such type do they have? You get access to that information. In theory, that all makes good sense to me. But these demos are mixing that in with things like, okay, now generate 3 courses of action that are non escalatory and having the AI, presumably the LLM, just like write out potential courses of action for how you could engage this enemy with what kinds of, you know, tactics from what kind with what kinds of routes with this caveat of non escalatory of like how the hell does the LLM know what's escalatory and non escalatory? Like what who who is making these decisions? How is that being evaluated? And these kind of different use cases were all kind of being mixed together in these demo videos and so part of why we wrote this paper was to say, look, this is a category of use for AI that makes a lot of sense. There's a lot of ways that it can make help the military work better, help clarify, you know, get rid of fog of war, reduce fog of war, but these systems are not perfect and there are a lot of ways you could use them that could go badly for you and so how to think about that. And so, you know, just briefly in the paper, we talk about kind of 3 types of considerations that we suggest be considered if you're thinking about whether to use a system based. So the first is scope. What is the what is the scope of this system? How tightly bound is it? How well can it be tested for that particular set of activities versus is it more sprawling? Is it more general purpose? Is it more kind of whatever happens to come to mind that you might wanna type to your battle buddy? Second is data. So what data has it been trained on? How confident are we that that data reflects the situation that you're in? A huge problem for, you know, military operations in general is both that you're likely to be in sort of situations that are novel and also that you have an adversary trying to mess with you. So how do you think about whether the data that a system has been trained on are gonna be really reflective of the real world situation you find yourself in? And then the third kind of factor that we talk about which I think is really really important and really often neglected is this kind of human machine interaction component. So how is a system designed to help the person operating it, understand what it can do, what it can't do, help them make decisions that are good decisions, help them not over trust, and there's a long history of kind of user interface I think being undervalued in military circles and then contributing to unwanted outcomes as well. So, yeah, essentially we're trying to lay out this sort of category of systems and describe both why militaries want to employ them and how they can employ them productively rather than counterproductively.

Nathan Labenz: (1:20:34) Do you see any stable equilibrium in the future here? I mean, it I was struck by I'm sure you read Dan Hendrycks and Alex from Scale was one of the coauthors, Eric Schmidt, the main thesis. I sad to say, like, didn't find that super compelling as an idea of, like, what a stable equilibrium could look like. Although I really applaud the idea of trying to articulate something that could be a stable equilibrium, but it just feels like, you know, where we sort of inevitably end up, especially if we don't, like, get on the same page with, you know, the countries that we're currently most fearful of, is what I'm starting to call alpha go for the army, which is just like self play, you know, simulation driven kind of go to superhuman performance by just like having these things battle it out amongst themselves and hill climb to a level that human tacticians, especially when you consider speed, just can't get to. And, you know, that's, like, one way we get to Skynet, and that just seems like a pretty bad situation where we have these sort of inscrutable, but, like, hyperlethal systems that we build in theory to defeat an adversary, and maybe it even happens that way. But, boy, that seems like another way that we add a pretty scary sort of Damocles to, you know, all of future humanity's life and they you know, without giving them any chance to vote on it, obviously. Is there any way to avoid that, though? I mean, right now, it just seems like we're sliding into that. And Yeah. I don't love it.

Helen Toner: (1:22:11) Yeah. I mean, many thoughts here. The MAD paper was really interesting basically for folks who didn't read it. They had a few different things in there but the key idea, this main idea, I think was like mutually assured AI malfunction or something like that. The idea was which I think is I think there's a correct core to this which is if one country is going around saying, hey, we're gonna develop superintelligence and then we're going to rule the world, the universe forever, that's gonna create an incentive for other countries to react and respond. And certainly, if to the like, I think it could be quite stabilizing if it's true that it's much easier to sabotage an AI project than it is to build, to keep building it. That could be a stabilizing dynamic both because, you know, maybe you have these projects getting getting sabotaged, but also maybe that affects what kind of projects you undertake in the first place. And in the paper, they get into, like, how could you harden your project and make it harder to sabotage and how many years of development would that you know, if you have to build your data centers in a mountain, like, much many more years does that take, etcetera, etcetera. So I think there's a core logic to that that is, like, helpful, and I thought was a useful contribution to the discourse of saying, like, hey. There's actually gonna be, you know, multiple parties in this decision making system that you don't just get to say, hey, we're gonna race ahead and win the race. And the other parties have options beyond just trying to develop their own AI system. They can actually kind of engage with you in other ways. I thought that was helpful. I agree with you that I don't think it has the force that mutual assured destruction did in the cold war. I think mostly because of the clarity. The mutual assured destruction was so clear. We knew what nukes did. We knew how they looked. No one wanted to use them. We understood how it would work, you know, once second strike was really guaranteed. We understood what it would look like. Here it's just all so much unclear. Like what so much more unclear. What does superintelligence look like? How much does it matter strategically? So I think I think of their sort of main idea as kind of a helpful contribution, directionally useful, but definitely not that kind of crystal clear strategic logic that they I think presented it as. To your question about the sort of alpha go for war, I don't know. I don't think we're particularly close to that. I that just seems like such an intractable simulation problem because you don't just need you need the battlefield dynamics of a specific battle. You need the broader theater dynamics of, you know, what is where in the world. How are you bringing how are you bringing your assets to different places? That needs to be connected into broader economic and political questions of what is going on with the whole world. That needs to be connected to you know, the public attitude. It I it's just I think that is definitely people are trying to build simulations like that. I think they can be useful in limited ways, but I don't know. To me, it brings up this idea that I got from David Chapman who's a really interesting I think he was a philosopher but he doesn't identify as a philosopher and he's written a lot of interesting stuff on sort of rationality and what he contrasts it with as reasonableness meaning the how you make practical decisions in real world situations. Like when you're cooking breakfast, don't sort of sit down and make a 10 step plan to cook breakfast, you just start and then you see what happens and you adjust from there. And a point that he makes that I think is really correct is that people in technical domain sometimes think of the messy real world as kind of rough approximation of some much cleaner, more abstracted system. So, like, warfare is kind of a messy approximation of chess or of Go or of Starcraft. But in reverse in actuality, it's the reverse. It's that these clean games, these sort of simplified abstractable systems are really special cases, very special cases, very unusual special cases of the actual world that we find ourselves in. And I think that I don't know. This is maybe opening up a whole other can of worms that we sadly don't have time to get into, but I think it seems to me like a lot of the focus on AI development today is focused on assuming that you can build these clean abstracted systems that are amenable to, for example, reinforcement learning. And in the meantime, the models continue to struggle with that kind of real world practical troubleshooting, problem solving, readjusting along the way. All this a long way to say that I think military simulation is going to be incredibly, incredibly messy, way too many factors, way too many unknown unknowns, way too much ability for the adversary to deliberately throw a spanner in your assumptions and make your simulation sort of inaccurate. So I don't personally see that as alpha go for war as any kind of near term possibility. I think the broader question of what does equilibrium look like, I have no idea. I honestly don't know what the nation state looks like in a world of superintelligence. I don't know what democracy looks like. I don't know what the Chinese Communist Party looks like. So I definitely don't have kind of a broader answer for you there, unfortunately.

Nathan Labenz: (1:26:48) Well, that's maybe a great place to leave it. We have more questions than answers, and that's, like it or not, that's the reality. That's the timeline that we're in. Short or long, we've got a lot of questions that remain pretty vexing. This has been great, I really appreciate you for humoring me on some of the OpenAI questions at the beginning and also, you know, really admire how you've stayed in the arena and to look forward to your continued contributions to try to make the AI future a positive one for us and for our kids.

Helen Toner: (1:27:20) No. Thanks very much. It was a great conversation.

Nathan Labenz: (1:27:23) Helen Toner, thank you for being part of the cognitive revolution.

Helen Toner: (1:27:26) My pleasure.

Nathan Labenz: (1:27:30) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.