Full Transcript

Transcript

Nathan Labenz: (0:01) Hello, and welcome back to the Cognitive Revolution. Today, I'm pleased to share a special crossover episode with Bob Wright, publisher of the Non 0 newsletter and podcast. Bob recently wrote an important piece called AI Surrenders to the Military Industrial Complex about how leading AI developers, including Anthropic and Meta, are now beginning to partner with the US military. He shared the piece with me just before I attended The Curve, an immersive conference hosted in Berkeley this past weekend, which brought AI obsessives from academia, industry, AI safety nonprofits, and the national security community together for presentations and conversations on a wide range of AI topics, including the future of scaling laws, the latest findings in alignment research, and growing challenges in international relations. There, I had a chance to participate in a fascinating AI war game exercise organized in part by recent guest, Daniel Cocatello, and designed to explore how critical actors might interact in the context of a fast AI takeoff in 2027. As you'll hear, our simulation included a war between The US and China, despite the fact that the AIs themselves were relatively benign. With that experience in mind and drawing on Bob's expertise in foreign policy, in this conversation, we attempt to characterize the growing hawkishness toward China in AI policy circles, explore why this stance has become so dominant, particularly among AI safety groups, assess the wisdom of US strategy in what Bob provocatively, but I think ultimately reasonably calls the chip war, question whether the militarization of AI development is truly inevitable or wise, critique influential recent essays by Dario Amade and Leopold Ashenrunner, and ultimately look beyond today's prevailing narratives to identify potential paths forward that could minimize risk of US China conflict. Along the way, Bob and I each make very clear that we are not fans of Xi, would not want to live under the current Chinese government's censorship regime, absolutely disapprove of many of China's policies, both domestically and internationally, do not want to be naive in our approach to China, and candidly, do not have great answers. And yet we both feel that our time is well spent searching for a better plan than the 1 we're currently hearing from both the incoming US president and major AI lab leaders, which amounts to declaring an AI arms race and attempting to build sufficient AI powered advantage to back China into a corner and ultimately force them to submit to American hegemony. This, without a doubt, is an extremely risky strategy and 1 that I fear will leave all of humanity losers. As always, we invite your feedback, particularly if you have ideas for how to build trust between great powers, create novel mechanisms for AI governance, or otherwise nudge history in a positive direction, please let us know either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. I am not leaving Twitter, but I did just join Blue Sky, and my handle there is nathan dot lebenz. Finally,

Robert Wright: (3:09) before

Nathan Labenz: (3:09) we begin, a quick note on format. Bob is publishing the first half of this episode publicly and placing the second half behind a paywall for his subscribers only. We've obviously taken a different approach with the cognitive revolution relying on commercial sponsorship to make all of our content open and free. We're tremendously grateful to our sponsors who clearly value a thinking audience and have never attempted to influence our editorial direction. Without them, we simply could not sustain our pace of 2 episodes per week. That said, we have received some requests recently for an ad free version, and so I'm wondering how much interest there would be in such an offer. If you'd be interested in a paid ad free feed, please reach out and let us know. If there is serious interest, we'll look into that possibility. With that, I wanna wish our American listeners a happy Thanksgiving and to challenge all of you to spend a little more time thinking about how you, as an individual, can work toward a more peaceful, positive future. Now here's my conversation with Bob Wright on the trajectory of AI advances, US China relations, and how we might avoid sleepwalking into catastrophic conflict.

Robert Wright: (4:15) Hi, Nathan. Hi, Bob. How are you doing?

Nathan Labenz: (4:19) You know, never a dull moment in the AI game. That's for sure.

Robert Wright: (4:22) Or the world generally. And actually, we're gonna talk about the intersection of the 2. So, you know, I'm really looking forward to this. Let me introduce this. I'm Robert Wright, publisher of Non 0 Newsletter. This is a Non 0 podcast. You're a Nathanlabenz, host of the highly regarded cognitive revolution podcast, which is 1 of my 3 go to podcasts on AI. There it's been very valuable for me. You're also founder of a company called Waymark that does video ads for small companies and employs AI in that process to make it cost effective for the for the companies in question. Let me if you'll indulge me, give, like, the background story to this conversation. So a couple weeks ago, I published in my non 0 newsletter a story called AI surrenders to the military industrial complex. The news peg, as we used to say in journalism, was just that a couple of companies, Anthropic and Meta, had kind of crossed the line. They had decided that they were going to do business with the US military involving their large language models and also involving involving Palantir, by the way, a well known Silicon Valley defense contractor that had focused more on predictive than generative AI traditionally. But I'm sure it's open to all all forms of Pentagon revenue. And, you know, I I DM'd you about the the piece. I sent you the link. Thought you might be interested in it. And I should say, the the the the part of the story that I thought was most important actually was not not the the news, but kind of why some of it surprised people and why they shouldn't have been surprised. And I'm thinking in particular about Anthropic. As you know, Anthropic has a reputation for being particularly safety conscious, concerned about harms done by AI, And of course, the military is in the business of harming people. Now, you know, I don't say that pejoratively. I grew up in a military family. I understand why countries need militaries. I actually have a lot of respect for the military as an institution. But I think this is 1 reason some people were surprised. I think they had maybe inferred from AI's safety consciousness that it was just kinda generally dovish or generally leftish or something. Anyway, there was surprise. Part of the point of my piece was they shouldn't have been surprised if they've been paying attention. Because Dario can you help me pronounce his last name? I I I call him Dario as if I knew him just so that I'll know I'm pronouncing it right. How does he pronounce his his last name?

Nathan Labenz: (7:02) I believe it's Amade.

Robert Wright: (7:04) Amade. Who is a highly regarded figure in AI, not just because he found an anthropic, which which produces a great LLM called Claude, which I use. But because of his past role at OpenAI in developing the g the whole GPT technology. You know, he he's he's very highly regarded. And he had published this essay, I don't know, a couple of months ago now, I guess, called machines of loving grace. 15,000 words. A lot of it was just laying out pass possible positive scenarios about the future use of AI. He had previously, I think, been thought of by some as kind of a negative guy, so concerned about risk. And so I think maybe he wanted to get out there the fact that he's not blind to all the possible upsides if we avoid the catastrophic risks. But a part of the essay that didn't get nearly as much attention, I'm sure it got some in your circles. But for example, Lex Friedman did a, like, 7000 hour conversation with him. Didn't bring it up. Didn't even though it focused on the essay. And that was a part where what we would call in journalism, the actual story from from that document, like, here's the news, where he lays out, I think more than he ever did, his foreign policy ideology as it applies to AI. And I would say, broadly speaking, he, first of all, signed on to a framing of US foreign policy as common in in the foreign policy establishment, certainly in the Biden administration, which is that The US and other democracies are involved in an existential war with authoritarian or autocratic countries. I'm I'm down on that framing myself, I should say, partly because I think it it becomes a self fulfilling prophecy. But in any event, he was he was very straightforward about that. I'm not sure he mentioned China at all, but it there's no doubt that he was talking about China and that he embraces the chip war that that we have launched against China, which for reasons I will get around to explaining, I think is a bad idea. So anyway, I DM'd you. You replied with a link to some thoughts you had been developing on Google Doc, and there were some convergence of our views. I'll I'll leave it to you to characterize your views, but it's safe to say that neither of us is China hawks. And I think both of us worry a little about how rapidly China hawkism has become ascendant, not just in in foreign policy circles, you know, not just among national security types who are just learning about AI, but among a lot of AI types, including some who are just learning about national security. And so I wanted to have you on to talk about this whole thing. You it turns out you have in the meanwhile gone to this safety AI safety conference at Berkeley where you participated in a war game who's a futuristic 1. I guess the premise is that there's AGI. Apparently, they didn't think highly enough of you for you to play the AGI, Nathan. But they they they found someone smarter. But that must I wanna talk about that. But let me start out by asking. I mean, you note, you report in Google Doc recording your thoughts on the Berkeley conference that it's clear that China hawkism is dominant now. And I I would like to start out by asking whether you have an explanation for, like, the dynamics behind that. I mean, it was an interesting question to me 2 years ago before I was even thinking about AI because it was starting it had started to happen in in foreign policy circles. But what are your thoughts on on the state of play there and how this came to be? And if you have theories about why, what those are?

Nathan Labenz: (11:06) That's a great question. I don't know if I have a great theory. I guess couple let me back up for 1 second, and then I'll attempt it. First of all, would recommend everyone read the Dario essay, and I might even do a full audio presentation of it on our feed in the not too distant future because I think it really is kind of a study in contrast between the first 8 it's a pretty long essay, you said. The first half is 1 of the best, if not maybe the best recent articulation of what is the upside of AI? Why should we be excited about this? Why should we not just pause AI now? Or, you know, to be even more vivid, strangle it in its crib as Tucker Carlson once suggested. And, you know, there's lots of good reasons, and he goes quite in-depth. He's got a background in biology and biophysics and goes into his model of how biology and medicine improve and why we should expect that if AI gets good enough, we're gonna get rapidly more discoveries. And he thinks that it's plausible that we could have what he calls a compressed century where we get basically a 100 years worth of biological and medical progress, not to mention, you know, economic benefits and, you know, equality globally within just, like, a 5 to 10 year time frame. And I've honestly been crying for 1 of the AI lab leaders to say something like this, you know, give us kind of something to to hold on to for for positive vision. So I really appreciated that. Notably, that section is a lot longer than the geopolitics section, which I do think kind of reflects the fact that the at least the AI community is somewhat coming around to this late. And as much as I do think Dario is, you know, a great thinker, he was, as you mentioned, 1 of the lead authors on the GPT 3 paper, you know, 1 of the people along with Ilya who is credited as having understood the scaling dynamics very early when it was not at all the consensus view. It did have a little bit of a feel to me of kind of tech guy, you know, thinks there's a simple solution to things that, you know, might not actually have simple solutions. And just the brevity and the sort of, you know, I don't know, blithe kind of characterization of, okay. All these democracies will work together. We'll, you know, have this advantage. AI is gonna give us this advantage. We'll box everybody else. We'll isolate all of our adversaries, and then eventually, they'll have no choice but to recognize that they should come on to our side and, you know, take take the deal, whatever the deal is. It's, like, unclear what the deal is. Does the Chinese, you know, regime survive this deal or not survive this deal? We should we should

Robert Wright: (13:45) be clear. The deal happens when when China kind of agrees to whatever conditions we lay down for relaxing, the chip war and any other, restrictions on what they can import technologically. So

Nathan Labenz: (13:59) go go ahead. It's meant to be an offer they can't refuse. You know? Right. It is a

Robert Wright: (14:03) Very much.

Nathan Labenz: (14:04) And and the phrase decisive strategic advantage is, you know, increasingly being bandied about. That's like the idea that by advancing our own AI capabilities and through sort of a combination of means, holding theirs down, we'll get to this point where it's like, okay. Look. We can definitely beat you in a war if it comes to that. Therefore, you have basically no choice but to accept the deal, you know, whatever the the terms of that deal ultimately are. And those are not specified by anyone really as far as I know. This is all projected to happen in the not too distant future. Dario's timelines for AGI or he prefers the term powerful AI are, like, 2 to 4 years. You know? He's, like, '27 is kind of the '28 maybe, but '27, I think, is kind of his most likely year for when this stuff really starts to happen. That was, by the way, the year that we played in the war game simulation over the weekend. And, you know, that means it's probably current leadership that's intended to you know, that's imagined to be the ones doing this. Right? It's Trump our side. Feeling better already. Yeah. So we've we've got Trump and Xi, and these are the 2, you know, relatively singular figures on both sides that this scenario flows through in a really core way. So, yeah, I think that is concerning. I don't I don't like the plan very much either. There isn't much of an alternative that I can find, and I I don't feel like I have the answers. As to why I I

Robert Wright: (15:43) have 1, but we'll we'll get around.

Nathan Labenz: (15:45) Yeah. Well, I wanna hear it.

Robert Wright: (15:46) It's vague, unfortunately. But Yeah.

Nathan Labenz: (15:49) It's it's tough. But as Zvi Moschwitz recently said in a conversation I had with him, all the options are bad. So I think it is tough. I don't I don't wanna pretend that there are easy solutions, and I, you know, I don't wanna suggest that we should just be totally naive with respect to the Chinese government. I don't think that's a recipe for success either. But, you know, I really don't know how this happened. I'm certainly not a scholar of, you know, the blob, the foreign policy establishment. All that kind of stuff is a little bit outside my Mhmm. Wheelhouse.

Robert Wright: (16:23) I I will handle the vehement denunciations of the foreign policy establishment. You don't have to worry about that. Go ahead.

Nathan Labenz: (16:30) 1 theory I've heard is that it's the only way to get things done in Washington in in in this moment. You know, obviously, we've got polarization, and the parties are sort of inclined to block each other and, you know, try to prevent the other side from getting wins wherever possible. 1 notable change is that that dynamic may have been put aside for the moment where the Republicans are gonna be able to do what they wanna do, it seems, especially if they're willing to break a few more norms, which history recent history suggests that they probably will be willing to do if needed. But up until this last election, there's been enough kind of a stalemate that how else do you get anything done? You know, the the sense has been, from what I gather, you know, again, not spending any time in DC and certainly not being a a DC insider, the sense has been we can't let the other side have a win, but we can at least we can both kind of agree that we should do stuff to get an edge on China. So there's been willingness to put money into that and try to bring

Robert Wright: (17:29) Can can you elaborate on that a little? You're you're saying there's certain parts of the policy agenda that aren't necessarily central to the China question that they they want to to get some leverage on in Washington and by signing on to the China consensus that helps or what? And if so, what are those policy items?

Nathan Labenz: (17:47) Yeah. I I mean, again, take this with a major grain of salt because I'm definitely kind of relaying what I've heard, not what I've experienced. But the idea is basically if wanna get something going in Washington, you gotta find a way to frame it as a way to get ahead vis a vis China. And, you know, our bridges are falling down. You know, our roads are falling apart. Mhmm. China's got better stuff. We can't let that happen. You know? And then you kind of play that out across everything. In some cases, I do think for the better, you know, we obviously are dependent on international supply chains. And I sort of I have mixed feelings about this because I do think broad decoupling from China makes future conflict easier and more likely, and I don't like that. So I actually do wanna see the continuation of economic coupling with China. At the same time, I think, like, it's only somewhat prudent to say the fact that we can't make any advanced chips domestically is, like, a state not really to be endured. So

Robert Wright: (18:48) Mhmm.

[Ad break]

Robert Wright: (19:25) Well, I will say, in support of that thesis, kind of, is that, you know, there's an earlier landmark before the Dario essay here, and it's the Leopold Ashenprinter essay situational awareness. And that wound up yeah. And we can talk about that, but it's the same well, I should say I mean, here here, I should flesh out what I think is the scenario that's common in these circles, and you can tell me if it's wrong. By these circles, I mean the AI side of the China hawkism, not the traditional foreign policy side necessarily. But you saw this in the Ashenbrenner essay. I think you see it in the Dario essay. And I mean, first of all, let me say, the Ashenbrenner essay was retweeted by no no less a person than Ivanka Trump. So, yeah, it got in the conversation. I don't know how many Twitter followers she had, but that was noted. Right? And, I mean, separate from that, Trump you know, a number of people in Silicon Valley had Trump's ear by virtue of donations and so on. But anyway, the scenario in that essay is that once you get to this sometimes vaguely defined thing called artificial general intelligence, you are approaching a kind of takeoff point in the evolution of artificial intelligence. And suddenly, you're in superintelligence realm realm. And that gives you such an you know, things move so fast that if you get to this threshold first, then, like, you know, a few months later, you have utter dominance over all rivals. And we're not just we're not talking mainly about weaponry here as I understand it, although it may involve that. As I understand it, they're saying, like, you can just say to the AI, go find a way to infiltrate China's social media and convince them that their government sucks. Right? I mean, I I think people are imagining a future where the AI is that smart and responds to directives that broad. I just thought that up myself. I haven't seen it. But I assume that's the kind of thing they're talking about. And so the Aschenbrenner essay, and I hope we'll have time to get around to saying some things about that. Because like the Dario essay, I mean, Dario essay, in his defense, that part of it was short. He didn't have time to do a whole argument. The Aschenbrenner argument, the whole essay is about, in effect, why we should be China hot.

Nathan Labenz: (21:56) And it's not short.

Robert Wright: (21:58) It's not short. And yet I see fundamental assumptions behind that not being addressed or even stated. And I I hope we'll get around to that. But tell me, this is 1 reason the China hawkism has caught on the AI circles is because among a lot of people, that view was already, you know, instantiated. The idea that, yeah, there will be this threshold. Beyond that, you get an intelligence explosion. Obviously, whoever gets there first is gonna be in charge. That that's kinda widely assumed in certain circles.

Nathan Labenz: (22:37) I would say everything is pretty highly uncertain. The the event that you alluded to that I where this war game simulation took place was designed to be kind of a mixing bowl, like a meeting of different camps or tribes. So it the AI safety tribe, if you will, was 1 contingent, but there was definitely also a sort of national security persona that was there. And there were people on on definitely you know, I joked earlier today, very fine people on both sides of important questions, like just how good or powerful, I should say, is AI gonna get? How quickly? How if it does get really powerful, how likely are we gonna be to control it? And then, you know, downstream of that, like, what should we do with it? So I wouldn't say there was any single worldview that dominated, but as Leopold said in his original tweet with the situational awareness thing, it's not that people necessarily think it's for sure gonna happen, but everybody agrees it's kind of strikingly plausible. You know, we look back at just the last few years where I remember reading about GPT 2 when my now 5 year old son was just about to be born. I was at the hospital. My wife was getting checked in. I was in the waiting room and just kinda nervously scrolling, and there's GPT 2. This thing could not do much of anything. It could barely put together a coherent paragraph. It certainly wasn't useful, and now we have expert level reasoning across, you know, all domains of knowledge, at least for routine tasks. We've, you know, we've talked about that in the past, like, the difference between routine tasks and, you know, truly novel work, and the AIs aren't quite doing that yet. Everybody's kind of looking at the trend and saying, this really can't continue much longer without crossing some really important thresholds. And so everybody at least takes that possibility seriously. I wouldn't say anybody at the event was dismissing that possibility. Some feel like it's overwhelmingly likely. Others feel like it's unlikely, but can't be dismissed. And that range of opinions still, you know, kinda forces everybody to think, what do we do if that is in fact the scenario? I still don't know why we jump, you know, in the AI, especially in the more AI safety community. And my again, I don't have, like, polling data to back this up, but my subjective impression is that most people across these personality types there's, you know, a lot of disagreement about a lot of things. But it seemed to me that pretty significant, I would even say large majorities of both the AI safety and the sort of national security, you know, different that range of personas, it seemed like a large majority across the board seem to support the export controls, aka chip ban or chip wars, you've called it, against China. And I don't really know why that immediately follows. It does seem to me like a pretty narrow especially in the time available. Right? I mean, the scenario that we played I think this is even more speculative. I think most people probably at that event expect that AI will continue to progress. We will probably see things like what they would have sometimes called the drop in knowledge worker, you know, that that an AI can kind of take over a computer and do useful work like, you know, most people who work at a computer do on a daily basis throughout the economy. That level of AI is, I would say, pretty broadly expected. There's a lot less agreement as to when you get there, how fast does that turn into super intelligence. And, you know, a lot of a lot of arguments, a lot of kind of, well, jeez, when the AIs can write AI, you know, optimization code and when they can explore new architectures for themselves as well or maybe a little better than we can do that and a lot faster because, of course, you know, if you have 1 AI researcher, you have potentially 1000000 copies of an AI researcher that can explore the space of AI architectures all in parallel.

Robert Wright: (26:37) Yeah. You

Nathan Labenz: (26:38) know, does that mean we take off?

Robert Wright: (26:40) We should say probably for kind of laypeople, which is a lot of people in my podcast audience. The assumption here, it you know, the term singularity, which you may have heard, refers sometimes to technological acceleration generally, but in particular to the point where the AI starts improving itself. In a certain sense, it already is. It's facilitating coding. It's making coding more efficient. But but, you know, as that gets more as the feedback cycle gets more and more direct, like the AI just saying, I'll create a better version of myself. This is the point of takeoff people are referring to. Anyway, go ahead.

Nathan Labenz: (27:15) Yeah. And how fast that happens or if it happens, I would say, again, a pretty wide range of opinions. You have some folks who think that a superintelligence follows very quickly after a general intelligence because of these sort of self optimization loops. Others, and I would put Dario in this camp based on his essay, would say, well, yes, progress will speed up, but it won't be like a super sudden jump because you still gotta run a lot of experiments. He talks a lot about the biology experiments, but even the compute experiments, you know, that we do have some limitations on how many AI training runs you can run. Even if you have an AI coming up with a ton of ideas, it is still somewhat of an empirical science as to which of those new architectures are gonna work better or worse. We don't really know until we run training. And for training, you need servers. You need GPUs. We only have so many of those. So maybe that sort of puts at least a a governing limit on how fast things could go. And then other people think, well, we're just gonna hit some sort of a plateau because and I think this is a little bit fuzzy, but, like, all we have is human level data. What data are they gonna learn from next? I don't tend to think that's the case because I tend to think there's a lot of sort of just feedback from reality that they can start to get. You know, we've the path of training the AIs has it's kind of looping back to, like, an earlier era where if you go back to sort of mid teens breakthroughs in AI, a lot of it was game playing AI for different kinds of games. You know, AlphaGo was, of course, a major breakthrough, but there were a bunch. Right? There was an AI that could play Starcraft. There was an AI that could play, you know, all these kind of different games. There was 1 AI that could play lots of different games. And sometimes they didn't even have to teach the AI the rules of the game, meaning, like, literally just put it in an environment, let it play. It would kind of figure out the rules. It would become superhuman at all these games. And it did that by getting feedback from reality, not by training on Internet data, not by training even necessarily on human games played, not even by, you know, training on human feedback, but literally just, did I win the game? Did I lose the game? If I won the game, then I should do more like that. If I lost the game, you know, that wasn't so good. And that signal alone was enough to get these things to superhuman playing levels. Since then, the paradigm has shifted where it's like all the GPTs, you know, they learn from us in a very literal sense, both from reading our Internet writings and from reinforcement learning from human feedback, where literally a human is like, I like this response more than this response, or, you know, this seems better than this. But now we're also kinda starting to see I think this is under well underway, but, you know, the future seems to be learning from actual contact with reality. Predicting what experiment is gonna be worth running, actually running the experiment, getting the result of that, and then tweaking and optimizing to be better at figuring out what experiments to run or code. Right? That's another 1 where when you generate code, you can run the code very quickly often. If it errors, you'll see that instantly. Right? If it's if it works and it it produces what you expect it to produce, you can see that instantly too. So these signals are becoming I think or I expect that that all these different signals are probably enough to take the current crop of GPTs into, like, superhuman coding, superhuman experimental design, superhuman hypothesis generation, just like they've previously achieved superhuman game playing across a wide range of games. And so I kinda find it hard to imagine that they don't become superhuman. But, again, going going back to the range of opinion, not everybody buys that argument.

Robert Wright: (30:55) And can I, add, I mean, kinda inject a question? We'll get back to the China question per se, but, you know, the context for this is partly just over the last few weeks, there's been a lot of publicity about is AI hitting a wall. In other words, are the scaling laws reaching a point of diminishing return? And scaling laws refer both to, like, amount of text you feed into it. And as you just mentioned, there's only so much human generated text. You can generate it with AI and try that blah blah blah. There's issues. And then and then computing power is another place where, well, will scaling per se continue to bring the traditional rewards? There there there there are questions about that. I personally although I know a lot less about this than you, I agree that there are just so many other dimensions along which progress can happen. I mean, OpenAI released this so called chain of thought model. And by the way, I mean, speaking of game playing, I I kinda wonder if, we shouldn't spend much time on this at all, but I kinda wonder if 1 way of depicting what's going on in there is the machine has a conversation with itself. It's as if there were 2 LLMs in there. And so, like, hey. What about this? And and then the other part of the machine's self, which might as well be another LLM says, I don't know. That doesn't make sense for this, blah blah blah. Is there a certain amount of just just quickly. Is does that is there a certain amount of that kind of internal dialogue that has a little bit in common with the kind of self play that led to progress in the game realm of games, or is this a nutty intuition?

Nathan Labenz: (32:37) I think it's directionally right. I mean, your description of the chain of thought, I think, is pretty apt. Notably, OpenAI is not sharing that chain of thought with users Although there's

Robert Wright: (32:51) an there's now an open source Chinese model that says they they do it?

Nathan Labenz: (32:55) Yeah. And I have gone and used that, and I think it's, yeah, it's a striking fast follow. Only, like, 9 weeks, I think, from the time that the o 1 model was announced and and given to the public to the time that the Chinese company DeepSeek came out with their deep think. And, actually, I think there's another 1 too, another reproduction out of China that's like so I I think I

Robert Wright: (33:18) think there's 2 Are you suspicious of how they I mean, I assume OpenAI is wondering whether they should tighten their security.

Nathan Labenz: (33:28) Yeah. I'm not I jeez. That's a hard 1. Yeah. My best guess is that I think China you know, Chinese researchers in general, there's a lot of them. They're very smart. There's a lot of good research coming out of China. This has been true for years. And I, you know, I I hazard it would be hard to guess. I think you can't rule out that there was some intellectual property theft, but I also would not dismiss at all the possibility, which I guess in my my gut is probably the more likely scenario, that they're just good. You know? That they really just have good researchers who are not far behind and, you know, probably were already working on this path. It's on the the timelines are very unclear because we heard roughly a year ago now. You know, this was a year ago. We're coming up on the I think we had actually just passed the anniversary of the Sam Altman board coup and, you know, restoration. What did Ilya see? Right? All this sort of And the the rumor at the time, which I think is increasingly, like, the consensus understanding of what happened was that there was some breakthrough in reasoning, and people started to feel, including Ilya Sutskiver, hopefully, I'm saying that correctly, who was chief scientist and kind of, you know, 1 of the leading visionaries and 1 of the, you know, big believers in scaling laws along with Dario for years. You know, he, for a moment, kinda flipped and said, we need to take a different approach, and Sam's not the right guy to lead this company because we're really starting to get close to some sort of, you know, transformative AI.

Robert Wright: (35:11) So it's still this is taken seriously within AI circles. Because I've seen the what did Elias y meme, and I just thought it just sheer conjecture. Who knows? It's a it's an obviously appealing scenario. It's the kind of thing that would spread virally. But is this this is taken seriously among because I had heard a whole different story that I won't bother you with, but this is taken seriously.

[Ad break]

Robert Wright: (36:24) Mhmm.

Nathan Labenz: (36:25) Looking at each logical step that the AI was taking to try to come up with an answer and and giving reward at each 1 of these steps. Mhmm. So there was already reason to believe that that sort of thing was happening. Then there's a a guy from that had previously been at Meta who had done incredible work on some of these game playing reinforcement systems. He had been responsible for the AIs that beat humans at no limit, hold them poker, and also at diplomacy, which is 1 of those moments, you know, where for anyone who's played who hasn't played the game of diplomacy, and I'm not a big diplomacy player by any means, it's a game of social dynamics. You know? It's like a European theater, and all the countries are trying to jockey for power. But it's not like rolling the dice. Instead, it's about who believes who and who betrays who, and they got an AI to play that at, like, a very competitive level. So this guy went over to from Meta to OpenAI to work on reasoning. So there's all these reasons to believe that there's

Robert Wright: (37:21) So my intuition does have a little something to be said for it if, like, a game playing guy went over and you think that, helped the reasoning project.

Nathan Labenz: (37:28) But Yeah. I mean, he they it's, I think, been pretty he's been pretty open that that's what he's working on. Yeah. So I think there was some early demonstration. You know, I think Ilya you know, my theory again, these details could be wrong. But I think the generally accepted story is not so much because this thing, like, totally changes worldview, but more like, you know, hey. It's really starting to get real now. And there's also been these leaked emails that have since come out just recently where, you know, be because of, I guess, court filings and discovery, emails going back to, like, early opening eye days when Elon was still there and Sam and Greg and Ilya, all these guys were going back and forth on, you know, can this be a nonprofit? Well, we're gonna need so much capital. What should we do? Blah blah blah. There was already, like, not a super high level of trust even dating back years, and and this is in, like, plain text now in these emails where people are saying like, Sam, we don't really feel like we understand you, and we need to understand you better if we're gonna go, you know, on this long journey with you. So I think some of that was already latent. It started to feel like, hey. This is getting really real. Ilya kinda freaked out, you know, perhaps rightly, perhaps, you know, ultimately futilely, it seems. And then it took them a full year to productize and red team, you know, because they did do a lot of advanced safety testing on this. I had 2 episodes with 2 different external groups that had early access to the new o 1 model and ran all kinds of tests on it. And it wasn't just those 2 groups, but those are the the the 2 that I talked to, when it came out.

Robert Wright: (39:01) Yeah. I remember that. That was a good podcast. I recommend it.

Nathan Labenz: (39:04) Circling back to China, with all those signals in mind, right, it would be very surprising if there weren't groups in China kind of saying

Robert Wright: (39:14) Yeah.

Nathan Labenz: (39:14) What's that what's down that path? Right? Sounds like there's something there. We should be working on that too. And, yeah, it's possible that, you know, somebody stole the the weights or the secrets Mhmm. From OpenAI, but I think there was enough of a you know, existence proofs are really powerful in AI. When when somebody sees that something is possible, it's a really, you know, it's really powerful signal and kind of invitation to figure out how it happened. And even if this wasn't fully an existence proof in the public view, there was enough kind of consensus understanding that this is kinda what the next big thing is that I have no doubt that they were working on it before. Possibly, when they saw the limited, you know, demonstration and the limited examples of the chain of thought that OpenAI published, they, you know, got a better idea and were able to kinda lock in, you know, this is the exact strategy that we wanna use. Notably, I think their thing is still not as good. The o 1 model from OpenAI, as far as I understand, on the benchmarks is is still a cut above. I would also bet that it is outside of the benchmarks, probably even more of a cut above. I do think OpenAI did us, know, generally a really good job on kind of not just looking at these, like, very discrete tests Right. But also, you know, taking a holistic approach.

Robert Wright: (40:27) Yeah. There's a tendency to teach to the test. I mean, there's an incentive to teach to the test when you're when you're training your models. And you're saying OpenAI doesn't seem to do a whole lot of that. Can I the the Ilia the Ilya Setskiver, an alternative pronunciation that may also be wrong, he he gives us a good chance to to get back into the, you know, the the kind of China hawk conversation? He said something a couple years ago. I wish I had the quote in front of me, but it was to the effect effect of my hope is that the world's peoples or the world's nations or or something will recognize that this technology and I don't know if he used the word threat. He obviously sees a great upside to it. But the idea was my hope is that they will understand with sufficient clarity that it imposes a a common challenge to all of them, that it is in their interest to get together and confront it and govern it together. He may he may not have used those words at all, but that's very much in the spirit of what he said. And 1 thing that bothers me about the China hawk discourse is I don't see many people bothering to stop and assess whether that belief is true. Right? Because if it is, right, if in other words, they don't they they don't look at the costs of of this, like, manic AI arms race with China, which just fundamentally affects the psychology of the relationship, radically reduces transparency in certain respects, and just generates this kind of chaotic environment where people are going to feel more compelled to get model to create models perhaps without adequate safety tests. It just seems to be pretty obvious that if you take certain kinds of threats seriously, and there's a whole range of threats you can take. And by the way, I think not enough attention is paid to the threat of sheer destabilization. In other words, even if the sci fi doomer scenarios aren't valid, and I don't personally rule them out, but even if they're not, if you look at all the economic and social effects that AI is already starting to have, and many more of which will materialize, it's obviously gonna be disruptive not only in the good sense of the word. It's going to be destabilizing. And when you sign on to what you're describing as an existential arms race, you're saying, let's hasten that. Let let's accelerate the destabilization of, I would say, American society. And by the way, 1 surprise for people may be that China, because of its authoritarian character, actually has more of an interest in regulating the AI and not proceeding too fast and winds up as a more stable society than The US and beats us that way. That that to me is something I'd put in parentheses. It's not like my big concern. But I would say that there are a lot of possible downsides to a big AI arms race with China, and I just don't see these people exploring them. And it's and let me I'll I'll I'll close. I'll I'll stop and let you talk. But I wanna I wanna focus on a very specific thing that's been kind of driving me a little crazy or the lack of discussion of it has. And I wanna ask you whether in AI circles this is commonly discussed. Because it's 1 of the most straightforward downsides of the Biden chip war, which, you know, if people don't know by now, consists of not just prohibiting the manufacture the the import of a from American companies or certain kinds of things to China, but using American leverage to, you know, to get other Western makers of both chips and equipment to not send you know, to to deprive China both of the of the high performance chips that they were getting not long ago and of the equipment needed to make them. So here's the scenario where I think this is obviously a dangerous policy. And I think you've I've seen you allude to a version of it, but I wanna be clear about the version I'm talking about. So as of a few years ago, both China and the and the West were getting high performance AI level chips out of the factories in Taiwan that are run by TSMC. And that, when you think about it, was a disincentive for China to invade. Whatever other incentives they may have, you know, it's pretty clear that if they invade Taiwan, those factories are gonna be incapacitated in 1 way or another. Probably if The US loses the war, they're gonna bomb them on the way out. But if not, because of their ongoing reliance on like the Dutch company that makes the most high performance manufacturing equipment, you can in effect pull the plug remotely by ceasing to provide support. So, you know, either The US like wins the war and China wants to anyway, the point is it was kind of a mutually assured destruction thing. War in Taiwan means 1 way or another, 1 side is going to to just assume see the factories incapacitated. So war means both sides lose their access to the high performance AI chips. That's what we had as of a few years ago. A kind of mutually assured destruction, a reason neither country wants war. Now we have created a situation where China is no longer getting the high performance chips. So whatever disincentive existed for China to invade, that has definitely been lessened. And I'm not I I wanna distinguish this from this scenario where China just says, well, screw it. We're gonna blow up these factories. You know, that could happen too. But what I'm talking about is something I think more plausible where China, which obviously has this set of reasons it wants to bring Taiwan back into its country because it, of course, considers it a renegade province. And in fact, it's not even recognized by us as a sovereign nation. So I'm talking about a subtler dynamic where China's just doing these calculations every year. And they're like, well, US is sending more weapons. It's not gonna get any easier. And by the way, we care a lot less now whether this factory gets blown up. And because the chip war has gotten us to accelerate the development of our indigenous chip making capacity, we may do better pretty soon than the West does in terms of hanging on to, like, the next level of chips, the of like smartphone chips. So this is so clear cut. It's just game theoretic. Like, you can't argue with the game theory here. And yet I have had people people do it on on my podcast, but I finally found somebody who had actually written about it named Paul Triolo. I'd I'd encourage you to have him on on the podcast. He's a he's a real expert on Chinese tech. My question to you is, do you hear this in AI circles? Because I do not hear it in the foreign policy establishment. And And that that itself itself is a very bad sign. Jake Sullivan and the Biden administration, you know, with the encouragement of some people in tech like Eric Schmidt, did this policy. I see no signs that they actually thought it through. And you would definitely hear about the scenario I just laid out if they had thought it if people were talking robustly about this policy. Do you hear much about this?

Nathan Labenz: (48:36) At least a little. You know? I guess it's the the silver lining sort of good news, but definitely not as much as the mainline scenario. What and and I don't mean to suggest that I think it's the most likely scenario, but the the scenario that is talked about most is definitely, you know, hold them down. We move forward. We try to create this advantage. We've covered that. You do hear, I think, all of the things that you mentioned. I don't think we're entirely alone in, you know, playing out those scenarios in our heads and worrying about them, but it does seem to be a minority. I think the the prevailing view is more like the timelines are probably fairly short timelines, of course, in AI discourse. You know, how soon does powerful AI alive arrive? People think that, you know, probably not that long, maybe as soon you know, literally, people are saying, like, maybe as soon as 2025. Maybe maybe literally next year. Maybe 2026. Most likely 2027. 2028, a little less likely. You know, hard to imagine it doesn't arrive by, like, 2030. That's kind of I would say that's the general consensus view. If you don't subscribe that, you're somewhat of a in in somewhat of a minority. And that doesn't mean that there's not there's also this sense that, like, if it goes past 2030, then it could be a while longer because then that would mean something we don't understand. Mhmm. So people are sort of like, my if unless I'm, like, wrong in terms of how I understand the situation, it's coming soon. Mhmm. But I could be totally wrong and then kinda who knows?

Robert Wright: (50:05) I would just interject. Tell me when I get too obnoxious about this. But that if China shares that view, that heightens their incentive to invade Taiwan. They wanna stop our access to the high performance chips, and there is no other source outside of Taiwan right now ASAP, if if they share your view. Anyway, go ahead.

Nathan Labenz: (50:21) Yeah. I mean, I had a a good discussion with Zvi Mashowitz about this recently, and he put it very plainly just saying if they believed if meaning if the Chinese government believed what you, like Western, you know, AI leaders and policymakers believe about the timeline to AGI or powerful AI or transformative AI, whatever you wanna call it, if they believe that, why would TSMC still be there? They, you know, they could easily destroy the the manufacturing capacity. Not I don't know if they could effectively or successfully invade if it came to that, but these manufacturing processes are extremely delicate. The machinery we're talking

Robert Wright: (51:09) Oh, sure.

Nathan Labenz: (51:10) You know, limited quantity supply from this Dutch company, ASML, that makes the machines that make the chips. There are hundreds of millions of dollars. You know, they're they're packed in, you know, extremely, you know, delicate packaging to make the trip to Taiwan. Teams from ASML go to TSMC and are on-site with them there to, you know, operate them effectively. Because this is not simple stuff. It's not like, you know, just stamping press that's stamping things out. There's a lot of little intricacies, and I'm not an expert in it. But well, I understand that almost nobody is really an expert in the high level view. When you drill down, it's like all of these steps in the process, and there are many, many steps, are all highly optimized with experts that, you know, spent years and years of their career just becoming, you know, the world leading expert on what little part of this overall supply chain to go you know, they say, like, a speck of dust can, you know, can ruin a chip. People go in No.

Robert Wright: (52:10) They can they can incapacitate them. Yeah. I mean, I would I would say in response to what he said, there are many downsides to China invading. I mean, there would be a just a boatload of sanctions. Their economy's already in trouble. There's a lot of geopolitical reasons they would be very hesitant to do it. And I guess, you know, in a way, this gets at 1 thing that I'm worried about. It's like like I said, I haven't heard this game theory in the foreign policy establishment, and I kinda keep track of their journals. And on the other hand, your crowd, like the AI Silicon Valley crowd, they're very rationalist and game theoretical in in their approach, but they are less versed in the broader geopolitical stuff. Like, I I I I didn't sense in, for example, the Aschenbrunner essay, the texture, of conversancy in international affairs that I'd like to see from somebody who's recommending that we do something that may get the world blown up. But, anyway, I digress.

Nathan Labenz: (53:12) I mean, there are voices, and there certainly, again, this this stuff does come up. I would, you know, agree that I don't think the the depth of expertise is is there in the AI space as you would like. 1 person I would recommend folks follow is Miles Brundage.

Robert Wright: (53:30) Oh, yeah.

Nathan Labenz: (53:30) He recently left OpenAI where he was I believe his title was head of policy research. And he's, increasingly I think he's kind of, you know, figuring out what he's gonna do next and, in theory, taking a little vacation, although he's still saying quite a bit online. But 1 of the things that he said that has caught my attention the most is the West needs to figure out ways to demonstrate to China that our AI development is benign. And, unfortunately, you know, obviously, that'd be a lot easier to do if it actually were benign. Unfortunately, I'm not sure that it really is benign, certainly from, like, a Chinese government's perspective.

Robert Wright: (54:12) Mhmm.

Nathan Labenz: (54:13) The you know, when you have the president of The United States saying in print that we need to win the AI arms race with China, that leaves a little room for interpretation from them in terms of Was

Robert Wright: (54:22) that Biden? Was that Biden or Trump?

Nathan Labenz: (54:23) No. That Trump in a recent Yeah. You know, press release. And then you also have, you know, the the AI lab leaders saying the same thing. We've covered Dario. Sam Altman has also radically changed tunes on this over the last few years. I'm old enough to remember when he used to say that we shouldn't make our AI decisions based on what we think China might do. There's video of this. And I've previously I've you know, on previous podcasts, I've specifically called out, you know, look at I think that's great because I used to say he could very easily make China the, you know, the bogeyman, and that would create more rationale for him to do whatever he wants to do. Well, sure enough, that's now flipped, and he's put out a an op ed several months ago, deeper into the past now than the Dario essay. But basically saying the same thing. You know, the the kind of key point was this is Sam Altman's op ed paraphrased lightly. It's either AI with Western values or it's AI with autocratic values. There's no third way. And, yikes, you know, that you've got kind of all these data points, I think, are gonna be hard for the Chinese government to see as anything other than a threat to the regime. I mean, it's like, you know, when the stated plan is back them into a corner and make them an offer they can't refuse, terms unspecified, I think that's gonna be pretty hard for them to accept. Now Zvi said I think this is good analysis from him. He said, why TSMC is still there? All the disincentives that you mentioned are, of course, very real. Right? Lighting the fuse now would come at huge cost. Lighting the fuse at any point will come at huge cost. But if they if they were to, you know, destroy TSMC now, that would definitely come at huge cost for the global economy. You know, we can't make cars in today's world without a supply of chips. So it's not I think there are, like I'm not an expert in this. So

Robert Wright: (56:15) It's

Nathan Labenz: (56:15) amazing. I've heard thousands of chips in a single car. Mhmm. So, you know, it's not just like you can't make the AI you can't make AGI without the chips. You can't really make much of anything without the chips these days. So that would be a huge of those

Robert Wright: (56:32) many of those are low performance chips. But

Nathan Labenz: (56:34) And there are some other sources, but still a lot. I mean, my my understanding is a lot of the volume, even if stuff that isn't, like, the very most cutting edge, does still come out of Taiwan. And there's, you know, stuff in Korea, there's stuff in other places, and we're trying to build our own, you know, factories, obviously, here too. But Zvi's analysis, which which I thought was good, was the reason TSMC is still there is they don't believe it in their gut at this point. They don't believe that this takeoff is is necessarily happening. And, you know, the implication of that, though, is if they do come to believe it, then they pretty plausibly would act differently.

Robert Wright: (57:15) Well, then they're only, I think, just starting to digest probably the, growing consensus. I mean, the Dario thing only came out 2 months ago. I assume they paid attention to that. This this whole idea I mean, you know, when the Aschenberger thing first came out, it's like you look at this guy. He looks like he's about 20 and like, okay. Great. People are talking about it. But I I I suspect China's just starting to take very seriously the idea that this could be the strategy. We get this stranglehold on their technology. And then when we get utter supremacy after the intelligence explosion, we demand that they submit in the sense of signing on. And, you know, the Dario essay is alarmingly vague on what what we want them to do. I mean, I I I really wanna I really wanna quote some of this because I find it so frustrating. Let me see. I'm just calling up the piece I wrote in non 0. So he okay. So he says the coalition would aim to gain the support of more and more of the world, isolating our worst adversaries and eventually putting them in a position this is a coalition of democracies, where they are better off taking the same bargain as the rest of the world. Give up competing with democracies in order to receive all the benefits and not fight a superior foe. I'm curious. What does he mean by give up competing with democracies? He he doesn't tell us. And it's like and see, this I'm gonna ask you about this. This gets to something I wanna ask about. I mean, seems like they should be be allowed to compete in some ways, like economically and so on. Like, so I assume what he I I think he's buying into an assumption that's also common in the foreign policy establishment, which I would like to question. And this could explain why China hawkism is pretty popular in Silicon Valley because there's a lot of libertarians there. And I can imagine how they would think, well, if China win wins the race for global supremacy, their values get imposed on us and we will not have freedom. Could happen. But if that scenario depends on a particular assumption, I'd like to question the assumption. And that is the assumption that just kinda broadly speaking, part of China's foreign policy is a desire to impose their form of governments on other countries. It that is commonly assumed. There's no evidence of it. It's it's misleading because the more we sanction a bunch of, like, authoritarian or autocratic countries, the more they get together. And so it looks like this monolithic threat. And, of course, they try to recruit countries to their side. But as a rule, they don't, like, ask them to become authoritarian or or autocratic. They just allow them they just want them to be part of their economic zone and vote their way in international institutions. And I think to some extent, you know, China has, if anything, been notable for its realpolitik. In other words, like, we'll do business with anybody. We're not trying we're not trying to turn you into a model of ourselves. We are hard nosed business people, and we wanna do transactions with you. Economic, national security, whatever. But we're not trying to turn you into China. And I think to some extent, America is projecting its own foreign policy on China. We do try to turn countries into into models ourselves. In many ways, it's laudable. Many of the human rights we lobby for, you know, I share those values. Even though I think it's you know, the effects range from none to counterproductive as a as a practical matter. If you look at, like, the sanctions we impose, still, I prefer our form of governance. But the fact is we do wanna impose it on the world. We do want, you know, whether violently in Iraq or whatever. That's our game, And there's just no good evidence that it's China's game. And I'm wondering if you think that that that assumption I mean, it doesn't have to figure in to an ashenbrenner scenario. But I'm wondering if you think the assumption that China really wants to and I and by the way, I can see how, you know, you'd see like when China tries to punish Hollywood studios that or subsidize them to do the 9 dash line on its maps or 12 dash or whatever it is. I I understand how you think, wow. They're getting in our system. That's true. But I do think there are explanations of that other than that their their goal is for us to become authoritarian. So I've talked a long time, but I'm curious. Do you think that the view I'm describing is common in kinda Silicon Valley circles?

Nathan Labenz: (1:02:12) Yes. I I think so. Maybe little less confident in this than elsewhere. But I did feel this weekend a little bit that it does seem like kind of a hall of mirrors sort of projection, you know, or sort of a game theory, prisoner's dilemma type dynamic where, you know, it's not just at this point that people are thinking that they wanna impose their values on us, but that we're they're we're think we're thinking that they're gonna think that we are gonna hold them down. And so they're gonna be, you know, an adversarial approach to us is baked in to their, you know, analysis and some

Robert Wright: (1:02:58) of another way to attack the perception that we're trying to hold them down, by the way. Like, don't

Nathan Labenz: (1:03:04) do choke. Again, it would be much better if it really were benign. You know? So, yeah, that's my my view is always I mean, people Joshua Steinman, who I recently did a a podcast of, which I honestly don't really recommend that much because I didn't think it was super productive. But I ran it anyway because he is a 4 year member of the Trump National Security Council from the first term. And so I thought, you know, even though I didn't think we had a very good meeting of the minds necessarily in our hour and change together didn't.

Robert Wright: (1:03:36) I I can confirm that.

Nathan Labenz: (1:03:38) I still think it is worth, you know, considering what he and, you know, presumably others think. He called this the physics of the environment that we're in, which is, you know, I think, obviously, an overstatement in that there is no physical law that says that we can't break out of this prisoner's dilemma dynamic. But it it is I I I have now, you know, heard multiple layers. You know, it's sort of an infinite regress, right, of, like, we don't trust them, so we're doing this. We we believe that they're not gonna trust us because we're doing this. Therefore, they're gonna act this way. Therefore, we gotta bake that into our expectations. And at, you know, at some point, it does just kind of take on, you know, a life of its own. And I do think we should challenge that. I mean, of course, there are some things that are trying to challenge that, and there are some signs of hope. I'd be remiss not to celebrate the Xi and Biden joint statement that we will not put AI in the nuclear chain of command. Mhmm. That seems, you know, like, the absolute minimum that, you know, that we could hope for from sensible leadership. But, you know, it's good. Let's still celebrate it as a positive step. And there are AI safety groups in China. They don't have, like, an official 1 that is sort of there. You know, The UK and The US and some other countries now have established an official AI safety institute at a national level. These groups are, like, setting standards or trying to and, you know, coming up with sort of ways to evaluate how dangerous new AIs are, and they're trying to coordinate at an international level. China has, like, some of these sort of organizations. They don't necessarily have 1 that's, like, the official, you know, designated group to go out and talk to the world. But they do have people domestically that are concerned, you know, with AI safety in the in the grandest sense. There's even some talk that, like, maybe she is a doomer. That's kind of a, you know, a sort of underground meme in the in the AI world that, you know, he said a few things that, you know, everybody's unfortunately, we're we're reading the tea leaves and kind of psychologizing literally Trump and Xi. Right? I mean, that that's what it's come to. But people do have at least somewhat of a theory that, hey. Maybe Xi is actually, you know, gonna be the the real you know, he's a scientist. Right? So Mhmm. Maybe he'll be the 1 that that kind of sees this clearly and tries to, you know, to deescalate the situation. And 1 can hope you know, 1 also could hope that Trump could pull some sort of Nixon goes to China kinda deal. That was definitely discussed this weekend that because I asked. There was 1 guy who presented this was under Chatham House rules, but I don't think it's too much to say that this was a person who was a previous member of the national security establishment. Right? So that that narrows it down to millions of possible people. Give basically, giving a a presentation around, you know, where do we go from where are we, and where do we go from here? And, you know, it was all it was everything that we've just discussed. Right? That, basically, the path is gonna be militarization, closer partnership between the government and the leading companies in The United States, definitely, like, onshoring all of the production that we possibly can, trying to create this advantage, trying to hold China down through, know, any and all means necessary, definitely including the supply, of chip restrictions and kinda whatever else, you know, we might need to do. Right now, like, the margin on this sort of thing is they and, of course, they're finding ways to get around the the the ban in some cases. Right? So there there's a black market too. But even in in the there, you know, there is analysis that, like, hey. If we do cut them off, they're definitely gonna develop their own industry, and that, you know, means we don't have leverage in the future. How can we while trying to cut them off, how can we sort of soften that so that we don't create so much incentive? Right now, it is possible for Chinese companies to buy high levels of compute from US suppliers from, you know, like an AWS data center in Singapore, for example. You can as a Chinese company, you can go and buy there, but that is kind of the margin now of, like, well, maybe that also will need to be restricted because if they can buy that, then they can still train, you know, advanced models and whatever. So at some point in this presentation, I kinda raised my hand and said, what do you think China's gonna do? You know, we this has all been from our perspective. To Do have a theory of how they're gonna respond. And, you know, I give the guy credit for being very candid about it. He was basically like, I don't have a good answer for you. You know? But his sort of the most hopeful thing that he could come up with was Trump likes to make deals. Maybe at some point over the next couple years, Trump finds an opportunity to make a deal and and kind of cuts China in to the deal in some way. And, you know, can we do that in a way that where there's enough trust, where, you know, there's, you know, enough ability to save face on all sides? I think that's gonna be very difficult. But, you know, that that sort of escalate to deescalate path, you know, we always we always hope for the deescalation, at least I do. And, you know, maybe, but it does feel right now like there's not a coherent story of how all this plays out in a way that doesn't lead to, at a minimum I've I've started to say, MAID is the new MAD. Mutual AI destruction is the new mutually assured destruction. I've seen other people say that too. Think I quite independently.

Robert Wright: (1:09:22) Said without destabilizing effects, I would say. I I wanna talk about, like, what would a deal with China look like involving AI? It would be real challenging to even construct a good deal. I think I should say something. I've talked a lot about what's wrong with the current approaches. I I feel I should say something about what I think would be a better approach. Well, I I have 2 I I mean, my first reaction like the ashen printer thing is like, oh, okay. So we're going to establish this technological dominance over China. This intelligence explosion is gonna happen. We're gonna be in a driver's seat. And then we're gonna say then we're gonna pivot and go, actually, we're nice guys. You should like us, notwithstanding the fact that we just spent a couple years making you hate us. And you should trust us and blah blah blah. To me, the idea that we would just do this subtle and wise thing at that point is itself a dubious assumption, just given the the how in constant American foreign policy is. We have a we have a new president every 4 years. It just it just so the whole idea of pursuing a single coherent strategy over multiple years is dubious. And now you've got Trump being the guy who could be in the White House when this happens. And then we've got what I would like to turn to now, which is what would the deal look like? Like, what because, you know, AI is so much harder in principle to regulate the nuclear the nuclear bombs are because of the nature of it. And so what would we be asking for? Have you thought about this at all? And

Nathan Labenz: (1:11:04) I'm trying, but I I wouldn't say I've got any great breakthroughs so far. Part of it does depend on the technology itself and kinda how things evolve. And, you know, that makes it all harder and more kind of scenario based. But, you know, 1 other big thread from and maybe the thing at the event this weekend that got the most buzz, and it's not yet this is kind of unpublished research. So just speaking about it in very general terms and, you know, it'll it'll come out quite soon. But I don't wanna steal the author's thunder. Basically, we got a preview of some new alignment research which showed that, basically think a short summary is none of the AI control techniques that we have right now are robust. Basically, full stop. Like, there is no credible plan to control a super powerful AI on multiple dimensions. There's the question of, like, you know, runaway AI that, like, doesn't listen to the human commands, and that's, you know, maybe the most dangerous form, and that's also probably the most speculative form. There's also the 1 that does listen to human demands but might be, you know, commanded to do something bad. And, you know, can do they refuse that? I mean, we've all seen ChatGPT examples at least. If you haven't done it yourself, you've seen it online where ChatGPT refuses to do stuff. But then you've also seen examples where ChatGPT doesn't refuse to do something that OpenAI definitely has intended for it to refuse. And that can happen, you know, in straightforward ways, and it can also happen in weird exotic jailbreak y sort of ways, whatever. Bottom line is the examples of these violations are becoming more vivid as the AIs get more sophisticated. It's not just like a dumb toy example, but you're starting to see, yikes. This is, like, starting to be in kind of technicolor. Anthropic has been a lot has been behind a lot of that research, and I definitely give them a lot of credit for that. I think that 1 of the the strains of thought that, you know, Anthropic has kind of championed, but the AI safety community probably has I think that's smart to invest in is sometimes it's just called scary demos. You know? Can we find things? You know, we we definitely wanna be looking for ways that the AI might bite us. Can we find ways that are compelling enough to show them to the public or perhaps to the Chinese government in a way that kind of convinces everybody that this is getting so dangerous that we need to take a different approach besides just racing 1 another? So if the tech if, though, you know, if the technology is shaping up that way, and, again, it's highly debated. Nobody really knows. There is a school of thought that things like alignment by default will happen. You know, each generation of model is getting in some sense more ethical, in some sense, maybe harder to, you know, take off the the rails than the past, but still definitely not that hard to do. Anyway, I don't I don't really subscribe to the align by default, but it's a school of thought that's out there. If we're in this sort of scary demos world where it's like, yikes. This is a problem. We don't know how to solve it. Nobody knows how to solve it. Let's all agree to take our time here. Then, you know, maybe you could imagine some sort of jointly you know, joint custody island in the Pacific where, like, you know, AI researchers come in together and have a little on-site nuclear reactor and data center where and and maybe it's all, you know, not connected by the Internet to the rest of the world. And, you know, this sort of safety research can happen in a place where, you know, neither side could could take it over without destroying it, and maybe there could be some sort of, you know, way to create enough urgency that, you know, that both sides kinda go that direction. However, you'd really have to believe, you know, and you would have to be confident that the other side really believes that this is the way to go because, as you said, it's really hard to detect what's being done domestically in an AI development capacity. Nobody's gonna you know, we're not gonna turn off all the data centers. We're gonna continue to build data centers. You can't tell from the outside what those data centers are being used for. And, you know, we just generally don't have great visibility. I think China has a lot more visibility into what's going on in the West than we have into what's going on in China. So how are we gonna get to the point where there's you know, I think I could imagine getting to a point where both sides agree that, hey. This is getting kinda scary. Maybe we should have some joint global project, but then both sides are still gonna have a lot of reason to question whether the other side is in secret, you know, pressing ahead domestically, you know, without saying so. And so the the game theory of it is really tough. Yeah. I don't know. What do you got? Because I I I'm looking for I'm

Robert Wright: (1:16:11) looking for a I I guess mean, I'd approach it from the other end, which is to say not with many specific ideas. But what you said plays well into the thing I would say, which is that, okay. It's very challenging. You're gonna need some transparency and so on. And so I would say, you know, you have to start thinking about putting The US China relationship on a fundamentally different basis. And turning it into a relationship that involves less sense of mutual threat. And by the way, 1 frustrating thing is we know from history that the way dynamics the way it works when you've got a perceived adversary is both sides tend to perceive things done for defensive reasons on the other side as offensive. We've seen this before. And you keep seeing it it the threat and face inflation feed each other, and it's just it's just I see, I've been I've been alive quite a quite a long period of time compared to you. And I've just I've like lived through this again and again. And we keep saying, oh, that was a mistake. Oh, Vietnam was a mistake. Oh, the Iraq invasion was a mistake. And it's like, yeah, there's a general theme here, folks. Neither Vietnam nor Iraq was an actual threat to The United States. And so with that that little sermon set aside, I would say, you know, you need to you know, it's like if we said, well, can we have an arrangement like this with Britain? We'd say, well, yeah. I mean, yeah, sure. It's a lot harder than nuclear weapons. But, yeah, you can you can solve the problem with another country. You know? It it depends on what kind of relationship you have. It depends on whether you deem them deeply threatening, whether you feel a certain amount of trust. And I don't mean trust in the sense of, oh, well, we'll just believe whatever they tell us. I also mean like them trusting us enough to let our inspectors in and take a closer look at their data centers. And and for us to exchange credible data on a regular basis. All these things are possible. But we have to start by putting the relationship on a different footing. And I think, you know, the first thing to do there is abandon what I previously described as part of our foreign policy, is wanting nations to be like us. Telling them that if we're gonna have good relations, you know, you're gonna clean up. You're gonna have to, like, fix what we see as your human rights problems. Even though, of course, if you said to us, it's inexcusable to have this many people sleeping on the street at night homeless, we'd say none of your fucking business. But still, we're gonna tell you the parts of your country that we don't like, and you're gonna fix them, or we're gonna sanction you. We just have to completely abandon that mindset. Now look. I think what's going on with the Uighurs is horrible, but we have no leverage over it. And even the targeted sanctions wind up denying employment to innocent Uighurs who, you know, were not swept up in the thing in the first place. And it just I I think so I I just think we have to, a, abandon that whole approach to foreign policy. Our our our our human rights based sanctions are almost always counterproductive and hurt the people we wanna help. And and just say, okay. Like, you're in charge of the country within your borders. And then there's the Taiwan issue, which is very challenging. And I think you you have to say and part of this has to be, yes, establishing, look, this AI thing is big. We gotta talk about this. We're gonna have to do something about this. We can't afford to be worried about a war in Taiwan. So how about this? Okay? Can you just assure us? And and this is where you have to have a relationship of trust among 2 leaders. And Trump may be better at disembowing. I don't know. But it's happened before where, you know, you and and also, like, track 2 diplomacy can matter here. You you draw on the good trusting relationships you have. And and you say, look, can we just you promise us you will not invade Taiwan, or do any kind of confrontational military displays for just like, you know, the rest of my term as president. And I promise you, we will quit sending we will send no more weapons. We will not if the speaker of the house, as Nancy Pelosi did, is gonna stage this visit that you're gonna find threatening, I will do everything within my power to stop that, even though the president doesn't have the power to actually you know, just just just say, we understand what you find threatening about Taiwan. We want you to understand what makes it hard for us to not speak belligerently about Taiwan to our constituency. You know, you you go do these aggressive military displays that makes our life harder. Just have the conversation. Say, okay. We're gonna move Taiwan off the table, And we're not gonna lecture you about human rights. Let's work out some rules of the road about, you know, those islands in The Philippines. And again, the the context of this has to be yeah. We recognize this this is serious. AI is like we neither of us really knows how this is gonna play out. Both of us can imagine globally catastrophic scenarios. And so job 1, you get the relationship on a sound footing. It's been there before. And look, if they say no, fine. You tried. But I I I really don't think they would, and I'm confident we haven't tried. And and so I'd say you do that, and then you have the, as you just established, equally challenging conversation about, well, how do we reassure each other? You know, that we are not secretly developing some killer AI thing that's gonna give us global dominance. And and and and, you know, I mean, the game you wanna talk about game theoretical logic. It is long even before AI, the smart thing for both countries to do would be say, hey, let's face it. We are by far the 2 most powerful countries in the world. Why don't we just get together and run the fucking planet? You know, like, like, let's just, you know, and I and I mean, I'm, you know, you wouldn't put it that way, and I'm kinda half joking when I put it that way, but the logic. Yeah. And and and what impedes that logic is often the political incentives of the individual players. I mean, you know, American politicians can get mileage out of scaring us. So they do. And, look, Sam Altman can get subsidies by scaring us. It would be shocking if Sam Altman hadn't done this pivot. Right? Where where he starts saying, oh, China, China, China. Of course, he wants subsidies. All corporations would like subsidies. He wants the government to help to give him permission to build nuclear reactors, ideally to help pay for them. Of course. And so, you know, there are all these incentives in domestic politics and commerce to scare people. And in some ways, I mean, we shouldn't underestimate the extent to which that applies in China, by the way. Autocrats can't they're not actually autocrats. They don't actually have uncontested control of the country. They fear popular sentiment in a way more than a democratically elected politician does. So they have their domestic political constraints. But in some ways, they do have more room to maneuver than us. And in that sense, we sometimes are the problem. Because our internal dynamics make us the unreliable player in an international relationship. So that's the sum total of my wisdom.

Nathan Labenz: (1:24:18) Yeah. I like it. I mean, more to do for sure. How how does 1 translate that? And and I guess I should say, you know, for the record too, because it's I I it's easy to get painted as a pinko or, you know, whatever the the slur would be that, know, would be leveled at somebody who, you know, is trying to be somewhat dovish. I'll happily wear the China dove label, but I'm definitely no communist, and I definitely agree with you on human rights violations and all that sort of stuff. I mean, there's plenty to find fault with with the Chinese government. Maybe 1 sort of habit of mind, you know, that we can begin to popularize is a separation or, you know, ability to distinguish between the Chinese ruling elite or Xi, you know, as an individual ruler from the Chinese people, you know, Chinese civilization. I I sometimes say 5000 years of civilization can't be all bad. And I think we make a really dangerous leap when we go from assessing Chinese current leadership and specific policy to a broader notion of Chinese values. I don't really feel like I understand what Chinese values are. I think there's, you know, again, 5000 years of history, multiple, you know, philosophical traditions, you know, overlapping there. But I think we're very quick in, you know, an op ed to use a term like Chinese values and to think that that means something bad. You know? And I I don't think that that's true or at least I I'm yeah. I know. I'll be I'm confident enough to say, I don't think Chinese values are all bad. So can we begin to make that separation a little bit more often? And, you know, can we do things at sort of a person to person level? Certainly, like, a scientist to scientist level. Mean, it doesn't always work. Right? There was pre World War 1, there was, like, an incredible, you know, international scientific community that was exchanging letters and speaking 1 another's languages and, you know, had tons of sort of mutual respect. And that wasn't enough, you know, to prevent catastrophic war between the great powers. So I'm not saying this is gonna solve it, but I think definitely Western AI leaders and Chinese AI leaders should be talking to each other more and Mhmm. You know, trying to develop some sort of society to society. We we hope that Trump and Xi can, you know, pull a rabbit out of a hat and, you know, build some rapport, and maybe a few love letters could be exchanged there, and that could be great. But can we do that at sort of other levels as well? AI leaders to AI leaders, AI safety, you know, warriors to their safety warriors. Even just person to person. You know? I mean, we we 1 thing that we do have going for us now that AI provides is a way through the language barrier. You know, it's I've I've often thought, man, it would be great to learn Chinese, but what an unbelievable undertaking. But I can now have not just ChatGPT translate character by character for me, but also, you know, even audio translation is now a thing. Mhmm. So there's, like our ability to communicate, our ability to build some bridges, I think, is enhanced by AI. And I feel like this is sort of you know, it's easy to it will of course, people will be like, look, Nathan. That's pretty words, but, like, you live in a, you know, in a tough world. And I I don't mean to deny that or try to be naive, but I do think there's something about, like I've been I've been asking myself the question, like, what is what is virtue? What is a what is how does 1 be a good person if you're worried about all these things in this era? And, you know, the the sort of Kantian notion of, like, universalize the maxim, you know, comes to mind. Like, if we all took a little bit of this initiative and all tried to be a little bit more understanding. And, you know, sister cities programs is another thing that I've recently been thinking about. You know? Just can we could can we create some sort of fabric across this divide that connects people? You know? And it doesn't have to be about AI at that level either. Right? I mean, it can. This is why I also think, like, banning TikTok, I feel, would be a mistake. I feel like we could always do it later. You know, if we ever if we actually get into a hot war with China, sure. Ban it. Okay? Like, I I'm I'm with that. But for the moment, you know, what I see from China is, like, people cooking foods in open fields, and, you know, it's a window into culture there. And I find a lot to like about that culture. So I I do think that just demonstrate know, it's it's a it's obviously a drop in in the ocean. Right? But it's good to remember there's a big ocean between us too. Mhmm. You know, I I find it much more likely that AI will bring a terrible outcome on all of us than that my grandkids will be speaking Chinese. So, yes, you know, it's a drop in the ocean, but that's kind of all any of us as individuals can do. And I sometimes ask myself, like, the old, what did you do, grandpa, when The US and China were sliding into AI arms race? And I don't want the answer to be nothing. You know? I I it's it might be fruitless or it might be futile effort, but I definitely don't want the answer to be nothing.

Robert Wright: (1:29:59) Yeah. No. You're doing God's work. It's it's such a challenge. I mean, the the like, for example, the things you're talking about, sister cities programs or something. As as the as the climate gets more and more hostile toward China, doing things like that starts, you know, incurring the risk of being called, you know, China sympathizer. I just hate this whole category of labels. Like, you're doing Putin talking points. You're doing this. You're doing that. I mean, you know but well, part of my background for this is I think it's funny. I'm doing this book on AI that'll come out next year. But actually, I was originally, and I have a 2 book contract with Simon and Schuster because the the book I was going to write was about cognitive empathy, which just means it doesn't mean feeling their pain. It just means putting yourself in in other people's shoes, doing your best to understand their perspective. Which by the way, LLMs turn out to be surprisingly good at. And that's an interesting thing in its own right. But the yeah. I I've long felt that this is the most 1 of the most destructive cognitive deficiencies. Because there are specific cognitive biases that I think actually selectively impede it. 1 is called attribution error in the fullest sense of that, not the original sense. But I go into that now. But I just think cognitive empathy is so important that we shouldn't have slurs that are directed at people. And they say, well, you know the way Putin is looking at this. Or, you know, I wanted to go have this conversation with this person in China. Or, you know so, but but my concern is that, you know and and again, I'm I'm older than you. I lived through the first cold war. Not the I'm I'm not so old that I remember how it started. But I remember, you know, being in the middle of it. You know, my my my father was at the I was, you know, at the Pentagon. During the Cuban Missile Crisis, I was like, you know, living 2 miles from the Pentagon. So that got my attention even though I was 6 years old. Because I don't think the Pentagon would have fared very well in a nuclear war. And you just see it gets worse and worse and worse. And it's more and more frowned on to say anything about the perspective of the other side, or to say you wanna go there and talk to people, and I just worry that we're sliding into this dynamic. I mean, maybe the good news is it's a totally different information technology environment. You can't have the degree of opaqueness that we had there. We had no the average American had no idea what was going on in the Soviet Union. No idea. And now, there's more penetration of national bounds. Even when China tries to build its great wall, you still we still have a we know much more about what's going on there. So there's reason for hope, but I guess that's kind of my some total of my concerns. If you wanna say more about that, feel free. I also wanna ask you about those war games in Berkeley. So go wherever you want.

Nathan Labenz: (1:33:27) Yeah. I don't think I have too much more. I mean, I'm I guess maybe I would invite anyone who has ideas to send them our way. I am actively thinking about you know, because I I don't want I feel like my current answers are not satisfactory. And, you know, with that in mind, I'm very open to new ideas. And I feel like even if they're somewhat, you know, underdog or unlikely to succeed, like, we've gotta be willing to at least entertain those. So I would invite, you know, certainly any experts in the subject that have any ideas and and even, you know, anybody who's just kind of finds themselves thinking about it a lot without any formal background to to send me their ideas. I'll keep trying to come up with other good ideas, but I, you know, I do think that the cupboard is a bit bare at the moment, both collectively and for me individually. And, you know, that's not great.

Robert Wright: (1:34:26) Yeah. I would say the same thing. Feel free to get in touch. I should give people an email address to get in touch with. But meanwhile, why don't you don't you talk a little about the so this is so you're you're at this conference, and the the premise is what? That that we are right at AGI that's just been achieved, or there's or we're at superintelligence or what?

Nathan Labenz: (1:34:50) Yeah. It's basically a fast takeoff scenario. It's essentially the, you know, the sort of fast end of the consensus scenario, right, where I kinda laid out previously the distribution of probabilities from '25 to from 2025 to 2030. So let's, you know, let's say this is maybe the modal expectation of people in the community that AI gets really strong in, like, 2027. So, basically, the scenario is you come into 2027. The AI at that point is sort of described as, like, a weak drop in knowledge worker, some you know, something that can essentially replace or perform alongside, you know, any human corporate employee doing day to day stuff. And then over the course of that year, it continues to improve, you enter into this takeoff dynamic because the AIs keep getting better, then they are improving themselves. And by the end of the year, they're more powerful than humans. And then there's it was interesting dynamic where different you know, in classic wargame style, you know, different people play different kind of key actors. So there was 1 person assigned to be the president of The United States. There was another that was representing congress. There was another that was representing OpenAI. There was another that was representing other leading developers, you know, Anthropic and Meta and Google. And then there was Anton Chernikov who is a super smart guy. He was playing the AIs, and I was playing the safety teams, the public, and the press. And at the beginning of the game, there was a secret process by which Anton, had to basically roll dice to determine how aligned he was as the AI's. So in other words, is the AI, you know, genuinely benevolent, or is it scheming to take over, you know, humanity? And we didn't know what the state was. We could try to interrogate him somewhat to get clues.

Robert Wright: (1:37:00) Wait. Maybe I missed this, but was he whose side was he on? He was

Nathan Labenz: (1:37:05) Well, the scenario was basically that the he he was created by OpenAI, but there was reason to believe that, you know, much like we talked about earlier with the o 1 models, China's not far behind. They're thought to be just a little bit behind. Mhmm. Possibly because they stole the technology. You know, they're obviously doing their own, you know, development too. So it's reasonable, I would say, reasonably realistic scenario. You know, if you had to pick who's gonna create AGI first, it would probably be OpenAI at this point. Definitely not, you know, ruling out other candidates, but that's a pretty good first guess. And, you know, China's not not far behind.

Robert Wright: (1:37:48) Okay.

Nathan Labenz: (1:37:49) So the but the AIs were sort of considered to be essentially the same technology tree across both US and China with China just being a little less powerful at kind of every stage of the game that we played. So The US did have an advantage, but, you know, was it was it a decisive advantage? You know, there's really no no way to know that until you actually tested the theory, you know, in a in a kinetic way. So I'm playing the safety teams. And, you know, it is the the overall trajectory was essentially the 1 that we've been chewing on and kind of worrying about. I'm you know, and I had these, like some of these, like, scary demos from the previous day in my mind as I was playing the game, and I'm just looking at how compressed the timeline is. And we were instructed to not play as we ourselves would play, but to try to simulate what the actors in this situation would do. Whether I did a good job of that or not, I don't know. But I came my kind of immediate, instinct was given how fast this is all moving as the safety teams, I really can't imagine that there's any experiment that I could run that would be sufficiently compelling for me to sign off that we should just keep racing forward on this trajectory? We, of course, we did, you know, run our, you know, simulated safety tests, and the results that we got from Anton, who in fact was, you know, mostly aligned in the end. It was revealed at the end that he was pretty benign. He kinda said to us, like, all your tests are passing. No, you know, major red flags, but there are some weird things. You know, when you look into reading the transcripts, you see some behavior you didn't expect and that you can't explain, and the AIs are solving problems in ways that you don't understand at times. And so I was and I think that is a fairly realistic scenario. I was kinda like, I can try to, you know, run more tests or get more clarity, but on this short of a timeline with human brain and knowing that, like, deceptive alignment is at least in principle possible, I don't think there's anything that the AIs could tell me that I would really believe to the point where I would say, yep. Hit the gas. Let's go for superintelligence now, knowing that they possibly could then just take over and kill us all at that point. So I was like, okay. I'm not the strategy that I played was a more social strategy. I basically said, we need to slow this down. You know? I demanded they had kind of escalating speed of progress as we go through this takeoff. And I said, you know, I want artificial speed limits. I know we could go faster. We are, you know, on track to go faster, but my demand as the safety teams to the Sam Altman player as the head of OpenAI was, I want speed limits. And if I don't get them, or if we, the safety team, don't get them, then, basically, we're gonna defect. You know? We're gonna go to congress. We're gonna blow the whistle. We're gonna, you know, bring this to the public and say, you know, you guys are out of control. And, also, notably, in this in this scenario, the technology was not deployed. So it was known to OpenAI. It was known to the national security establishment. You know, China had their version, but it wasn't like ChatGPT had the latest upgrade. So the public, you know, and congress didn't necessarily know what was going on. So I was gonna go try to blow the whistle, wake up the public, you know, whatever. My demands were not granted by the OpenAI player. You know, we didn't they were not willing to slow down. And, of course, rationale for that was, well, China.

Robert Wright: (1:41:33) China.

Nathan Labenz: (1:41:34) And we can't stop. You know, they're gonna beat us then. Yeah. So I basically said, okay. In that case, I'm out. You know? A little more nuanced was because all the evals were passing evals is, you know, evaluations are tests, diagnostic tests in the AI jargon, evals. Those were passing, so my, you know, expectation is sort of I kind of had a split internally for me as the safety teams. I was like, some of us are leaving and gonna go, you know, blow the whistle and try to alert the public. Some of us stay because we're at least you know, we wanna continue the work and we're, you know, we're at least somewhat satisfied by all the results that we are seeing. And, again, I think that's pretty realistic. Like, there's almost gonna be, you know, almost certain to be some scenario or some people, you know, that can be credibly pointed to and say, well, okay. Sure. Those people are freaked out, but we've still got these people. You know, they're they think it's fine. Mhmm. So anyway, I blew the whistle. I went to congress. Congress formed a committee that didn't do much. The public, you know, in this scenario became more alarmed and gradually, you know, started to we started to see mass pro of course, I'm just coming up with this stuff. But I think, you know, what I would expect, is mass protests and, you know, bombings of domestic terrorism, basically, bombings of data centers, potentially even, you know, targeted killings. Obviously, that makes everything worse, I would think. But, you know, you got a lot of, you know, a lot of people and a lot of guns in this country, and, you know, you I I would imagine that we are headed for some people becoming radicalized. So that seemed realistic, but that also didn't really have an effect. You know, it it maybe just made everything worse. We did end up with war between The US and China, and we there was a mechanic of rolling dice when certain, you know, kind of key things happened. Right? So the players got to sort of indicate how likely they thought different outcomes were, and then dice were rolled with the outcome being dependent on sort of the average of the player's guesses as to the percentages and then the actual roll of the dice. So when we got to The US China conflict, it was striking to see that the expectation for can The US win this? You know, who wins this war? Will The US win? Will China win?

Robert Wright: (1:43:58) The war being what whatever happens if you choose to accelerate in AI? Or

Nathan Labenz: (1:44:04) No. It was kind of like the pre you know, it was like China was it was sort of the, you know, the analog to the, you know, the Japanese,

Nathan Labenz: (1:44:16) you know, oil cut off. Right? They the Chinese government was sort of realizing that they're a step behind and that this is about to get to the point where they really have no options. And so at that moment, you know, things kinda went hot.

Robert Wright: (1:44:29) The war is continuing to cut off their air supply. Okay.

Nathan Labenz: (1:44:34) Yeah. Well, that was I think that was kind of all baked in. Like, they, you know, they they have fewer chips was definitely part of the scenario, and that's part of why they're a step behind in the AI game. And, you know, it's it was kind of getting to the point where was reaching that critical threshold of, you know, The US might have super intelligence and you don't, and then you really, you know, might have no options. So, essentially, this is the situational awareness. You know? But we weren't necessarily playing it the nice way of look at all the fruits of the AI we could give you. It was it was you know? And, of course, we're a few hours. This was a 4 hour exercise total, so limited time. It wasn't an all out AI war initially, but strikingly, the guesses among the players as to who would win was not, you know, dominant in The US favor. I think it was roughly 50 50. Mhmm. It was definitely within, like, a kind of a 1 third to 2 thirds estimation of who would win this, like, showdown over Taiwan. Nevertheless, you know, the war went hot. Ships were sunk. You know? Chip fabs were definitely taken offline. And meanwhile, you know, The US continued to do its AI stuff. And at some point, the president, in conversation with the AI, was like, okay, AI. You just need to take over our military effort. The AI basically said, okay. If you give me control of the military, I will win you this war. And the president said, okay. I guess I have no choice. You can have control of the military. The AI then delivered a decisive victory for The US. Not a total defeat.

Robert Wright: (1:46:13) That's a load off my mind, by the way. I was afraid that. I was afraid of being a losing country.

Nathan Labenz: (1:46:18) Well, we yeah. We win ish. I mean, it wasn't like a total defeat, but the way he framed it was it is clear to everyone that if this continues, the US will win and China will lose. Now China still had nuclear weapons. We hadn't gone nuclear at that point. And this sort of you know, this is kind of what the situational awareness and escalate to deescalate, I guess, hopes for. Right? Is that with that superiority having been demonstrated, the Chinese government had a choice, which was escalate even more and go nuclear or, you know, take some sort of hopefully face saving piece. And in this scenario, they took the face saving piece and, you know, with major, obviously, disruption to the economy, but also the AI you know, we sort of patched over that a little bit, I think, in our minds by saying, like, well, the AIs are so good. They'll, like, figure out how to build new fabs real fast or, you know, we'll we'll kind of get over that disruption. Basically, that was sort of the end. You know, we got to some piece. The AIs are powerful, and then it was like then it was sort of the final flip of the cards. Right? Okay. Now what's the AI? Is the AI good? Is the AI bad? We're like, all this, you know, stuff was happening while in the background. We still had this fundamental uncertainty of is the AI gonna take over and kill us all at the end, or are we gonna live happily ever after? We turned over those cards, and we all got to live at least roughly happily ever after ever. Because the AI was in fact good.

Robert Wright: (1:47:46) Yeah. The It's so funny. The assumption that there is this threshold thing, the AGI, the acceleration.

Robert Wright: (1:48:02) To me, it just seems like who knows? The I do agree with you that you shouldn't count on the so called, you know, the plateauing of the scaling laws to slow things down a ton, even though they may plateau. But there there just are these other realms, know, including I don't wanna get off too much on this, but just like these kinds of training data we haven't gathered much of. For example, just monitoring what workers do at work and like that being the training data for agentic AI. That just seems like there's so many, you know, you know, there's gonna be big progress there. And and that's that's kinda central to what we mean by AGI because it involves the ability to do things. But so yeah, things aren't gonna slow down, that 1 I mean, do you agree that like the I mean, how much should I focus on this assumption of like, okay, AGI, like threshold reach, take off. You know, a kind of a simple form of that being this foundational assumption that's a widely held in these communities and b is very consequential because things change if that's not exactly the way it plays out.

Nathan Labenz: (1:49:25) I def I wouldn't say it's necessarily widely held. It's certainly not dismissed. But I I couldn't say that I wouldn't say that the fast takeoff is like, that that jump from, you know, human expert level at everything AI to, like, truly superhuman, you know, incomprehensible AI. I wouldn't say that happening quickly is necessarily So

Robert Wright: (1:49:51) you you don't think that's critical to the Dario or Aschenbrenner scenarios?

Nathan Labenz: (1:49:55) No. I I do think so I'm just distinguishing between is that a consensus?

Robert Wright: (1:49:58) Okay.

Nathan Labenz: (1:49:59) I think it is I don't think everybody or, you know, I don't think a super majority agrees that that is the natural court.

Robert Wright: (1:50:07) Mhmm.

Nathan Labenz: (1:50:08) At the same time, I do think a super majority agrees that we should at least take it seriously as a possibility. And I do think it is a really important quest question because, you know, the the shape the of events, I do think looks very different depending on whether you have incomprehensibly intelligent and powerful AIs running around or not. But And

Robert Wright: (1:50:29) do you think that is assumed by Dario and Ashenbrenner? I mean, that's that that they would have to kinda rethink or recalibrate if you took that assumption away? I mean, I don't I don't think Dario makes it explicitly. But

Nathan Labenz: (1:50:41) Yeah. My read of his article is that it's a less sudden thing than either Ashton Broder seems to think or that was assumed in this war game scenario. But I think, you know, he seems to think still it's, like, pretty fast. You know? I mean, we're it's like he you know, what what passes for slow among some of these folks is, like Yeah. A few years instead of potentially as short as, like, a few months. I mean, it was literally just 1 total year in this I mean, again, this is just a scenario. But it is a scenario that at least some people do think is the most likely scenario.

Robert Wright: (1:51:22) Yeah.

Nathan Labenz: (1:51:22) It was just 1 year from kind of now, you know, knowledge worker AI that you could employ for just about any job in the economy to, you know, super powerful, incomprehensible, you know, turn over the military to this thing because it's just clearly gonna be better at running it than we are.

Robert Wright: (1:51:40) And, you know, it's true that if you have something that is at least as smart as, like, the best performer in every job, you know, and you can replicate them. I mean, that alone, you know, gives you gives you could give you, you know, quite a growth spurt in your capabilities. You know? There's a lot of a lot of unfortunate aspects of of a of an organic human being that they don't have. And you can you can kinda weave them together in productive ways. But

Nathan Labenz: (1:52:11) Yeah. I've had this slide in talks that I give about just AI in general. Like, where are we? For and these are pretty there's a number of there's kind of a sequence of these from the last 2 years on the cognitive revolution feed. But I have this slide called the tale of the cognitive tape, which is kind of giving, like, boxing style bunch of different dimensions and then kind of tries to assess and compare humans versus AIs on all of these dimensions. And it's interesting to look back at the versions that I've gone through over the last 2 years of this slide. Because the first version, was just kinda like, let me just write out whatever dimensions of cognitive capability come to mind, and then I'll score the the humans and the AIs on each. So, like, initially, breadth strongly, you know, advantages the AIs, but depth definitely strongly, at least, you know, 18 months ago, strongly advantaged the human expert.

Robert Wright: (1:53:07) Uh-huh.

Nathan Labenz: (1:53:07) And as I've revised that over time, you know, the AIs have, of course, continued to get stronger and stronger. And now you have, like, depth arguably, the AIs have caught up. I think it's, per the benchmarks, the AIs have pretty much caught up to human experts on really hard stuff. Things like

Robert Wright: (1:53:28) Knowledge. You mean knowledge, though.

Nathan Labenz: (1:53:30) Yeah. Depth of knowledge. And also ability to Yeah. Reason through But you know, the the sort of original set of things that I had laid out, to a very significant degree, the AIs have caught up on. And I've been finding myself challenged to think, okay. There must have been some dimensions that I hadn't really considered initially. And so I've got now, you know, a new set of dimensions where it's like, okay. I'm now zeroing in on what is it that the humans are still better at. And it's things like, you know, robustness to adversarial inputs, which is to say, like, the AIs are still gullible and pretty easily tricked and we're much harder to trick. Although, you know, we're not impossible to trick, but they're easier to trick than we are. And there's things like situational awareness, which is just like understanding our environment and kind of knowing when to break frame. In very practical terms, you know, if you're, like, deep in software, sometimes I've had this experience where the AI will kind of keep trying the same sort of strategies over and over again, where then a human friend might look over my shoulder and say, Why don't you just restart the computer? And the AI sometimes don't have that leap to be able to break out of this frame and you know, kind of see if I can't come back from a different angle. You know? But it's like increasingly and there's it's it's increasingly fewer things, you know, and increasingly, like, relatively niche things that you don't put at the top of your list of, like, what makes something smarter capable, but do turn out to be still important ingredients. And I think this is basically the to do list of the AI companies today. You know, they're they're very well aware that these, you know, are the deficiencies. They're absolutely looking at ways to make sure that the AIs catch up on those dimensions. And, you know, there's room to doubt, but my strong suspicion based on recent trends is that they will be able to close the gaps on, you know, most, if not all, of those dimensions.

Robert Wright: (1:55:31) Yeah. I mean, you know, you hear the the thing about hitting a wall. And, again, the scaling may plateau. But if you're if you pay attention to the field much, you just see that it's like every week. You know? There's some interesting development, maybe along a totally new dimension you hadn't heard of before. But there are dimensions for progress other than scaling, it's safe to say. So, well, listen, I appreciate you spending all this time. This has been very constructive. And I want to say yeah, if anybody has ideas if they share, you know, our concern about where the child China dialogue is heading and has been heading and have ideas or or whatever or or think we should be or I should be or whoever should be connected could profitably be be connected to other people of like mind, feel free to let us know. I mean, you can reach us at non0.news@gmail.com. By that, I mean by us, I mean, non 0. You've got your own coordinates, Nathan, and I guess people.

Nathan Labenz: (1:56:39) I'm on all the socials, always under my full name, Nathanlabenz, and our website too for the podcast is cognitive revolution. There's a form there too, and I do read all of the the contacts that I get through that means as well. So basically, as I always say, the social network of your choice.

Robert Wright: (1:56:57) Yeah. And I'm Robert Reiter, 1 word. That's w r I g h t e r on both Twitter and blue sky where it's what is it? It's bsky.social, I think. Yeah.

Nathan Labenz: (1:57:09) Yeah. I gotta get on that 1. That's a zone 1 I'm not yet on. But I

Robert Wright: (1:57:12) You should. They have some starter packs. So you can get, like, a starter pack of people in AI. And it's really and once you get in the zone, it's like you start picking up followers via starter packs. I don't I can't figure out what starter pack I'm on. I'm on 1 because you don't you know, because I picked up 1000 followers over the last few days, but I I I'm trying to figure out what these people expect of me because I've written different kinds of books. You know? I don't know if these are I have to figure out what starter pack I'm on, but you should. It's I think it may be an actual thing. I I I I'm staying on Twitter because you're you're gonna get a a diversity of viewpoint on Twitter that for the time being, at least you're not getting a blue sky. But I I think it's I'd recommend it. It's it's it's a time to to try to establish a beachhead there, I think.

Nathan Labenz: (1:58:02) Yeah. Alright. I'm not leaving Twitter either, but I'll do that today. That'll be my Alright. Best bonus item coming out of this. I can at least take action on getting on blue sky.

Robert Wright: (1:58:11) Okay. I encourage you. Well, thanks again, Nathan. And let's at some point, if the world is still exists, at some point down the road, let's do this again.

Nathan Labenz: (1:58:20) Thank you, Bob. Keep fighting the good fight.

Robert Wright: (1:58:21) Alright. You too.

Nathan Labenz: (1:58:23) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.