Full Transcript

Nathan Labenz: (0:00) Hello, and welcome back to the Cognitive Revolution. Today, my guests are Jeremie and Edouard Harris, founders of Gladstone AI and authors of the sobering new report, America's Superintelligence Project. Many listeners will recognize Jeremie from the last week in AI podcast, a show that I've repeatedly recommended and continue to listen to regularly myself as Jeremie and his cohost Andre do a consistently excellent job of getting into real and often quite technical depth on the most important stories in AI. With Jeremie in particular presenting a very sophisticated and nuanced perspective that balances genuine enthusiasm for AI upside with sincere concern about loss of control and other existential risks. Today's conversation offers a more zoomed out perspective on what is arguably the core dilemma of our time. Racing ahead with AI development to stay ahead of China increases the risk of loss of control and other catastrophic accidents. But at the same time, insisting on adequate safety standards and allowing time for safety research to develop potentially seeds technological superiority to a rival nation that we can't currently trust. This problem, which they describe as over constrained and admirably face head on without flinching, creates profound challenges for US policymakers, especially as AI capabilities continue to advance rapidly toward potentially superhuman levels, as US China tension continues to escalate, and as the national security establishment begins to consider a Manhattan Project style effort intended not just to maintain technological superiority, but perhaps even to achieve enduring strategic dominance. Rather than taking a side in the debate regarding whether such a project is wise in the first place, Jeremie and Ed set out to understand what would actually be required to achieve the level of security needed to protect both the infrastructure and the secrets of such a project from our national adversaries. They do this not just by reading and theorizing, but by spending time on location at data centers and elsewhere with seasoned intelligence officials, special operators, and other technical experts. The resulting report forces us to confront the gritty realities of security, espionage, and geopolitical hardball that absolutely must be factored in to any serious discussion about the future of advanced AI. Drawing on their unique access, Jeremie and Ed paint a picture of a US critical infrastructure stack from the electrical grid to the data centers to the highly international research teams that is far more vulnerable to disruption and espionage than most realize. And they argue that while perfect security is practically impossible, an acceptable level of security for such a sensitive project requires a deterrent strategy that both raises the cost and observability of adversarial action and credibly threatens controlled but meaningful retaliation. While some will no doubt read this as a blueprint for victory in the AI race, others, myself included, read it as more of a warning. The sheer number of vulnerabilities we'd need to patch to effectively prevent China or other sophisticated actors from stealing our AI secrets, the compromises to core American values this would require to operationalize, and the risks from the centralization of power that all this would presumably entail are for me reason enough to steer away from such a national project and instead try whatever measures we can, however unlikely they may be to succeed, to create both the technology foundations and the geopolitical conditions necessary to begin to rebuild trust with China and the international community at large such that humanity can work together both to safely develop advanced AI systems and to manage the societal transitions that are likely to result from their deployment. Is that naive on my part? Maybe. But hopefully after listening to this conversation, you'll agree that it's not uninformed or closed minded. I am very much open to changing my mind. But for now, with the latest generation of intensively RL trained models showing so many of the bad behaviors that AI safety theorists have long warned about, and with no particularly low risk plan available anywhere, I continue to remind myself that we have far more in common with the Chinese than we do with the AIs. And I personally would choose to bet on our ability to find common ground with our fellow humans if it proves truly necessary over our ability to not only control but effectively steer the future with hastily built superintelligence. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends, rate a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Remember too that I'll be speaking at 3 major AI events over the next few months. May at Imagine AI Live in Las Vegas, August at the ADAPTTA Summit in Sao Paulo, Brazil, and September at the Enterprise Tech Leadership Summit again in Las Vegas. If you'll be attending any of these, I would be happy to meet in person. And if you're a business leader who's working on AI automations, application development, agent design, or high level AI strategy, I also invite you to get in touch. I'm currently advising 3 companies for just a few hours per month each, and I have found it both very fun and highly educational to help companies strengthen and confidently accelerate their AI initiatives. For any of these reasons or just to give us some feedback on the podcast, you're always welcome to reach out via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. With that, I hope you enjoy and take some time to properly consider the implications of this critical examination of America's vexing superintelligence dilemma with Jeremie and Edouard Harris of Gladstone AI.

Nathan Labenz: (5:50) Jeremie and Edouard Harris, founders of Gladstone AI and authors of the new report America's Superintelligence Project. Welcome to the Cognitive Revolution.

Jeremie Harris: (5:59) Oh my god. It's so cool to be in your in your in your podcast bubble. I've watching it.

Nathan Labenz: (6:06) Yeah. I'm excited to have you guys here. I've been really looking forward to this for a while. Fun little backstory. I guess, first of all, I listen to you, Jeremie, all the time on Last Week in AI, which I think is 1 of the best podcasts in the AI space. I am pretty much an every episode listener and a frequent recommender for folks that wanna keep up. I think you guys do a great job with that and really appreciate all your analysis there. You know, we've been trying to make this happen for a while. I think you're the only guest that I ever had booked and then got preempted by Joe Rogan, which, you know, millions have heard you on, Joe Rogan round about a year ago now. So I'm glad that we're finally circling back and making this happen, and timing is is perfect because the sort of, you know, manifesto mega plan, you know, treaties on what the hell are we gonna do about AI are coming fast and furious. And your latest contribution to that tradition, I think, a a really provocative 1, and I'm excited to get into it with you guys.

Edouard Harris: (7:10) It's it's fantastic, and we've been looking forward to it for as long as you have.

Jeremie Harris: (7:16) Actually, I I will say too, like, I've been a fan of the Cognitive Revolution for a long time. Like, it's it's sort of it it kind of I find it a little bit funny because when I watch what what you've been covering on the show and then I think about you watching Last Week in AI, I'm like, what what is Nathan getting from us? Just because the the quality is so high. Like you guys do great technical deep dives, which is something that's so so useful. So, anyway, yeah, I'm super thrilled to be in the podcast box.

Nathan Labenz: (7:40) Well, thank you. That's flattering, but I do really find a lot of value in Last Week in AI. Think your analysis is really good. And, I mean, nobody can really be comprehensive in the AI game anymore, so everybody's gotta take some pitches for sure. But I do think you guys choose a a lot of great stories. And sometimes I even just go back. If I if I do fall a little bit behind, I'll just go back and listen to a couple episodes and be like, alright. Which stories in there do I really need to make sure I I follow-up on? So to a significant degree, you're almost kind of like assignment editing for me at times where I'm like, yeah, that 1 is 1. That's key to the to double click on.

Edouard Harris: (8:13) So you can maybe you can maybe flag some, items for Nathan there.

Nathan Labenz: (8:17) Yeah. I need all the help I can get certainly in today's world. Alright. So let's start with this. There's a lot of talk about AGI. There's a lot of talk about superintelligence. Not a lot of talk about what superintelligence looks like. So I would love to start off with just getting a little bit of your intuition for what the hell is superintelligence? What do we think it can do? You know, with obviously some error bars, how soon should we expect it? And what kind of strategic advantage might this technology convey on 1 nominal superpower relative to another?

Edouard Harris: (9:00) Yeah. So on the question of what superintelligence is and what kind of advantage could it convey, I'll I'll try to tackle both of those at once. And then, Jerem, maybe you can talk through the timing and and fill in any gaps. In terms of what superintelligence is, right out the gate, people have all kinds of different definitions that are are slightly different from each other. I've heard 1 person say that a calculator is superintelligent in the narrow domain of, you know, doing prescribed math operations. And like, okay, but that's not necessarily a useful definition. The definition that most people have kind of coalesced around in the space is it's something that can do almost at least almost everything, if not everything, far, far better than the best human. And from that perspective, it's something that is that is going to do stuff and is going to likely have goals that are not comprehensible to us. And so from the standpoint of what kind of strategic advantage does a superintelligence, as opposed to like an AGI, which is more kind of human level, does a superintelligence give your country? It's actually, at least under current conditions, and our current ability to control and align these systems, it's not clear that it gives you much of an much of an advantage because it's gonna do its own thing if it's that much smarter than you, unless you are much further advanced than we are today in the science of how to actually control it.

Jeremie Harris: (10:34) Yeah. I think that's that's definitely 1 factor. And to go back to the definition piece too, there's interesting implication to this, like, universality of the superintelligence definition that people throw around. Right? Smarter than all or more capable than all humans at all things. Right? Something that absolute really implies a a process. Like, you can almost start from that, work your way backwards. It implies some kind of trajectory. If you have an AI system that is necessarily better than human beings at, you know, autonomous AI research, then that means necessarily it is the product, almost almost by definition of an intelligence explosion. Whether you measure that explosion in in years, right, because hardware is hard and the world is slow and wet and biological, or or in, you know, weeks or whatever that kind of explosion time horizon is, something that in retrospect everybody would agree was like a fucking hockey stick. Right? So that's really in a weird way, the the process itself, I think, is more helpful than the definition of the thing. The thing what like, if you have superintelligence, if you have something that is genuinely more intelligent than all humans at all things, you have something that could can design better biological weapons. You have something that can design, like, the ultimate cyber weapon. It is the national security technology. It is the decisive source of strategic advantage, full stop. That's just what it is. That's what is implied. Then you can ask about how we get there, whether it can be controlled, and so on. That that's the raw capability that comes with the package. The controllability piece, the alignment piece is an open question, and this starts to lead into our thinking strategically around what are the constraints that America faces today as it enters into what is ultimately and clearly a race with China on AI. You can't assume that you can control the system. Like, is that is an important thing. Level.

Edouard Harris: (12:18) At that level.

Nathan Labenz: (12:18) If you're, you know, if

Edouard Harris: (12:19) you're something closer to human level, even that that's kind of fraught. Right? Because even if it's qualitatively human level, it's maybe thinking 10 or a 100 times faster than a person. And it's like maybe can be deployed on a on a data center at bigger scale. So that could still give you qualitative differences. But still, if you have something that's like human ish level or AGI level, you can kind of imagine maybe having some kind of temporary slippery control over it and, like, having it really contain and aimed in a particular direction. If you're talking superintelligence, I mean, I think most useful definitions are this is a thing that outstrips you to the same extent that you outstrip, like, a toddler at the very minimum. And if you think about, like, how is a toddler gonna keep you contained over any meaningful span of time? Forget about it. You're just they're just not gonna even think in the right ways to do that.

Jeremie Harris: (13:11) Maybe this is a a good point to kind of, like, just jot down or or plant a flag on the constraints that we see that kind of inform our our picture of the world. It's almost a, yeah, like, the signposts that guide us as we go through this investigation and just the work that we've done over the last year. This really seems like a pretty overconstraint problem, just to be clear. So number 1, we have, AI superintelligence that seems, again, it's it's unclear. Right? Like, this is an open question, and we should dive into this part. It's unclear what it means to align a superintelligence. Can we do it? If you do have it, can you control it? Like, what, you know, what does that look like? But that's 1 fact of the matter about the world that we ought to take very seriously. Unfortunately, people who take that view seriously seem to have an allergic reaction to taking other points of view seriously as well, and the converse is also true. So just at the same time as it seems like it actually will be pretty hard to control these systems, and there's all that uncertainty. We know that striking a deal with China is going to be incredibly hard. And when I say incredibly hard, I mean when you talk to people, whether it's at the state department, whether it's in the intelligence agencies, people who've had firsthand experience dealing with China in a, like, real world setting, it is extremely clear to them that there is no deal to be done with China under current circumstances. Anything that involves a deal with China has to happen with a trust and verify framework in place. We don't have the hardware tooling to do that. We cannot do what we've done with some nuclear technology with some success. We just can't do that right now with China. So those 2 realities, like, simultaneously coexist in our world, And what we keep finding is you got folks who believe 1, take 1 seriously, and just refuse and have an allergic reaction to acknowledging the other and vice versa. And so our sort of philosophy here is how can we square the circle between these 2, what appear to us to be just facts of the matter about the way the way the world is?

Nathan Labenz: (15:08) Yeah. I think there's a lot there. A lot of different directions to unpack. I'm a big avoider generally of analogies because I always am trying to understand these crazy AI developments on their own terms as opposed to, you know, by proxy through some some other conceptual framework. But recently, I at least made the concession that to the degree people are gonna use analogies, it's it's often helpful to broaden the set of analogies that they'll consider. So, partially against my better judgment, 1 of the things I sort of hear you saying here is, like, the nuclear analogy is very often invoked, but maybe we should also be thinking about, like, a biological weapon analogy as well because those things are the ones that, you know, whether or not they can be properly said to have goals in the sense that we typically understand them, They certainly are unwieldy, and we, you know, have a whole other dimension of control problem that is layered on top of the conventional challenges that we have in achieving some stable equilibrium in the context of, like, a nuclear, you know, technology wave.

Edouard Harris: (16:16) Yeah. The biological analogy is is closer or has many more points in common with AI than the nuclear analogy exactly for the reason that that you've, you've given. And and for other reasons too, there is yeah. There is a control problem, at least with current, instantiations of biological weapons. There is chatter about, you know, adversaries trying to develop biological weapons that will target specific DNA profiles and specific, like, races and really, like, ultra, ultra scary stuff like that. But, yes, there there absolutely is a control problem that keeps everybody, like, kinda running scared on the biological weapon side of things. It also has the the same kind of potential for devastation, at least from our perspective as humans. Like, you actually in in principle, you could design a bioweapon that exterminates the entire population of the world. Like, that that is a thing that could happen. And then and then finally, the the other aspect of it is we actually have seen containment breaches of biological research labs, in the past. And not just talking about the elephant in the room type stuff with potentially COVID and that now likely being seen as a as an escape, but also other stuff, you know, in the 19 seventies and just research on kind of viruses, which doesn't even necessarily need to have come from biological weapons research, but can have come from just our attempts to understand these diseases to cure them. And so we know we know we can't control these biological agents, and they do escape containment. And biological agents are a lot stupider than the AIs that exist today and certainly the ones that we're going to build.

Jeremie Harris: (17:58) That's also a bit of a yeah. The the whole biological agents are are are stupider thing. It it's simultaneously true, and then it's 1 of those analogy breakdown moments. Right? Where the we we are ourselves are like, human organic matter is the mechanism of conveyance of that biological information the pathogen.

Edouard Harris: (18:17) We're particularly susceptible to the biological vector, and that is a difference. But mean, I would argue intelligence more than makes up for that at these scales.

Nathan Labenz: (18:31) Yeah. So let's flesh out a little bit more of this picture of the superintelligence. You know, in a sense, you're sort of saying, like, as something that is by postulate sort of better at humans and everything and sort of by, development is sort of the process of, like, a recursive intelligence explosion that's, like, a bunch of generations potentially compressed into a narrow frame of time, but still nevertheless, like, a bunch of sort of steps away from the AIs that we have today. It's like sort of just hard to say what that might look like. But I'd love to just, you know, spend a little bit more trying to develop intuitions or or share intuitions. I guess, you know, 1 thing that I think is like these things have really weird profiles. Right? The the jagged edge is just extremely jagged. I did 1 episode with Adam Gleave from Far AI who with, you know, collaborators at Far did work showing that even the superhuman Go players are, like, in fact, dramatically vulnerable to certain adversarial attacks. And I wonder, like, is it realistic to think that we're gonna have the AIs sort of solve all their own problems and, like, actually patch all those adversarial vulnerabilities? Or might we end up in a situation where in some ways these things are like super and and and by the way, this could be a good thing. Right? Like, it might be a great scenario if it's like 9 you know, every 98% of the time it works every time and that's enough to make all the, you know, drug discoveries we might ever want. But that 2% of the time is just maybe enough for us to never feel comfortable using it in warfare, you know, for example. Like, that could be a really fortunate natural path. But, I mean, do you do you see hope for that, you know, sort of luck, or do you see all these, like, loopholes being closed? Hey. We'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (20:39) Being an entrepreneur, I can say from personal experience, can be an intimidating and at times lonely experience. There are so many jobs to be done and often nobody to turn to when things go wrong. That's just 1 of many reasons that founders absolutely must choose their technology platforms carefully. Pick the right 1 and the technology can play important roles for you. Pick the wrong 1 and you might find yourself fighting fires alone. In the ecommerce space, of course, there's never been a better platform than Shopify. Shopify is the commerce platform behind millions of businesses around the world and 10% of all ecommerce in The United States. From household names like Mattel and Gymshark to brands just getting started. With hundreds of ready to use templates, Shopify helps you build a beautiful online store to match your brand's style, just as if you had your own design studio. With helpful AI tools that write product descriptions, page headlines, and even enhance your product photography, it's like you have your own content team. And with the ability to easily create email and social media campaigns, you can reach your customers wherever they're scrolling or strolling, just as if you had a full marketing department behind you. Best yet, Shopify is your commerce expert with world class expertise in everything from managing inventory international shipping, processing returns, and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into cha ching with Shopify on your side. Sign up for your 1 dollar per month trial and start selling today at shopify.com/cognitive. Visit shopify.com/cognitive. Once more, that's shopify.com/cognitive.

Edouard Harris: (22:35) Yeah. So that's an amazing question. 1 thing I'll I'll mention and point out here, you're saying the jagged edge is really jagged. That is totally true. What is often missed is that we humans are also super jagged, but because we're humans, we don't realize how jagged we are. So if you have a blind spot that's just like a structural blind spot in your brain and just hasn't come up in societal history or in in your own life, that stays a blind spot. The reason we perceive jaggedness in AI systems is that they're so alien and they think so differently from us. Right? So, like, the kinds of mistakes that they make are just different from the kinds of mistakes we make. And that means when we see an AI make a mistake, it makes the kind of mistake that lights up like a Christmas tree to my human eyes because my brain and architecture are just cognitively different. But the reverse is also true. I am making mistakes that are really, really obvious to an AI system. And the the more the more advanced the AIs are, the more they'll be able to see how to hack me and, like, all these just really obvious mistakes that I make, And that's true of all humans. So it's maybe somewhat less of a question of the frontier being jagged and more that every, you know, every high capability system is like jagged in different ways. And when you bring 2 different systems together, they can see each other's flaws, but they don't immediately see their own. I'll bump it off to Jer there actually because I think your main question was around, you know, is that good?

Jeremie Harris: (24:14) Well, yeah, I think so to to the jagginess point, I think there is also this question about how jagged is it. I I think humans are regularized by our interaction with our environment in ways that AI systems are. Right? We get data sources that are much more diverse in general. We're much more natively multimodal. We we work through, like, a a perverted kind of RL, which is kind of less prone to overfitting to begin with. So so there there's, like, a lot of what we're what we're doing that I think would argue for a less lumpy capability surface, but we still have those jagged edges. And the the rest of it I'm not so sure that that that in any way affects the analysis. It's more kind of a bit of a footnote.

Edouard Harris: (24:50) It's true. And we've been in competition with each other as well, and that adversarial pressure tends to smooth out jagged edges. That's totally true.

Jeremie Harris: (24:57) Yeah. Yeah. So, like, when it comes to the idea of, you know, the the sort of will superintelligence be like x y or z or what's the setup there and the connection to the the jaggedness? I mean, I'm not I'm not so sure that it matters. And and the reason I would say that is internally when we talk about the capabilities, the threat models that we're considering, we don't necessarily start with superintelligence and then work our way backwards. What we do is we say, hey. What are the specific capabilities that are tied to specific threat models? So for example, do you have if we're talking about superintelligence, loss of control, which is what we started off talking talking about, we we should get into weaponization as well. But for loss of control, you know, typically, you're looking at something where AI systems are automating a large fraction of AI research. At least that's 1 big exacerbator driver of loss of control risk. And there are footnotes and asterisks everywhere that we could dive into, but generally, that's what we think about. And so when when you wanna narrow down the conversation kind of to make it maybe constructive so we can get our arms around things, I think it can be helpful to think about, like, what is the first AI gonna look like that does recursive self improvement? Can it be, let's say, not jagged enough? Can it be smooth enough in the part of the capability surface that matters for driving that outcome such that you actually get something that can notice the jaggedness when it's getting in the way of pursuing real goals in the real world. I think any part of an AI system's optimization surface that isn't kind of optimized for or regularized for is gonna end up jagged because it's essentially the unoptimized parameter lag. You have just kind of these these hanging, like, grotesque parameter threads that haven't really been touched by contact with that sort of optimization pressure. So, I mean, I I think in terms of whether or not it's good, I think it's good for a lot of loss of control threat models because at the very least, you might hope that the thing, like, gets like you said, you know, it's like agents today. Right? They'll they'll have, like, a 99% chance of success on any given step, but you chain together a 100 steps steps for 1 coherent goal, and suddenly their success rate is, like, 20%. So you might just hope that the thing would get partway through, give us some clear indications that it's about to do something really bad, and we can kinda go, oh, damn. Like, let's stop that. That's no good. So definitely agree with that perspective for sure. I think it's a it's an open question, but to me, it's like about can you make the system that makes the system? And that's a way a more well kind of framed question that I could see you ironing out the jaggedness enough to kind of solve for.

Edouard Harris: (27:22) And there's some other aspects to that. Like, you know, once your system becomes capable enough, you kind of anticipate up to superintelligence, it starts to it starts to actively consider its own cognitive processes and, like, to some extent maybe start to fix its own flaws and problems. That's speculative, but there are indications that AI systems even today can at least consider, for example, the impact of how they're being trained on what their future versions will want. Anthropic had a famous experiment about this. So the idea of AI systems considering their own flaws or needs, if I can use that anthropomorphic, you know, language, that's a thing that is happening, at least early versions today.

Nathan Labenz: (28:10) Yeah. I'm a frequent, nag, I think, when it comes to reminding people that we are really still just in the early innings of uncovering the bad behaviors that the latest AIs are starting to demonstrate. I had a episode with Ryan Greenblatt talking about the alignment faking, and then haven't done yet 1 yet, but Apollo research also put out some really interesting work. There's a way showing a pretty significant step change in situational awareness where the latest Claude model is starting to say things like, this feels like an evaluation. You know? So there's like the the number of sort of levels at which these things are thinking when they are ultimately just deciding, you know, am I gonna do this, you know, nominally harmful task or not? It's like, well, the user asked me to, but I was trained not to, but they're telling me I might be trained differently. So maybe I should do it in this case, but wait, maybe actually I'm being tested for exactly that. So maybe I should go back. It is we're definitely like entering a hall of mirrors and it's getting pretty wild. What is it 1 beat on multimodality? Because you mentioned how we are more natively multimodal than current certainly than current AIs are.

Jeremie Harris: (29:21) I do take that back for what it's worth, but more yeah. Anyway, something we can dive into. I think you can arguably you can arguably train AI models on more modalities obviously than humans, but anyway, we

Nathan Labenz: (29:33) Yeah. Well, that's exactly what I wanted to ask. I mean, so we've seen recently this and this is my vision of an early superintelligence at least. I mean, I I think there's sort of 1 vision is like, you sort of put o 1 in a pressure cooker and it becomes o 3 and you put o 3 in a pressure cooker and it becomes like I think at that point, it jumps to o 7 or something. And then, you know, it just gets so good at this, like, super long chain of thought reasoning that it can sort of reason its way out of any problem and it's kind of like a you know, it starts to look a little bit like maybe the earlier Elijah type visions of like, you know, if this thing is just the perfect rational Bayesian and it's got, you know, these perfect fundamentals, then, you know, you can infer the whole structure of the universe from like 1, you know, picture of a plant and the way that, you know, gravity appears to be bending the leaf. Right? And you you're just extremely sample efficient, and you're you're just, like, extremely lean and mean. And, you know, that was a scary vision to me that I heard a number of years ago, and it would still be scary, obviously, if it if it were to come to that. Notably, the current AIs are a lot softer and flabbier than all that. And so my what seems a little more likely to me, and I wanna get your take on it, is, like, take GPT-four o image generation where there's clearly been a step change in the, let's say, depth of integration between the text and and image understanding such that now it's no longer like this arm's length thing where the the 1 model has to say to the other, like, make me an image of, you know, all this text. Instead, that's integrated in the latent space and the same model is able to kind of work it from both sides. If I had to guess what a superintelligence first superintelligence we're likely to see would look like, it would be basically that times, like, 10 to 20 more modalities such that the AI has this sort of intuitive physics in a lot of different spaces. And by intuitive physics, just mean like, if I throw you a ball, you don't have to calculate, you know, in sort of intensive, you know, molecular, dynamic space, like every molecule in the air that the ball is displacing as it comes to you to catch it. You just kinda know there's where it's gonna be, and you can just catch it. We've seen a lot of examples of this now from different modalities like material science and protein folding and whatever, where a proper simulation takes a lot of compute. A model trained on those, even sometimes just trained on pure simulation data can do the same thing at like orders of magnitude faster. And so again though, I have some hope maybe that like if we do that, maybe we still end up in this regime where it's, like, superhuman enough to solve lots of problems, but that thing might be, like, pretty unwieldy. And, you know, if if you heard the Jeff Dean on Dourkech conversation, like, could require, like, an unbelievable amount of your RAM for all the experts, you know, to be sort of splayed out across the Pathways architecture and, you know, potentially multiple data centers or whatever. And maybe it's still kind of, like, flabby enough that and and I'll just, like, big enough footprint that, you know, in some ways, it becomes, like, easier to control because it's, you know, god knows how many trillion parameters, like, not super easy to, egress off the network. Anyway Yeah. It's a lot there. How do you would you react to my, superintelligence my flabby superintelligence vision? The

Edouard Harris: (33:09) the funny thing is you're describing like, the baby version of what you're describing is Gato. Right? Like, this is a a model like, it's an old model, like, 2021, I think. Maybe '22

Nathan Labenz: (33:20) Gato 2 is 1 of my, big questions.

Edouard Harris: (33:22) Yeah. It was like when I when I first read that paper. Sorry?

Jeremie Harris: (33:25) No. I was saying he's asking what happened to Gato 2. I just said Gato 3, but that

Nathan Labenz: (33:30) was true.

Edouard Harris: (33:31) Gato 3 Gato 3 ate it. That's what happened to Gato 2. Yeah. When I I I first read the the Gato original 1, Gato 1 paper, you know, '21, '22, and I was like, oh, yeah. This is just the future because it is just like this. You have 1 model. It's like 1000000000 parameters. It was very, very small by today's standards, but, like, you know, pretty hefty ish by by even those standards. And for a multimodal 1, it integrated all of the modalities that they could basically get their hands on. It was, you know, it was looking at images and describing them. It was it was chatting on text, and it was even manipulating robot arms pretty successfully to, you know, move blocks on top of each other. And, it was all done through a single kind of residual stream ish thing or a single kind of encoding where all of those modalities were mapped into the same, like, the same stream, the same token set. So this is absolutely something we know how to do. Yeah. Maybe the future version of that is you have experts that are kind of sub trained across those modalities. But I think that 1 of the trends that we see is that you you start with, like, an exploded GPT 4.5 thing, then you distill it down, and it doesn't, you know, quite have all the world knowledge, but it is capable. It's faster. It's cheaper, and it runs, and it can be used as the basis for the next gigantically splayed thing. So I think it's a step along the line. Absolutely. But we're gonna see more of more of that multimodality and I think across, yeah, things like protein folding, things all of these different areas.

Nathan Labenz: (35:06) Hey. We'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (35:11) It is an interesting time for business. Tariff and trade policies are dynamic, supply chains squeezed, and cash flow tighter than ever. If your business can't adapt in real time, you are in a world of hurt. You need total visibility from global shipments to tariff impacts to real time cash flow, and that's NetSuite by Oracle, your AI powered business management suite trusted by over 42,000 businesses. NetSuite is the number 1 cloud ERP for many reasons. It brings accounting, financial management, inventory, and HR all together into 1 suite. That gives you 1 source of truth, giving you visibility and the control you need to make quick decisions. And with real time forecasting, you're peering into the future with actionable data. Plus with AI embedded throughout, you can automate a lot of those everyday tasks, letting your teams stay strategic. NetSuite helps you know what's stuck, what it's costing you, and how to pivot fast. Because in the AI era, there is nothing more important than speed of execution. It's 1 system, giving you full control and the ability to tame the chaos. That is NetSuite by Oracle. If your revenues are at least in the 7 figures, download the free ebook, Navigating Global Trade, 3 Insights for Leaders at netsuite.com/cognitive. That's netsuite.com/cognitive.

Nathan Labenz: (36:34) In business, they say you can have better, cheaper, or faster, but you only get to pick 2. But what if you could have all 3 at the same time? That's exactly what Cohere, Thomson Reuters, and specialized bikes have since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. And better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is the cloud built for AI and all of your biggest workloads. Right now, with 0 commitment, try OCI for free. Head to oracle.com/cognitive. That's oracle.com/cognitive.

Jeremie Harris: (37:45) Yeah. I I think, like, 1 of the things that'll probably shape this, I would guess, in in a fairly significant way. So 1 1 thing that Gato showed and that we've certainly seen since is, like, you know, positive transfer and the power of positive transfer with scale. Right? So, you know, by that meaning, you you you train on 1 modality, and then what you learn from that actually leads to, like, lower loss when you combine it with learning from a new modality. Typically, obviously, you have some some, like, catastrophic forgetting type effects or some sort of, like, model capacity issues that that cause you to not be able to perform as well on, you know, say the the language modeling task, once you move on to images. But now we're seeing that positive transfer. So that's certainly an argument in the direction of where you're gesturing. I guess another kind of on the other side of the coin, I would expect a power law roughly in terms of, like, the the value that new modalities add. And we're always asking ourselves, I think, if we're OpenAI, we're if we're, you know, whatever these labs are, like, what is on my critical path to to recursive self improvement? Right? That that is you know, they see that as the target. So there there comes a point on that curve where I'm I'm no longer interested in adding, you know, all the MRI data or I'm not interested in adding certain kinds of, like, infrared data or whatever the modality may be just because the the skills that I really care about look maybe more like autonomous coding. And and there, I I'd rather spend that marginal compute on inference to allow the thing to kind do the the RSI thing or on RL loop or whatever it is. So I think ultimately it always boils down to how do you spend your compute. I think the the the story now is like the ways to spend your compute just keep like, there's an ever growing list. And I think that there's a real kind of a la carte thing going on within labs where 1 of the ways in which maybe they diverge is we're seeing them choose to trade off inference time compute versus pretraining compute a little bit differently within inference time compute. Right? Not all inference time compute is fungible. There is a great kind of bit of research that Epic AI put out a couple months ago that went into this. It's like, what kind of what kind of inference time compute compounds rather than sort of like is getting at the same core things. So if you think about, like, your model distillation versus pruning, both are about trying to, like, squeeze more into less. So so those are you know, you're not gonna be able to stack your inference time budget. They're they're a little bit more fungible, whereas you have relatively independent axes of inference time compute elsewhere. So Interesting. Lesser.

Edouard Harris: (40:01) Yeah. So what you're saying is basically that when it comes to the labs themselves, and I think I agree with this, where there's increasingly gonna be a competitive pressure that's just kind of forcing them to go, well, you know, what is it that actually gets us to RSI? Because that's just kind of how you have to go. But then you you would kind of imagine that

Nathan Labenz: (40:21) Yeah.

Edouard Harris: (40:22) I don't think that's

Jeremie Harris: (40:22) how you have to go, but I think that's well Hold on. No. I mean, it's like more than anyone. Yeah.

Edouard Harris: (40:28) Yeah. What I'm talking about is, like, when you have 1 lab that is is doing that, it creates a strong pressure for the others to do it. So you have this, like, logic of the race type thing where, well, to keep up, I I have to kind of raise my bar along that particular line. But then what what we would expect then is the other modalities to be kind of maybe tacked on after the fact by by fast followers, folks who take, you know, open source models that are maybe a generation behind and maybe fine tuning them on on other scientific data, which is a really interesting picture.

Nathan Labenz: (41:02) So you guys have done some interesting, like, what's the right, word? The sociological work, exploring the culture of AI development in Silicon Valley. And I would love to hear a little bit about how you understand and we could go, like, maybe lab by lab on this if you have that kind of resolution. What is their goal? And, you know, to what degree is it really a sort of monomaniacal focus on getting to this point where the AIs can take over the AI research, and then we sort of let what, you know, come what may from that point. Like, increasingly, you know, we just have obviously seen AI 20 27 from Daniel Cocatello put out, and I haven't spoken to him about that. I did have a chance to play his war game, you know, that sort of basically is the scenario that that document describes. But I I can't speak for him in terms of, like, exactly what he's trying to communicate. But, like, 1 way that I read that manuscript is as a warning that this is what OpenAI is trying to do. Like Yep. They are Yep. You know, they have their sights set on this recursive self improvement dynamic. And I don't know why if that's, you know, what he thinks and what he's trying to communicate, why we're not a little bit more matter of fact and declarative and saying, like, this is what they are trying to do, and you should be afraid of it. But what have you guys I mean, you've you've done a lot of this, like, on the ground hanging out at the infamous Silicon Valley parties and all that good stuff. Like, how would you describe what they and and perhaps other leading developers are actually envisioning themselves doing to get to a superintelligence?

Edouard Harris: (43:01) Yeah. I mean, without without directly speaking to, you know, Dan's opinions or his coauthors opinions, Yeah. The I mean, yes, in terms of what the game plan is at, at many of these labs, some of them are more direct and forthright than others. Some of them are more gung ho. Some of them are less less so, less enthusiastic than others. But the idea of like we're gonna get to a point where AI is gonna just do our homework is kind of like especially if you've been following the curve, it's like it's the obvious kind of default path. Back when Jan Laika was at OpenAI, that was that was his, like, his take when it came to the alignment side of things. And, I mean, you know, it's like it's not clear how good of of odds of success of that are, but it's obviously not a crazy approach under current conditions, and he's trying the same thing over at Anthropic. Yeah. I mean, yes. People have absolutely told us that without, like, singling out OpenAI because it's again, it's like the the obvious thing you do if you're kinda yolo ing this. Yeah. OpenAI is considering this, and the path to superintelligence is faster if your researchers operate at machine time. And from the leadership perspective, it's also so much better. Right? You have, like, you hope perfectly loyal AI employees who will not leak your stuff at parties or in in conversations at bars and who will work faster and work work at night. But, obviously, you know, then the question is actual control, actual alignment. And I I think that AI 2027, 1 thing they did really well was conveying the kind of emotion of like, oh, this beast is like starting to take on a life of its own. And like, I can understand as the researcher working on supervising this thing that these are the last few months where my input and my work is actually gonna matter at all to the outcome, so I'm gonna burn the midnight oil. It's evocative, and it could could be how things end up going.

Jeremie Harris: (45:21) Yeah. It's I was a really big fan of the AI 20 27 framing just because I I do think it's a it's great to have the story that's concrete out there, if only because people can then criticize it. There are specific things that not a huge number, but there are specific things that we found that we would have written differently, let's say, in in the the document, but that's the the virtue of it is you can actually point those out. Whereas 1 of the the really interesting things is just to tie back into the cultural point. When you look at how and I will single out Sam A on this because he's, I I think, by far the worst offender on this. Like, the the vagueness with which he describes what superintelligence actually is, what it means to get there, and what he will do with it is an asset to him. It is only an asset to him. It has always been an asset to him, and he has used it as an asset in recruitment. Just the same way that they've used, for example, the, you famous nonprofit's control over the for profit entity, that's I I think I mean, any reasonable person would say that was used for recruitment purposes. They're now, you know, an amicus brief out there saying from from all these researchers sharing that view. Also pretty much in the email chain that, you

Edouard Harris: (46:29) know, that was that was made public and yeah. Yeah. No. I mean,

Jeremie Harris: (46:32) this is, like, not even open secret. This is just known stuff. Right? So I think when you think about the kinds of people who work at the different labs, they are different. There's a selection effect for sure. If you the the first thing I'll say is having talked to a lot of current and former people at OpenAI during this investigation, you see you see a very sharp contrast between what the the researchers themselves say and believe and are concerned about and then what the executive says and and seems to be concerned about publicly. Those things do not track. As a simple point of fact, they are at odds. That's not something you see at Anthropic. What you get when you talk to somebody at OpenAI or at least, as we did during the the investigation is you'll get people who are very nervous, very keen to tell you, please don't tell anybody that we spoke. Please don't communicate. Like, it's it's really, really, like, a a tight kind of leash that people feel themselves to to be on. You talk to people at Anthropic, that story is completely different. So you get the sense that, yes, you're talking to people who may maybe sometimes have disagreements with the leadership, for sure. But you have the sense that they would be comfortable voicing those disagreements and in fact that they have. And that's a a kind of a really key piece of almost cultural bedrock that I think is is most clearly like, you see that divide between those 2 companies in a pretty big way. And, obviously, OpenAI's character has changed in fundamental ways. There's no getting around it. You know, they used to be a lab much more focused on the sort of at least overtly focused on on the, risks of 1 person having too much power. Right? That's a big theme. Like, I'm old enough to remember when Sam Altman was not supposed to unilaterally be able to, like, jujitsu his way into kicking off an entire board, having friendlies basically come in and sort of rewriting the script on that. And you just keep seeing those goalposts get shifted and shifted and shifted. And so I think, you know, 1 of the the the challenging plays has been, from our perspective, trying to get a a very, like, objective clean sense of where all the different labs stack. You you know, we want American champions to be at the head of the game here. That that's necessary. But at the same time, you need to think about, like, okay. Who can you actually rely on to make the right calls for a variety of reasons? And frankly, at this point, given the industry incentives, we're not seeing anyone invest in security in particular nearly in the way that they need to. I mean, these these labs are CCP penetrated. Like, there's just no way in hell that they're not. And if there's 1 thing that the investigation, like, made extraordinarily clear, it's that that is a wildly pressing problem that is not being taken seriously enough right now. That needs to change in a really big way. But there are cultural factors too. There's all you know, obviously, the like, whatever version of transhumanist so many of these researchers are and effective accelerationism and all that stuff, like, there certainly is an influence of those things. I think a lot of folks at these labs, especially OpenAI, do kind of seem to have blinders on. The ambition of their kind of their their line of sight seems to be a lot shorter than the ambition of the lab's line of sight. The lab is saying we're making superintelligence, and it really seems when you're talking to these folks, they're focused on, like, the next beat. Okay. How do I solve this narrow problem? Yeah. Yeah. Yeah. We'll deal with the, like, the security, the alignment, the all all that shit. We'll deal with that down the line. That is its own kind of problem, and it's only true in in some cases. It's not universal,

Edouard Harris: (49:49) but And not that there's not that they've done 0 in security relative to, like, when we Right. So we we published this that state department backed report about a year ago now. The situation was truly catastrophically bad at that time. It is still really bad. It's nowhere close to where it needs to be from a security standpoint. We know that. But they've made some some motion in in the direction of progress there that we should just we should note even though it's, like, it's not even close to be enough.

Nathan Labenz: (50:18) So would you describe this whole project as ideological? I mean, we have heard similar things from Dario with respect to, you know, their leaked fundraising deck, which said, like, the, you know, leading companies in 2025, 2026 might get so far ahead, nobody can ever catch up. More, colorfully, I would say in his more recent and public writings saying that, you know, we can get this edge on China and then we can sort of use our edge to build an ever bigger edge and, you know, whatever will somehow make them an offer they can't refuse. I'm not sure if I should understand it as almost religious or what exactly, but it does I mean, I I just made a small donation to a project called building god, which is a documentary that's trying to, you know, trying to bring more of, this to light. But I I, you know, support that because I genuinely don't know. Like, are these people trying to build their own new god? Is that an overstatement, or is that, like, an actually apt description of the culture and the ideology that animates it?

Jeremie Harris: (51:26) You you mean did Ilya lead prayer sessions at OpenAI about with the, feel the AGI mantra? Or yeah. Yeah. No. I mean, it's it's a thing like I think objectively, they are trying to build superintelligence. Right? Or or at the very least AGI. And and we should get into into that calculation too for strategic reasons, but that's the goal. Whether you view that as a spiritual imperative I mean, certainly they've said this. Right? Like, I think Miramaradi said, you'd like use the the word spiritual or something like that in kind of describing what motivates people to come to work, and and that's absolutely the case when you talk to a lot of these people. With others, it's not. But I think that's they believe they are building the hyper object at the end of the universe type thing. Right? Or well, anyway.

Edouard Harris: (52:10) The labs also are not monoliths. Right? Like the different people have different beliefs and stuff. So take OpenAI as an example. They grew really fast, and so some people there are in it mostly for the money. This is an incredibly fast growing company. There's a story around how it goes vertical and goes to infinity. And so if you're thinking of things in terms of money or in terms of just economic value or value to yourself maybe in a more abstract, you wanna get on that rocket ship and that's 1 kind of pathway. Another 1 is absolutely the transhumanist kind of ethos or ethic and the idea that there's kind of this destiny aspect to it that we are going to upload our consciousnesses or transcend through this object. And there's a number of ideologies kind of all mixed together. But yeah, certainly from a secular perspective, the messaging and a lot of the internal conversations are around, like, the very least, this is history making. This is an inflection point in the arc of history unlike any other that has come in the past. It's beyond the wheel. It's beyond fire. It's tantamount to the inception of the human species and its impact on the earth and the universe only more so.

Jeremie Harris: (53:39) I do wanna touch on the the Dario China thing though too because I think that is distinct from the sort of, like, you know, religious zeal that I think some people in the space are and I'm pseudo religious zeal, whatever. Like, you you know what I'm getting at. That that some people are are motivated by. I think Dario, I mean, frankly, it's difficult if you if you take the view that that China is a competent adversary in the space, if you take the view that they're committed adversary, that doing business with them is is not going to work on the timelines that Dario seems sees as plausible. So if you think about the 2027 or, you know, anything like that, then then suddenly you are forced into some pretty tough choices. I mean, we said earlier this feels like an over constrained problem. This is really where that comes from. Right? I mean, you have this tension. And and and so I think that's why you're seeing Dario move in that direction. I I think increasingly, it's just become obvious to a lot of the Frontier Lab COs that genuinely, the situation with China is very dire and requires us to forge ahead. And it means that you are playing chicken with a cliff. And the question is, like, can you can you turn your car into an airplane before you you you hit the the lip of the cliff? What am I even saying? But, yeah, I mean, that that sort of I I think that there's a very reasonable reading. I'm I'm quite sympathetic actually to Dario's argument there, And that's unfortunately part of what resurfaced in the the work that we did was just, like, the extent to which China holds us at risk, holds our infrastructure at risk. And, really, the our option set is a lot more narrow than I think a lot of people in the kind of AI safety ecosystem tend to think.

Edouard Harris: (55:13) Yeah. And this is this actually comes back, Jared, to what you were saying earlier, which is like there is a fundamental dilemma here. Right? Like and the dilemma is kind of realized or exemplified by these 2 different camps that are not talking to each other and are not hearing each other to the extent that they need to to resolve the dilemma. 1 is, yeah, the the AI safety ecosystem kind of broadly construed, which is saying like, look. These systems may not be controllable or alignable. And and some folks are like, no. They're absolutely at superintelligence scale. What are you even talking about? Like, we're just not gonna be able to do that. And, like, that's potentially a reasonable a very reasonable view. And so, therefore, this is a coordination problem, and the only thing we can do is we have to everyone has to slow down to give us time to solve the alignment thing, or these systems are basically just gonna, like, rule us and and probably kill us because of all of these these kind of instrumental convergence arguments and stuff like that. That's the 1 camp. Right? It's a coordination problem. We can't solve alignment, so we need to slow down. The other camp is the national security folks. They're starting from the position of, look. I've seen how China operates. I've seen how they leverage international agreements to constrain us while they have a free hand and basically, like, break whatever agreements they want. I've seen how they treat members of their own population. And, like, some of the stories, like, they, you know, they they put people onto buses to to just, like, go off to be executed. It's not like an incredibly common thing, but absolutely, there's, like, a there's a treadmill of that going on over there. That's, like, a just a thing that they do. Right? Like, no due process. The the the view is, like, we would have to be crazy to make a deal with these folks and that government and have any assumption whatsoever that they're going to keep to it. Because a deal is impossible, we have to win, therefore, we have to race.

Jeremie Harris: (57:22) And and 1 says, yeah. And implicitly, we're either we either gotta hope the loss of control problem isn't a thing or or that we're gonna solve it in flight. Right? Like, that's the implied premise there. Like, I I I'm not looking at that problem versus I'm not looking at that problem. And, yeah, anyway, that's the gap.

Edouard Harris: (57:39) Bingo. And and the safety side is is looking at the same thing on on the China side where it's like, yeah. You know, we have to slow down, and therefore, I'm I'm just gonna ignore or or I'm gonna minimize the arguments that say, like, no. All of the evidence shows that this government will do whatever it takes to get ahead. Like, they will lie. They will, like, use criminal surrogates. They will, like, pull every possible lever to do what they wanna do. And those 2 sides are not listening to each other, and that's where the argument comes in. But that's it's it's fundamentally a hard problem. It's it's a dilemma, and that's the that's the dilemma that we are trying to make some steps towards resolving in this report.

Nathan Labenz: (58:27) Before we get into the findings of the report, how maybe this is because it's sort of starting to get there, I suppose. But how AGI pilled, generally speaking, do you guys feel the US government is today? I mean, I'm not even sure that's a coherent question because we have obviously a very stable genius, who I sometimes call he who must always be named now as, it seems like more and more of these, you know, AI documents are being, you know, written with an audience of 1 in mind. And I'm not sure there's any, like, coherent center to it, but I guess it's, you know, multi part question is, like, how AGI pill do you think the US government is broadly? How subject

Edouard Harris: (59:09) to

Nathan Labenz: (59:09) the whims of 1 individual is that degree of AGI pilledness? How much does it matter if the guy at the top sort of says 1 thing or says another? And then how would you compare that to China? Like, how AGI pilled is the Chinese government today?

Jeremie Harris: (59:24) Yeah. We're we're limited in in in what we can say necessarily on on the US government side with respect to the particular people that we've spoken to. I mean, what what's known publicly, right, is, you know, Trump has come out and said, hey. There, you know, there are things about AI that look really exciting. It could be transformational. He's talked about that a lot. He's also alluded to the risks too. There were some podcasts that he did, I think, 1 with, 1 of the Paul brothers. I can never remember which one's which, but, anyway, we're just talking about it with us. Yeah. Yeah. The and he made

Nathan Labenz: (59:51) a deep faked him, I think. Right?

Jeremie Harris: (59:53) Yeah. That's right. Yeah. And and, actually, funnily enough, I was

Nathan Labenz: (59:56) Somebody was selling some bullshit online with his likeness. Can you imagine?

Jeremie Harris: (1:00:00) Well, I think I think he was also, like it it wasn't a comment about the deepfake thing. It was a comment more generally, which I think, you know, you can read into, any number of ways. Obviously, JD Vansky gave that speech in Paris, which I think is, sort of the most concrete thing apart from Kratios' speech that recently happened, where we're getting a bit of a sense for the the positioning there, which does reference at the end. Like, it it, you know, says we we can't be hand wringing about safety. And by the way, this is, like, this is in a context where he sees himself as so many do as in a race with China. Right? This that is that's what it means to buy that argument. The flip side at the end, he does say, you know, acknowledging that there are real safety or security risks or whatever with the technology or the or real risks, do exist. And so there there is an awareness that there are real risks here. There are, you know, people, as you might imagine, in various national security context within the government who are tracking this and and are quite aware of the the full range of possibilities. But what to do about it is, like, we've seen congress wrestle with tech. We've seen the executive wrestle with all these things for for, you know, administration after administration now. I I think it's it's still very much being formed. Like, people are figuring out what they think, both about the prospects of the technology, the risks of developing it from a loss of control standpoint, and then as well the risks of losing the race to China. All of that right now is still very much up in the air.

Edouard Harris: (1:01:20) Yeah. And that's like that's exactly right in terms of the administration side of things. More broadly in the US government, you kind of alluded to, like, you know, the the coherence of the question, well, the reality obviously is the US government is kinda defined by how huge it is, and everybody has their their, you know, 3,000,000 people or or however many million people with that many millions of opinions. But certainly, many, many more people in the government are AGI pilled than they were, like, a year and a half ago. A year and a half ago, there would be, like, the odd individual person who was, like, kinda getting there, but not sure about their colleagues and so kinda didn't speak up. And so you would talk to them and be like, yeah. Yeah. So actually, like, I kinda see, like, where your head's at, but, like, I I don't like, nobody else, like, da da da. Whereas now it's like you have entire offices that are AGI build and, like, talking openly about it. 1 other thing that's, like, actually, like, an interesting data point. Every everyone or almost everyone whose work touches on this area even obliquely knows about or has read situational awareness and, like, kind of absorbed the high level points. This is actually a a really interesting illustration of, like, how, you know, memes travel across channels. Situational awareness wasn't really reported much in the traditional media, which is like, you know, Leopold Ashenberger didn't write a Wall Street Journal op ed, for example. And yet it traveled, and it's in the water in those spaces. And make no mistake, it's in the water in China too. There were translations in Chinese of that manifesto circulating over there Not long after the original was published here, that awareness is is present. And when you see the CEO of DeepSeek, like, sitting with bullet bureau members in in this, like, in this context where it's like, you guys are the captains of industry that are fighting the fight for us. And and the fact that the Chinese have have invested now, have announced this, you know, quarter trillion dollar investment package into AI infrastructure, we're starting to slip into that zone and that space for sure.

Jeremie Harris: (1:03:50) As Ed was quick to point out when that first happened too, that's a quarter trillion dollars in in PPP terms. Right? Because a lot of the early reporting got this wrong and was saying, like

Edouard Harris: (1:04:00) Yeah.

Jeremie Harris: (1:04:00) You know, 100 I think, you know, 137,000,000,000 was, you know, semi analysis is kind of, like, take home, which is incorrect. It is 200 and fifth or it's it's over a quarter trillion dollars. Yeah, it's still in that.

Edouard Harris: (1:04:12) They're building all this stuff themselves, so it's like how much can you procure in country?

Jeremie Harris: (1:04:20) Increasingly so, right? I mean like it's obviously there's like, you're you're taking those those age twenties as well. It's a balance. But but it yeah. It's it's a it's a, like, a really big issue. I think another piece to this too is, like, like, we we had some intense conversations early on when we were doing some of our work a couple years ago with people who are like, well, you shouldn't be talking to people in the US government saying using the term AGI. And the there's this, like, kind of concern, this sort of, like, nervous anxiety in in the community of people who worry about AI policy, which I both understand and think is is very counterproductive. And and we've actually seen the consequences of that anxiety play out, both culturally because it's profoundly off putting to people who are builders, and this administration is an administration of people who value the building culture and vibe much more than the kind of regulatory sort of cautious culture and vibe. That that's a huge mistake. That that is a a thing that also anybody could have seen coming. Like, you're gonna get a Republican administration that is sort of more, in a sense, libertarian leaning, at least more kind of pro accelerationist. And and this idea that you're somehow going to prevent people from talking about AGI is, I think, 1 of the most dangerous things that this movement has has sort of done. 1 of the consequences at the time, what we were saying was, look. You have before you a unique opportunity to be 1 of the only groups of people who is speaking to this issue before it comes into the mainstream. People haven't made up their minds yet. This is your time to inform people about the stuff that will become politicized. Right? Like, again, like we just talked about this polarization around, well, are you either a China hawk or are you concerned about loss of control of these systems? It's fucking stupid that those 2 things are, like, somehow anticorrelated. Like, they have nothing to do with each other. Right? But we live in that world because that's just how people have made up their minds. We've lost the opportunity to shape a lot of that landscape because people were hand wringing about can we even have conversations about this, which I think was just a profound self own and and fortunately something that we didn't heed, though we we thought about it very deeply at the time, obviously, because so many people were telling us, you know, don't talk about the thing. I think it's it's pretty self evident now that had more attention been paid to that in certain kind of tactical ways, I think that the landscape would look quite different. But it yeah. It's it's now kinda become that that China hawk versus loss of control bit, which is just it's inaccurate and it it's pretty unhelpful.

Edouard Harris: (1:06:44) You and you see the I mean, you can you can sympathize with the view of, like, every, you know, every incremental thing you do to accelerate or to, you know, broaden the concept is a thing that you can't take back. Like, now everybody's gonna be thinking about like what a big weapon AGI is, what a big weapon superintelligence is and stuff. The reality is, like, you're in you're in the space where, even even absent the geopolitics, there's, you know, there's, like, 4 or 5 different, like, superintelligence projects, depending on how you count, going on, right, in in America with with all the frontier labs. And there were always going to be.

Jeremie Harris: (1:07:23) Right? And there were always going to be people who are gonna publish whatever manifesto about these obvious strategic landscape. Like, at least this seemed like a very likely outcome. But sorry. I just like, I think that's kind of part of the context is, like, this will happen eventually. People will talk about AGI. But yeah.

Edouard Harris: (1:07:39) Yeah. You end up in this, like in any case, you're you're kind of in the the the mollock, if you like, part of the scenario where the race like, now just zooming in to the corporate race between the the American labs, forgetting China, even just that race has a life of its own. Right? It has its own logic. The the CEOs of the labs, like, have relatively little control and little agency over the dynamics of the entire thing. Like, if Sam a goes, like, oh, shit. Actually, this is, like, super fucking dangerous. Let's, like, bow out and, like, shut down OpenAI or whatever. Well, that doesn't actually change anything. Now you've you've got, like, x AI, you've got Anthropic, and you've got, like, maybe meta and a couple others, like, still trying to race ahead. And so at the end of the day, it's like you're the you pass a point where, like, little tactical interventions on on individual, like, pieces, those matter when there's, like, 1 or 2 or maybe 3 players. But you get to a point where, like, beyond a certain point, you have to like, you may have to execute an intervention, and you need the intervention to be based and grounded in an understanding of what is being intervened on.

Nathan Labenz: (1:08:53) Yeah. I mean, I might have a little bit more sympathy for a couple things than I think you guys I mean, broadly, I think that's it's a hard analysis to avoid. But if there were a couple things that I would push back on a little bit, 1 would be that I think there is some interaction between The US China dynamic and, like, how likely we are to lose control of AIs if only because if we get really reckless in racing to superintelligence, then that seems like we're more likely to lose control. MSO 1 hunting, I think, is like

Jeremie Harris: (1:09:25) Just to just to clarify on that point when I was saying they're independent, what I mean is the conclusion, like, China is a real player in this race. They're an adversary. Logically, getting to that conclusion is independent from getting to the conclusion, oh, AI is potentially, like, loss of control is a real threat factor. The like, those 2 thoughts should be able to live simultaneously in the same head. What you do about it, that's where you get into the contradictions for sure.

Nathan Labenz: (1:09:47) Yeah. I mean, there's a lot of motivated reasoning going on for sure in in a lot of places, and I'm probably guilty of at least some of it myself.

Jeremie Harris: (1:09:54) So are we

Nathan Labenz: (1:09:54) Nah. None of us are guilty of it. Yeah. Present company excluded. The other thing that I I do hold out maybe a little bit more hope for is, like, if Sam Altman you know, it's a hard, you know, hypothetical to really imagine, but let's say that he did have a sort of, you know, awakening type moment and was like, I'm leading a bad dynamic. And he actually did what you said and, you know, genuinely bowed out or, like, made, you know, some costly signal style commitment to a different trajectory. That could be, like, a pretty powerful signal that could perhaps create a cascade. Like, I'm a big believer in, like, multiple eagle eyptia, and I think we are in 1. That doesn't preclude in my mind the possibility of others existing. Like, how to switch to them is hard, but maybe what it takes is, like, 1 or 2 brave and foresighted individuals to say, I'm trying to move to this other equilibrium. Like, who's with me? And, you know, maybe the rest could fall in line. I don't know. It doesn't seem so Yeah.

Edouard Harris: (1:11:00) Fancy. There are there are paths, yeah, there are paths that look like that. Right? For sure. And and so if it was just like Sam a bowing out, that would be a big deal. Be news around the world. Be all kinds of stuff. I don't know that that would that would on its own, like, affect the race. But if it was, you know, for example, Sam a bowing out, and he's like, hey. I'm bowing out because the latest model that we trained, like, started doing all this fucked up shit that we don't understand, and it looks like super scary, and here are the results, everybody. Then maybe that has a different or or broader or more sobering effect. Another possibility is just simply like you get some kind of malicious use of an open source model in a way that has very broadly bad consequences. Like where people get killed, for example. Like where a lot of people get killed and it's clearly attributable to this thing. Then I think that's like that's a a potential avenue where we go like, oh, okay. This stuff is for real. There's blood on the floor now. We wanna look at this in in a sober way and with that kinda general understanding. Then I think you get kind of pulled up into the geopolitical context because regardless of which those 2 which of those 2 things happen, you are in a you're in a space where the risk and dangers and therefore potential military and strategic capabilities of systems like this are now in sharp relief, and that's a somewhat different space than we're in now.

Jeremie Harris: (1:12:37) Yeah. I think it's always the, you know, the first mover's dilemma or whatever. Like, you you have whoever the most risk tolerant actor is who's gonna go out and do a thing. And in a context where, you know, I think the geopolitical dynamics or geostrategic dynamics with China matter here too, you know, the CCP views itself as being in a struggle for survival. I mean, this has always for them been an existential struggle with The United States, with the West, to push their world order, their agenda forward. It's, like, it's difficult to see people I mean, there's always that temptation to go like, oh, can we push it, like, a little bit further? Because now OpenAI has this scary powerful system that they've just announced to the world that they have. By the way, by then, China has for sure stolen it by the time Sam A says, oh my god. I'm free like, under nominal conditions and and even given the current trajectory, like, China has stolen that shit a long time ago. They're, like, polishing up the off the training run on their servers, you know, using their fleet of, like like, stolen h twenties or whatever the next version of the Ascend chip is. But bottom line is that brinksmanship keeps playing out. Right? It like, you get to continue the game from whatever step you're on. And so, like, I see how it would lead to a big media moment. Like, I see how it would lead to a lot of probably gridlock in congress. So, like like, regulation is probably not gonna happen then on any, like, relevant time horizon. So then you're you're at, okay. Well, what can the executive do on its own? You know, you can start to pull all kinds of strings and emergency powers and DPA and all that stuff. But, ultimately, the logic of the race, if nothing else, I think might get exacerbated in that moment where you have an administration that looks over its shoulder at China and says, oh, guess I guess we basically got to the endgame. And you you kind of recover the same conditions that we faced before. Do you trust China or do you not trust China? And the like, based on everything that we've seen and heard, the the list of reasons not to trust China even in extremis is pretty damn long. I mean, you know, the the the, like, lab leak stuff, the way that that-

Edouard Harris: (1:14:34) They've embedded, like they've actively embedded a bunch of Trojans in our critical infrastructure and they're holding a gun to our head. Like that's publicly disclosed. And and I think we we talked about this a little bit offline and we mentioned it in the report, but just like anecdotes, right, about folks in the national security space who dealt with the Chinese and and kind of the way their espionage system is structured. Like, the the CCP takes a controlling and, like, ethnically centered view of its diaspora as in, like, doesn't matter that you're not a Chinese citizen and that you're like a second or third generation American or something. You're still you're still like from the motherland and we own you, and so therefore we're gonna apply pressure to you. Like the story that this this like former

Jeremie Harris: (1:15:27) To be clear, this is also like this is deviating a bit away from what if Sam May came out and said this thing. Right? This is more like the tools and mechanisms that the CCP enjoys with respect to The US. I I think the the the logic we fall into is this question of, like, do you trust them or do you not? I I find it really difficult to imagine, like, actually like, we again, we don't have trust but verify. We don't have, like, flex tag. We don't have the ability to build those secure governable chips. Would be great if we could use very advanced AI systems to accelerate that development. That seems like a really important direction to push in. But right now, if we hit superintelligence in 2027, it's gonna be, you know, just nothing but fog of war. And right now, our ability to monitor what's happening in China is really terrible. Like, anybody who has in their head a scenario where somehow, like, we're we're at parity in terms of intelligence with respect to each other's stacks, that that does not track reality. Right? It is a a profoundly asymmetric scenario, that's 1 of the first things, well, that we think has to be fixed too.

Edouard Harris: (1:16:31) Yeah. 1 of the reasons why I was bringing up those some of those anecdotes was to put color around the Right. Yeah. The the trust issue. Right? So this story that I'll just quickly mention, this defense official, he was talking about a power outage in Berkeley, like, around 2019, and all the Chinese students were freaking out because they are they were mandated to report back on a regular cadence to the motherland or their their handlers or whatever. And these are just, like, regular people, like, regular students, not people particularly working in any sensitive areas or whatever. It's just, like, you report back or your mom doesn't get her insulin or your brother, like, loses out on a job opportunity. And there's an escalating ratchet of just systematic pressures that gets applied more and more and more, and basically, the weight of the the state in this very kind of optimized set of of pressures is applied down to you. And it's challenging for anyone to resist that degree of coercion.

Jeremie Harris: (1:17:32) And which is, by the way, like, I mean, an absolute travesty and a tragedy. Like, no 1 is a greater victim of this than the Chinese people themselves. Chinese nationals working in The United States are subject to at least the potential for this kind of activity. That's, like, that's I can't think of a word for it. It's despicable. I mean, it but and and Chinese researchers have made amazing contributions to our frontier AI, you know, achievements. You can just look at the names on the papers. Right? So so this is this is 1 of that's another issue. Right? You look at these labs, the the composition is then, like, you know, large double digit percentages of these these folks are are at risk from this sort of activity. And so imagine building a Manhattan project in a context where that's the case. You quickly get into, like, these the really dicey things. Right? Like, we're not happy to have to report this. Very much aware, obviously, of, like, yeah, there there's there's these awful like, you know, there's, like, Japanese internment camps is the other side of the coin here. Right? That is World War 2. That's what it means to, like, to take an argue like, an argument like that to 1 extreme. But there's also just the reality of, like, what would it look like to build the Manhattan Project in a world where it's not just that, like, you know, your your, your team building the project is has got a bunch of Germans on it. It's the world of technology has fundamentally changed, and these people could be monitored nonstop. And they have family back in Germany, and they have all their you know, a bunch of financial ties to that country, and and and and and, that's just a fundamentally different calculus. And so it's it's just a real challenge, but it's a reality that somebody has to come out and just say without fear of, like, you know, being called names. It it is just a fact of the matter. And, again, if there's 1 thing that we are here, it's just sort of reporting the news. It's it's an unfortunate reality. We wish we had different news to report. That's kinda how things are shaping up right now.

Nathan Labenz: (1:19:21) Let's go down this rabbit hole of what does it actually look like to try to do an American superintelligence project, aka the Manhattan Project for AGI or ASI or whatever. You know, maybe start off with, like, how does it happen? You know, you've mentioned, like, the president and the DPA, but kind of interested in, like, what you think it actually looks like for us to go from sort of market normal to this, like, national, you know, Manhattan Project style thing. Then I'd love to hear a bit about how you actually learned what you did. I think you guys are great examples of, you know, putting, you know, shoe leather to pavement and, actually getting out into the world and and learning some stuff. And I I think that's really both credibility building for the report and also just something that should inspire other people. And then we can get into the actual, like, okay. If we're actually gonna do this, like, what is actually required to really do it? 1 So take it away. Yeah.

Edouard Harris: (1:20:25) 1 thing I'll just mention out the gate is, like, the way that we started out working on this report was it was in the wake of of situational awareness. And we're like, well, you know, some picture like this needs to be fleshed out whether it comes to pass or not. And so our view was, like, we're we're approaching this like we're not we're it's not clear that it's the right decision to do something like this, but it's sure as heck the right decision to think through what it would look like if in the if, like under the condition that, like, what if we end up doing this? Someone had better have thought through those things in advance. Right? And so we we started working on it through that lens and through that angle. Yeah. And I I think

Jeremie Harris: (1:21:11) the other piece is we explicitly stayed away from partly as a result of that, but we explicitly stayed away from questions about what are the specific authorities that would be invoked in the pursuit of this sort of thing. Bureaucratically, how would this be done? If only because, a, it's just not best and highest use for us. We did include some thoughts about that because we we ended up talking to people in relevant departments and agencies and hearing their views, and we didn't wanna lose that value. So we did kind of park some thoughts there, but they're not set in stone. They're just kind of like they're there to be considered. Our main focus was at the gears level, what has to happen? What are the verbs that that have to, like, be written into the storybook for this to unfold well? What does a data center have to look like? What does a supply chain have to look like? Like, all of these questions that in in many cases are it's interesting. Like, the the challenge you run into is the space is very long on on opinions and opinioneering, much of which is very well informed, and we've leaned heavily on a lot of the great work that's been done in this space. But what is really missing is if you actually talk to the very, very minuscule number of people no. I'm not talking about, like, intelligence and special forces. I'm talking about zoom in on intelligence and special forces at our, you know, tier 1 units. These are our, like, mostly SEAL team fucking 6, Delta Force. Then zoom in even more. Zoom in on the specific people who who, anyway, specialize in the relevant areas and ask them. This is an extremely small community, like, you know, and they're tightly kind of networked and and and it's a very high trust ecosystem. They are aware of TTPs, like tactics, techniques, and procedures that others are not aware of and on which your entire assumption stack can unknowingly rest. Right? So when you have conversations with people in the space and and you're like, hey. So, like, let's just go out and do their their recommendation is like, let's just go out and do x. And you're just like, well, I just had a conversation with somebody about an approach that they know of that makes this totally moot. Like, that's the kind of thing that you run into. And so unless you talk to this very specific group of people, you kinda have an incomplete picture. And, you know, going back to this idea of the jagged capability surface of the models. Right? This is the jagged capability surface or or under or, yeah, capability surface, you could say, of the US government, of US industry. Right? When you actually talk to people at the labs, you get a very nuanced perspective on what's possible. It's not the same happens when you go to the intel community. The same happens when you go to the special operations community. You get a picture of, like, well, actually, this is trivial. This thing you thought was hard is trivial. This thing you thought was trivial is really hard. And that suddenly just refactors your whole picture.

Edouard Harris: (1:23:54) Some of the stuff some of the stuff is just wild too. Like, we we visited a data center with some former special ops folks and not like tier again, tier 1 folks from these, like, restricted very, very, like, elite units. And they're just wandering around asking, oh, what happens if you do this to that thing? Oh, that. Okay, what happens if you do that to that? Okay, cool. And wandering around a little bit and comes back to you and just says, yeah, so I can think of 1 or 2 ops you could pull for $30 that would knock out this data center for about a year. And everyone's like, Wait.

Jeremie Harris: (1:24:29) Like a $10,000,000,000 facility.

Edouard Harris: (1:24:31) Like a $10,000,000,000 facility. And when you're talking at that level, you're not even talking about, Oh, China could do this. Like, a smart person with a bit of time and money can do this. And so that means that adversaries can just like proxy that and use surrogates and use like whatever kind of smoke screens and kind of sock puppets to go and do that with total deniability. And that's the state of security. And we actually have been asked, don't give any details on what those vulnerabilities are because those vulnerabilities remain. And they are critical, and they are universal, basically, across data centers today. That's kind of 1 of

Jeremie Harris: (1:25:13) the challenges here. Right? It's like the right answer is in so few heads and then needs to be integrated. And we're under no illusions that we have the right answer across the board. Right? Like, that that's another implication directly of what we're saying here is the full picture resides in no one's head, but the combination of the oh, an awareness of that uncertainty and then having been shocked by a couple of these individual things is sort of what is shaping at least the strategic picture behind the document.

Edouard Harris: (1:25:40) Yeah. And and just quickly, it's a reason why or 1 reason why it can make sense to publish something like this at all, which is it gives folks something to point to and say, like, hey. I actually know that's wrong because my specific niche area of expertise says this and this and this. And that is the beginning of a constructive discussion.

Nathan Labenz: (1:26:01) So without getting into the specific vulnerabilities, let's talk about what it would take to actually secure such a project. I mean, we're in the scenario now where governments are wide awake. Governments have decided we're in this race. We gotta win it. We can't just have, you know, people, spilling all the secrets at San Francisco parties. You can't have, whatever data center vulnerabilities exist at the infrastructure or hardware level. It's time to get serious. What is that getting serious actually look like?

Jeremie Harris: (1:26:42) So the the first step and, again, we're gonna kind of go a bit meta just to help us have a productive conversation without getting, you know, too deep. And then there are a couple things we can say at the object level, but so first thing is if you think that timelines on the order of, like, late 20 26, 2027, whatever, are are plausible, you ought to be in the business of finding all of the 1 way doors that currently exist in the critical infrastructure stack from supply chain to data centers and all that stuff. Right? So by 1 way doors, what we mean is these are decision points that we are walking through sometimes without realizing it, where you either, like, start so data centers will take, you know, 18 months to build or whatever, at least nominally with current rate of the regulatory environment. So that basically means that, hey. It like, it's time to show your homework. Like, the world's knocking, and the data centers that we're breaking ground on today and designing today are the ones that will have to house a lot of these superintelligence great training runs if they happen in 2027. And so the question then becomes, well, what can we do? What are the sort of, like, cheapest things that we can do that the market will will support today that bias optionality down the line. And and that actually has been a huge fraction of what we have focused on, talking to these special forces operators, talking to the intel community, talking to folks on the hardware side, really deeply understanding what is that set of 1 way doors that we're walking through so we can, at the very least, not rule out the optionality that we need down the line. So from a philosophical standpoint, that's kind of it. There's a very small number of people who can actually give you at least what what we consider to be the right answers here. And we've seen a lot of interactions between people who think they have the answer and then they meet another group of people and you're kinda like, oh, okay. I guess not. To the point where we finally kind of, like, coalesced around some things where we're

Edouard Harris: (1:28:27) like, okay, interesting. Like this is now sort of locking in, but but that'll evolve too. I'll maybe give 1 example of like a 1 way door that we can talk about that is in the report. This is what's called tempest attacks. So these are, actually kind of like awesome and wicked James Bond like things where you can actually figure out what's going on inside of a computer or a system or, like, pull data from it by just, like, watching the electromagnetic emanations from that system. So, like, in the most extreme version, you're in a room with a computer. The computer is air gapped. So it's like it's literally not on the Internet. It's just like sitting there on a desk, not connected to anything. If you have, like, a particular virus in that computer, this is 1 example of this, that accesses the computer's memory in a specific pattern, you can have a radio receiver that's, like, a few feet away from it across the air gap that's listening for literally the radio emanations from that little, like, memory access stuff that's going from the CPU to, like, to the memory. And you can actually have information transmitted to you across the air air gap by just, like, listening for the memory access patterns. It's insane that it's possible, but that is that particular attack and detection is public information. It was published about a year ago, but almost surely has been known by intelligence agencies and many kind of the usual suspects for quite a while before because it's like something you can do, and it's extremely useful. And so there are standards around how you defend against that. And some of those standards are, like I talked about, you know, you're a few feet away and you can detect. Kind of the the only way to realistically do that and and and block that off at the data center level is you literally have to have, like, a few feet of spacing between your compute racks and the walls of your data hall because you can have a visitor space on the other side of that wall and some dude who looks all innocent like has an app on their phone or something that's detecting the emanations from your stuff and you've planted malware on the system, and they're just pulling data from you at a bit rate. And they're just sitting there casually in your visitor center and pulling data from your computers and your GPUs from across the wall. And so the way to block that is you just create that space. And if you don't create that space, if you don't build your data halls with that space in mind, then you're kinda like, oh, whoops. Like, my data halls are just too physically small. And that's a 1 way door because the way out of that, like, yeah, you can retrofit, but it involves, like, knocking down and redesigning your facility.

Jeremie Harris: (1:31:14) And so the data center 1 way doors are really interesting because they affect construction today of a thing that you can, like, see right now. The supply chain 1 way doors are in a weird way they're they're actually thornier. So, because they involve, obviously, entire supply chain. So TSMC makes all our chips. TSMC sits on Taiwan. You know, I don't need to tell you that the CCP is very, very fucking interested in what's going on there. And if you were China, I mean, you could just draw the obvious extrapolations. What would you be up to knowing about you know, it doesn't even have to be tempted. I mean, it's like cyberattacks personnel. I mean, jeez, TSMC famously has lost executives who went on to found SMIC and steal their shit. Right? Like, this is a a known playbook that has been working for decades.

Edouard Harris: (1:32:01) Hundred years.

Jeremie Harris: (1:32:02) Yeah. So so there's that. There's also, like even at the level of these mundane, easily overlooked, boring components that go into server systems, So, like, baseboard management controllers is is a a great example. They're, like, 80% market share of BMCs goes to Aspeed, this, like, other company in Taiwan. And, like, I I guess I hope that they have their shit together because baseboard management controllers run on, like, a separate power supply from the GPUs that that, you know, and the CPUs that and they often have, like, read access and write access to to, like, firmware, and, like, it's a a dream backdoor. Like, it it is the soft underbelly of a lot of, like, HPC AI infrastructure. And so, like, there's 1000000 things like this. There's transformers and transformer substations, basically, like, component wise. Like, none of them have no components that are made in China, and we know that that the CCP has actually used transformers specifically to plant backdoors in them in in American infrastructure. So it's it's not just like so many of these things, it's it's tough because you talk to people about it, and there is correctly a view that, like, you can't cover down on all vulnerabilities, all threats. Like, you gotta you gotta pick and choose. The challenge is we're not even highlighting, like or let's say a good fraction of what we're highlighting here isn't even stuff that is, like, speculative. It's stuff we know China has literally done to us on our soil. And so it's yeah. It it it's 1 of those things where you just have to take take the reins in some way and Yeah.

Nathan Labenz: (1:33:37) Yeah. Maybe just give me a

Nathan Labenz: (1:33:38) little bit more on the sort

Nathan Labenz: (1:33:39) of backdoors as they're understood to exist in the, like, electrical grid. And then, you know, I don't maybe that'll be enough to sort of imply what, it means for the data centers, but I'd be interested too in, like, you know, in the report, there's like the cooling modules come from China. There's like, you know, we have sort of the sort of crown jewels of a data center might come from Taiwan, but then, like, a lot of the other pieces that are needed to make it go seem to come from China to the degree that, you know, my kind of takeaway was like, there's really just no way that we're gonna bring that whole supply chain to The United States in any sort of short term scenario. But, yeah, maybe let's start with, like, what is known, what is understood, you know, what when you say, like, China has a gun to our head with respect to the transformers that they've backdoored in our electrical grid. Like, what does that mean for me? What what am what am I vulnerable to that I may not realize?

Edouard Harris: (1:34:39) Yeah. So this means that they can hold our they can hold our infrastructure at risk and use that in some sense as a bargaining chip, if things heat up to disrupt us completely and distract us while they are taking hold of their own objectives. So this is this is 1, you know, possible scenario of a Taiwan invasion where there are many, many logistical challenges that The United States faces in defending Taiwan. 1 of those is just like, we're just gonna, like, run out of stuff after the first week or 2 of, like, intense high heat combat at those at those scales. But another thing they can do is, like, hey. If it looks like, if it looks like things are not going as well as they hope for a plan because of American intervention, sure. Just, like, turn off all of our grids after, after, you know, putting some propaganda in our information spaces. Basically, turn out the lights and, like, okay. Like, let the chaos reign and, yeah, see basically, just, like, throw a wrench in the gears. Why wouldn't they do that?

Jeremie Harris: (1:35:44) I think 1 important ingredient to strategically here is, you know, when when you have a capability or or set of capabilities to deploy against an adversary like China or like The United States, you whoever you are, you're you're never gonna show your cart your, like, your full set of carts. Right? You always go with, what is the what is the smallest kind of jab that I can throw that has the desired effect that doesn't reveal my whole set of capabilities? Right? The intent is always how can I learn without teaching? And that that means that you like, we have genuinely no idea how high the ceiling is on Chinese capabilities, but the floor is pretty goddamn high. Right? So as we as we think about stack ranking and prioritizing the the interventions, which is where your I think your question was heading, like, what, like, what do we actually do here? The the good news is that the structures you need to build are you need to build a relatively small number of structures relative to the, like, kind of power demands of the entire United States. We can't go we can't go into the solution because it is redacted from the document for obvious reasons. There are solutions that we've identified for a number of things, including well, I I won't mention. I can't remember if if we say what the thing is. In any case, they they have to do broadly with, yeah, supply chain techniques. But the fact that this is a focused problem does help you. There are ways in which the problem is constrained geographically, technologically, that that make it more more tractable. An another kind of corollary to this is you can't live in a world where China holds you for ransom and you don't do the same to them. Right? Right now, that is the world we live in. We have, like, very poor ability visibility to to see what is going on on CCP infrastructure. That gets hard actually. So, Nathan, I think we we talked a couple weeks back about, like, why did Google release streaming deloco and that that whole kind of like, this gets harder with fucking deloco. It gets harder with Yeah. You know, intellect. It gets harder with together AI, all these kind of distributed computing technologies that just make it so hard to track what is even happening. Right? So you can hope for today the big mega cluster 10 gigawatts of 3 gorgeous damn, like, power that you might detect through satellite imagery, but that's not always gonna be the case. And in fact, it seems plausible that it'll become less and less the case on relevant timelines. And so so the question then becomes, okay. Well, how do we establish a reciprocal capability? We need at least that option. Like, if China has the ability to hold American critical infrastructure for ransom, the the fact that we do not have that option in reverse is a giant problem and and should be a priority to fix.

Edouard Harris: (1:38:21) And we can do I mean I mean, yeah, without going into too much detail, our understanding is, like, there there is stuff we can do, in retaliation, but it's we're not where we need to be.

Nathan Labenz: (1:38:36) And just to make this a little bit more colorful, are we talking about, like, transformer substations in communities across America, like, suddenly going, like, Hezbollah pager and just, like, exploding? Or, like, what exactly is the

Edouard Harris: (1:38:54) Yeah. I mean scenario. Yeah. Stuff like that. So the thing is, like, there's, this actually comes back to what Jared was saying about you reveal the the capability that gives you the effect that you desire and nothing above that. So you've got, like, 10 levels that you can operate on as a nation state. You're operating on all 10 of them. You wanna reveal like the lowest level that does the thing that you want. So for example, there's like little touches that we see kind of across The United States where you have things like folks This has been reported where people, a couple of folks, like Chinese folks are flying drones to just observe critical infrastructure and things like that. You certainly can just, yeah, knock things out with explosives or literally just like short circuit a transformer when it's turned off and rewire into itself. And if it's not being regularly inspected, when that transformer turns on again, it explodes like a bomb. All of that stuff is within the arsenal of things that they can potentially do in certain critical places. Other stuff involves, like, you own the firmware on that transformer. You also are the supply chain that produced that transformer. So if you want to and if you're thinking far enough ahead, you can totally install backdoors that make that transformer effectively inoperable and just, like, bricks it. So you don't need to blow it up. You don't need to do anything that subtle. In theory, you could flip a switch and, like, okay. This transformer is, like, just a brick now, and that's just the way it is. And so now it's a it's a game of, like, well, we need to somehow replace, like, a large fraction of the transformers in random places in The United States under conditions where we are we many places in The United States don't have electricity or comms because the transformers are now.

Nathan Labenz: (1:40:53) Yeah. And we also don't build many of them, if any. Right? So that's another big problem. Yeah.

Jeremie Harris: (1:40:58) Means to build a transformer or a transformer substation is really interesting. Right? Because it's like, what so where do you source your materials? And how far up to the that supply chain do I have to go before I have, you know, a manufacturer in China? And it's actually, like, quite often not very far, and, and that's part of the challenge. Right? Generally speaking, this is a it's a it's a very high dimensional problem, obviously. Again, data centers are fortunately, they're an insanely high dimensional object. They're a much lower dimensional object than the entirety of The United States, and there are things you can do anyway to reduce the risk in significant ways. But you can see the geostrategic calculation too here, right, where it's like, okay. Given that this is the case for The United States, if if this is true, then we have a larger strategic challenge on our hands even absent ASI, and and it can be awfully tempting to look at ASI as partly a solution to that problem. So if you have this ability to build, you know, whatever kind of crazy offensive weapon through ASI that you might, there's then this temptation to say, well, if we can just secure this 1 thing and we can, like, use this thing to to have this effect, then, you know, then we'll be able to kind of transcend all these other issues. So so the kind of the strategic calculus here is super, super loaded and complex, which is anyway, it's a big part of what we can't talk about.

Nathan Labenz: (1:42:19) But, I mean, the general picture that I take away from the report is, like, we're moving AI research to some sort of secure facility in Nevada or something. Right? Like, some remote place, and we're building, like, data centers underground, and we're trying to, you know, at whatever cost. Right? We don't care about it being economically competitive. We just need to, like, figure out ways to make certain critical components domestically so we can have some sort of secure supply chains. We're, like, just pouring good money over bad onto that. And we're also, like, doing serious vetting of our people, maybe like not allowing them to go home, meaning like they literally have to, you know, sleep on the site and have their calls monitored or whatever. Maybe that would be a way to not have to like literally purge all Chinese, you know, nationals or, you know, even just, Chinese Americans. If we had that sort of level of surveillance, obviously, that's not super pleasant. It starts to sound, by the way, a lot more like China, which is a general trend, I should note Yeah. In American, governance. You know, just tweeted this morning, like, the, you know, the US government is trying to tell me that I should be afraid of Chinese values. At the same time, it is embodying Chinese values in a more and more egregious way all the time, like, literally disappearing people without due process for political expression. So that's a problem. But

Edouard Harris: (1:43:49) Well, with I mean, with a project like this, here, again, we're talking about, like, on the assumption that you're doing a national superintelligence project. Right? And and without necessarily talking about, like, is it a good idea or not for the for this purpose. If you're actually doing that, you you look back to the the original, the the actual Manhattan Project for nuclear weapons. And yeah. I mean, even back then, right, you had this kind of surveillance through the comms channels that they had available.

Nathan Labenz: (1:44:16) And Yeah. Feynman famously, like, made a code, system with his wife. Right?

Edouard Harris: (1:44:19) That's right. That's right. And in, like, little protest of that, and and, like, it's they were reading your letters, and that was, like, absolutely outrageously unprecedented at the time. Like so they had to they had to agree to it voluntarily to have their mail read and all this stuff, and people were playing around with the sensors and the code and figuring out the boundaries of the system. But then Claude Fuchs just went and stole the secrets anyway. So he and he

Nathan Labenz: (1:44:43) Hey. Like, it's the end in preserving the secrets.

Jeremie Harris: (1:44:47) Well, so a couple thoughts here, though, too. So on the Claude's Fuchs thing, this is another area where as research gets more and more automated, that that does help, right, in some ways. It's a double edged sword. It's it's a from a loss of control standpoint, holy shit. And then from a security standpoint, it's like, oh, okay. I see what you're doing there. The the other side too is so so we're not advocating for building these things underground or or in any such context. Part of the calculation was okay. So here here's the, you know, here's the list of the full suite of things that you would really wanna do to play it tight. Separate question, how do we pragmatically do this? That's actually a question that we've been thinking about working on not that we've been thinking about working on, that we've been thinking about and working on for the last sort of 6 months really intensely and, and and sort of, like, figuring out, okay. If we actually had to, like, to do this for real, what is the the kind of Pareto optimal frontier look like, and and what do those trade offs look like? And I think you actually can get to some pretty satisfactory places even in the face of the full Chinese and CCP threat here. But the question is always gonna be, how do I, yeah, how do I spend my marginal dollar and how how do I trade off, yeah, the need for more compute and and whatever else with with, you know, buffer for security? Yeah.

Edouard Harris: (1:46:02) And and to be specific about that, like, okay. How do you actually think about those trade offs? Right? You can't cover everything. You can't actually, in the time we have, build a data center on the mountain with, like, freaking, like, cooling cooling ducts, like, sticking out of the like, it's just like, forget it. That's just too much work. So how do you actually think about it? Well, you think about it in the context of, like, again, if you were doing such a project, you would need it to be embedded into your national security and counterintelligence apparatus. So you're you're taking this because otherwise, like, it it just doesn't make sense. It's like a stand alone thing. You're so, you know, the NSA is like has a team there. Right? Like, the other stuff like that. You've got you've got people actively working and collaborating. And 1 of the things that that means is you've got eyes and you've got attempts at collecting information on the adversary side. So, like, you're you're collecting information on China's efforts to subvert and attack and exfiltrate from that project. So now your your challenge actually is not defend against everything. Rather, the challenge is like, we wanna put enough measures in place to raise the cost for them to do an attack or an exfiltration to the point where we can see that because we're forcing them to marshal enough resources to be able to overcome our security bar. Doesn't mean that we have doesn't mean that we can actually fully defend against everything. At the end of the day, like, they could just shoot a cruise missile from offshore and, like, blow the bejesus out of us. And, like, nothing we can really do about that, like, at the end of the day. But if they do that, then we can see that cruise missile, and we can go, hey, guys. You just shot a cruise missile at us. We're gonna retaliate. And so you basically raise the bar to the point where you force enough resources to be marshaled from their side that that creates a signature we can see incredibly offers a threat of retaliation. This is this is like how that security posture gets integrated into a counterintelligence motion.

Jeremie Harris: (1:48:14) Yeah. The the kind of flip side to that, you know, if you don't have that set up, right, you essentially just if you're OpenAI or your or your x AI or whatever lab is is close to doing it first, what you find is like, oh, this, you know, this training run just I don't know. It's just not or or the or the inferencing process, whatever whatever the key stage ends up being, like, it's just not working. Like, it's super buggy. There there are bunch of issues weird. And in the meantime, obviously, like, you're not aware, but relevant IP has been exfiltrated. There are more scale training runs happening more efficiently in China or or wherever, and and your adversary is developing that capability, and you have no there's no consequence. And and this is 1 common theme that that we just, like I guess, just to kind of add some color, this is maybe backing up a little bit, but on The US China side, what's been missing, at least until now, what's been missing throughout, like, the the Biden years? This has been a complaint that we've heard. And and before too, in many administrations, has been consequence. So, you know, America's adversaries, including China, have been able to conduct operations in The United States that do a lot of damage without consequence. And that's because there's been a lot of anxiety about, like, oh, well, if we respond, then, you know, right away, the brain goes to nuclear escalation. The problem is you cannot do business with adversaries with when that's your mindset. You just can't. I mean, what you end up with is people who are going to take advantage of every little I mean, it's the whole I'm not touching the devil thing. Like, they'll get closer and closer to your face. They will eventually start to do things, which we know they have actually been doing to American critical infrastructure and people. And, I mean, you think about the the, like, Chinese police stations operating within US borders, basically, with impunity. What the hell is that? Right? Like like, if we acknowledge that these things were real, those are things that border on acts of war. And the fact that we have not offered consequence for those moves means that we are teaching the adversary to continue to push in that direction. We're now about to pay the cost when it comes to AI. If there's no consequence for what is done to American interests domestically or abroad, if people are able to act with impunity, all of a sudden, I mean, yeah, I get to for free try to take out your your critical infrastructure. Why would I even why would I think twice? Right? It's a free shot, whereas we don't get the same reciprocally.

Edouard Harris: (1:50:33) People people don't realize this, but, like, stability between powers today is actually is, like is not maintained through actual defense. It's maintained through threat of consequence and threat of credible retaliation. It's like it's this it's actually, like, kind of fucked up that the world is this way, but stability between, like, powers and major states is maintained the same way that, like, gangs in Chicago maintain each other's territory, which is like, I I know that I can't actually stop you from, like, busting a cap in in the ass of 1 of my boys. But if you kill 1 of my boys, like, I'm gonna fucking kill 1 of your boys right back.

Jeremie Harris: (1:51:15) And That's the first time

Nathan Labenz: (1:51:16) I've ever

Jeremie Harris: (1:51:16) heard anybody bust a cap. Okay. Cool.

Edouard Harris: (1:51:18) Yeah. Yeah. That's that's a first. That's an exclusive for for the Cognitive Revolution. But that's how stability is maintained in truth. It's unattractive to think about it, but it it is it is it's it's the bedrock. It's the air or in the the air we breathe in the world today as wild and scary as that is. So,

Nathan Labenz: (1:51:41) I mean, I will I guess I'll refer people to the report for more, depth and detail on the many supply chain vulnerabilities, you know, the difficulty in securing a libertine research staff, in the, you know, remote locations to which they might be absconded. And on top of that, we haven't even got into the measures that you recommend for maintaining control over the AI itself, which is definitely a, you know, a new type of capability that the I mean, for everybody, but, know, including the US government if it wants to develop and and deploy those to make sure that it's not inadvertently, you know, unleashing Armageddon on everybody. So we got a lot of problems. I know you're kind of trying to keep this somewhat neutral stance on, like, we're not saying do or don't do, but we're kind of just trying to tell you what would be involved. Would you be I guess, you know, to put my cards on the table, I read this and I kinda came away feeling like we shouldn't. You know? Like, it reads to me like a warning more than a manual that I'm excited to to sign up.

Jeremie Harris: (1:52:46) But actually, this is so funny because we've had so many conversations with people who are like, dude, why are you telling people to do a superintelligence project?

Edouard Harris: (1:52:56) It's a bit of a Rorschach test, right? People will see what they wanna see in it. And to an extent, that's fine. Think the thing that we took from it, after all said and done and kind of starting from you know, we like, don't know if it's a good idea or not. Then and then, like, having done it and looking back and be like, well, what is our conclusion? Like, you know, our pride ourselves. And big part of our conclusion is most of the measures we recommend should be implemented whether we're doing some kind of big national thing or not. Because if they're not implemented, then you get that situation that is exactly what Jared described, which is like, yeah, China just, like, steals your weights at the eleventh hour. They stuxnet you so that, like, you don't even realize it's happening. You just feel like your training run is like, oh, it's like it's it's everything's proceeding slower. There's so many more bugs and blah blah blah. And all they're doing is, like, they're they're throwing sand into your gears, and they're they're working and and improving stuff on their side, and you don't even realize that's what's going on. 1 of the things that, like, we learn from talking to these intelligence folks is, like, nonstate actors, like proxies for nation states, are are proxies for nation states and have many of the techniques and tools and procedures, in fact, almost all that the nation states themselves have. So the distinction between, like, China's doing this versus, like, some group that China is supporting that's a nonstate group is doing this is actually not that sharp in terms of capability. Like, they're trained, funded, and supported by the Chinese or by whatever adversary. And so you're facing down nation state capabilities even if your proximate adversary is not actually a nation state. And what that means is, like, the bar for security just has to be high all around to prevent this from happening whether we have a national superintelligence project or whether we have, like, 5 random superintelligence projects that are all kinda dipsy doodling from their own size and angles. This is just like the vulnerability surface that exists. It sucks that that's true, but it is.

Jeremie Harris: (1:55:15) Yeah. And it so and because when it says, like, a national superintelligence project, right, like, essentially, like, a US government backed superintelligence project, which then there's, like, backed and led and and, like, that's where we start getting into, like well, honestly, like, it doesn't matter. Like, at at the end of the day, what you care about is how is the rubber gonna meet the road? Are they or are they not gonna be able to just extract critical IP, stuck at your stuff, or, anyway, do any number of other things? Those questions all have answers that route through a lot of the same channels. There's a question of task organization, if the US government takes it on, but that's why we scope that out. Right? So a lot of this really is the story of if you're going to build superintelligence and treat it as the WMD capability that, you know, Sam Altman tells us it is or will be, with no apologies, then you ought to be doing something that looks fairly different from Stargate. Like, what it looks like to actually take that seriously, to take your responsibility seriously in building this tech looks very different from let's announce to the world that we're doing a $500,000,000,000 project, and everybody knows exactly where the facilities are. And, oh, by the way, our lab contains, like, x percent of, like, Chinese nationals. And by the way, we've got all these issues with people leaking, like, critical IP from Slack openly. Like, we've got, 1 of the things the report, uncovered was we had a an insider from OpenAI tell us, hey. I like, just digging around, I found these 2, really critical cyber vulnerabilities that would have allowed me to access and then extract weights for, like, a critical model that the lab was hosting. Like, this is just not what you do if you are a serious actor and you take your commitment seriously, and and so that's a big issue. It's not that OpenAI is like like a bad actor in the kind of zoomed out sense. It's that in part, they're just victims to the the racing dynamics that are playing out right now. And so how, yeah, how do you get your arms around that? Well, step 1 presumably is having sense of what the vulnerabilities are that you need to cover down on, and then you can price those in. You can start to make a list of them, prioritize by cost, prioritize by probability that they'll be exploited, and work your way down. That's, you know, all you can ever really do in this space.

Nathan Labenz: (1:57:28) I don't love it, and I know you guys don't either. Let me just try in the the few closing minutes that we have to to just get a couple reactions on on other things. 1 is, like, China's open sourcing all their stuff. You know? You know, 1 sort of Pollyanna ish view is, like, they don't appear to be trying to race us. They're giving us all their best shit for free. How do you interpret their open sourcing strategy? Do you expect it to continue?

Jeremie Harris: (1:57:53) Oh. They have an open sourcing strategy is 1 question that never gets asked. The answer surely is no. They have an or they have an open sourcing strategy, but it is not a universal open sourcing strategy. The 1 of the challenges is we do not know what's going on on those Chinese servers. Right? Like, we can't answer that question. What we see is, like so we actually heard from a, like, a former OpenAI researcher that he was having a conversation about, like, how do we know that China hasn't stolen any of our shit? And the response was literally at the time, well, we haven't seen any comparable open source models come out of China. So that basically tells us they haven't stolen our stuff.

Edouard Harris: (1:58:26) Like, dude Constant comparable models. Models with comparable capabilities. Yeah.

Jeremie Harris: (1:58:30) Right. Sorry. Things that are legible. Yeah. And so you're just not gonna get that intelligence. The flip side is anything that comes out of DeepSeek from here on out, in fact, anything that comes out of Huawei or anywhere else, Tencent, you name it, is coming out with the imprimatur of the CCP. There's 0% chance that those models are getting open sourced without people high up at the CCP going like, cool. At a high level, that makes sense. They may not approve every single individual 1, but the general strategy behind it is CCP approved. You can make of that what you want. If you are the CCP, you'd probably have an interest in, you know, flooding the West with language models that are gonna not talk about tin and min at a minimum. But as we move into more and more agentic systems, boy, does it look interesting to start to buy the public's confidence in your agentic models when there are interesting backdoors that could be baked in Yeah. Anyway, interesting behaviors you might wanna exploit.

Edouard Harris: (1:59:20) 1 other thought there is, like, we suspect that prior to the end of last year, DeepSeek was not that integrated into the CCP apparatus, though it is super integrated now. The reason being a a number of comments by the founder of DeepSeek around, like, man, American export controls really work against us. Jeez. They're really biting that he made, like, back in July on a random podcast, which just, like, utterly undermined the CCP's entire, like, propaganda around, like, yeah, American export controls, like, just don't work around ships, so you might as well give them up. You might as well just, like, stop. Right? Please? And they've been, we don't have time to get into this, but they've been putting, like, propaganda set pieces around that in place for years at massive cost and effort. And this, like, random dude who now has been catapulted into fame just, like, absolutely gutted that entire effort at great

Nathan Labenz: (2:00:16) cost to them.

Jeremie Harris: (2:00:17) And there there is also the economic argument too. And these are just, like, public arguments people are throwing around, and I'm I'm just offering them as explanations for why would the CCP be okay with this, which is the ultimate question that we really just need to answer. But the other 1 is, you know, the economy of flood flood the market with cheap LLMs. You've seen what that's done to Meta. I mean, like, Jesus. They're basically, like, they're now having to gut their

Nathan Labenz: (2:00:38) They're flooding the market with cheap LOs.

Jeremie Harris: (2:00:40) Yeah. Exactly. Yeah. Yeah.

Edouard Harris: (2:00:42) So How about meta?

Nathan Labenz: (2:00:44) Okay. I don't know if we have time for 2 more, but I would ask 2 more if we have time.

Edouard Harris: (2:00:48) 1. You want more?

Nathan Labenz: (2:00:51) Alright. Maybe I'll come I'll see if I can combine them into 1. How can I do that? Maybe I'll just put them I'll just ask them both and you can answer it as you will. 1 question is, do we even have a set of prerequisites for some sort of grand bargain? You know, if we were gonna say, okay. We're gonna come to the table. We know we don't have great trust, but is there some sort of technological checklist or trust you know, and or trust building exercise checklist that we could go down to say, okay. If we had these things, then we could maybe enter into some sort of sustainable grand bargain. And then part 2, another kind of angle on it is just like, I know you guys read, Dan Hendrix et al's, main theory. And I wonder if you have, you know, any hope for either an emergent equilibrium that sort of keeps things under control or perhaps a more engineered, you know, as maybe as part of a grand bargain, some sort of semi engineered equilibrium that could keep things stable between great powers even if, you know, trust continues to be a scarce resource.

Jeremie Harris: (2:02:09) The grand bargain side, like, the prerequisite. So this starts to touch pretty close closely on the whole FlexHead program and, you know, secure governable chips. Can you make can you make a tamper proof enclosure for your chips? Can you make at least a tamper detecting enclosure that requires, you know, inspections? Can you then have some sort of, like, you know, like, governance chip that's sitting on there too to verify inputs and outputs and all that stuff? I love that research agenda. I think it's a great research program. The problem with it is timelines. There's just like it's a really hard obstacle to overcome that fabbing is slow and chip design is slow and integration is slow and scaling is slow and debugging is slow. So love that program. Hope that it gets really significantly accelerated by AI, and this is something that I think should be a priority in the context of the superintelligence project. You would wanna couple the the kind of intelligence engine to the design process for for these these chips. You'd wanna couple it to, you know, advising on what what to kinda do next and and fab next. But but, yeah, that's that's kind of, I think, a prerequisite is we need to have visibility. Trust but verify has got to be an option. Until it is, we need to be treating China as an intelligent adversary and a committed 1. Maybe I'll I'll leave the Ed to do the cleanup on that, and if you have thoughts on the main thing you wanna throw out, Ed too.

Edouard Harris: (2:03:28) Yeah. Generally agree. I think we need to be pursuing every avenue. And if if there is any chance, even though it seems quite remote of some kind of verification around collaboration, Yeah. Like, let's throw a few billion bucks into that because, like, absolutely, if that turns out to be, like, a dark horse, like, win, totally. Yeah. That that's that's worth the investment. Timelines are are tough. Another tough aspect is, like, the folks that we talk to are not optimistic about can you actually build a tamper detecting enclosure at all that if it's subject to sustained pressure under full control of of an adversary that can resist even at scale. So, like, you know, you'd have to you'd have to, like, basically bust through 1000000 of these to have, like, a meaningful amount of of chip capacity is like less optimistic. And then on on the meme side, it's actually a shame we didn't get to discuss meme more because it's actually a really, really interesting piece of work. And I think that 1 of the things that it did really, really well is that it it was the first the first place where really this kind of, like, approach of no of, like, no. We actually need to be reaching out and touching the adversary. We need to be actively doing, like, counter counterintelligence and capability degradation stuff, like, soon really came out, and that's a critical pillar. And you can kinda see that, like, how the logic kinda forces you into it. Right? Because if if you wanna, like, have have have an aligned AGI or superintelligence, You need the margin over the second place adversary or or competitor to actually do that development of that that alignment technology. In order to get that margin, you can either accelerate yourself or try to decelerate the competitor. Accelerating yourself shortens timelines and creates, like, additional risk, and so the logic kind of forces you into, well, you need to do capability degradation for the adversary. And Maine did a great job of framing that and and pointing that out. 1 of the issues with it where we when we talk to some of the the intelligence folks that that we are connected with is it comes back to, like, you can't actually put a bar of security that's like nation states can go above this bar and can, like, hold your stuff at risk, but non nation states can't. And so and so you can you can defend against 1 but not the other. As we've seen, those sets of capabilities are actually not as distinct as many people think, and this idea that you can fine tune security levels and and do this kind of, like, scalpel stuff is not viewed very encouragingly in those spaces. Doesn't mean it's not a really important part of the truth, and it is. But there's there's some aspects missing that, like, I think I mean, I think it would be really productive to discuss with Dan about more deeply even on on your podcast at some point in the future.

Nathan Labenz: (2:06:34) I love it. We'll see if we can make that happen. Any other closing thoughts or, you know, assignments you wanna give to the audience?

Jeremie Harris: (2:06:43) Yeah. Well, I I would say none that comes to mind. Like, read the report. If you have thoughts or feedback, we would love it. We we think of it as a living document even though it's a point in time. It's not gonna be perfect and things change. But, yeah, I mean, really eager to get anybody's views, thoughts on it. We want to aggregate as much as possible. We are maintaining a list of, like, yeah, data center security techniques that we think would make the difference and that are 1 way doors, especially. So to the extent that you have thoughts on on that side of things, always eager to hear from anybody in the, you know, intel community, special ops community, and all that.

Edouard Harris: (2:07:23) Just the any any relevant community. Right? Like AI security, AI

Jeremie Harris: (2:07:28) Flex Hague, secure governable chips, you know, like, aligning people. Yeah.

Edouard Harris: (2:07:33) Yep. Cool.

Nathan Labenz: (2:07:36) Well, then they'll, know where to find you. For now, Jeremie and Edouard Harris, founders of Gladstone AI, thank you both for being part of the Cognitive Revolution.

Nathan Labenz: (2:07:46) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.