Full Transcript

Transcript

Nathan Labenz: (0:00) Hello and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz joined by my cohost, Eric Torenberg. Hello and welcome back to the Cognitive Revolution. My guest today is the 1 and only Samo Burja, political scientist and founder of Bismarck Analysis. Samo is the strategic mind behind the weekly podcast, Live Players, also on the triptyne network, where he and Eric Torenberg unpack sometimes esoteric but always critically important issues. From the evolving role of universities in Western society to the Chinese government's resource and energy strategy to the global problem of declining birth rates and lots more besides. I listen to every episode of live players, often on the day it's released. So it was a real treat to have Samo on the show to discuss the strategic dynamics of artificial intelligence through a geopolitical lens. We cover a lot of ground in this conversation. On AI's overall trajectory, Samo questions the transformer architecture's long term dominance, suggesting that we may soon see qualitatively different AI paradigms emerge and that nations should continue to emphasize fundamental science rather than putting all of their energy into engineering scale ups. When it comes to the advanced chip supply chain, Samo advocates for industrial policy, including regulatory overhaul to boost US competitiveness, possibly via the designation of special economic zones, and even including direct government purchase of domestically manufactured chips if necessary. On US China relations, Samo advocates for framing The US China relationship as 1 of economic competition rather than geopolitical conflict. And he critiques the 2022 Biden chip export controls as potentially counterproductive, arguing that they may well accelerate China's domestic chip industry. Regarding open source AI models, Samo identifies difficult trade offs. On the 1 hand, concentrating too much power in the hands of a few tech giants or governments could lead us to an Orwellian surveillance state. But on the other, unrestricted AI proliferation could enable bad actors to destabilize societies. Ultimately, there is no simple answer, and Samo argues that finding the right balance will be crucial. On the militarization of AI, Samo worries, as I do, that defense tech startups can and probably will begin to raise money to build a sort of Skynet and that this could dramatically increase existential risks and thus should be approached with extreme caution. Finally, on AI safety, Samo emphasizes the need for ideological diversity in AI research and governance. He suggests that a monoculture of thought, whether from effective altruists, corporate interests, or any single perspective, could lead to blind spots in safety considerations. And thus, he recommends that diverse viewpoints be intentionally installed in key decision making roles. As you'd expect, Samo brings both geopolitical expertise and technological sophistication to all of these topics. And I was very glad to hear someone who is known for clear eyed real politic analysis argue that The US can and should attempt to work more constructively with China as opposed to taking the ongoing process of mutual decoupling and escalation for granted. As always, if you're finding value in the show, we'd appreciate it if you take a moment to share it with friends or post a review on Apple Podcasts or Spotify. And we always value your feedback or your topic and guest suggestions either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now I hope you enjoy this highly strategic geopolitical perspective on AI development with Samo Burja of Bismarck Analysis.

Nathan Labenz: (3:48) Samo Burja, founder of Bismarck Analysis. Welcome to the Cognitive Revolution.

Samo Burja: (3:52) Great to be here.

Nathan Labenz: (3:55) So you are known for your strategic analysis and for your super crystal clear monologues. I'm a active listener to your show that you do with Eric, live players, and definitely recommend folks check that out for in-depth analysis of many of the nation state level live players. Obviously, 1 of the hottest topics in the world today and 1 that is gradually infusing all of geopolitics is AI. That's, you know, many dimensional topic, but I wanted to start off before we go deeper into different live players and what their, you know, interests and positions and strategies are likely to be. Just a little bit of context on kind of your AI worldview, I find that many conversations fork early on questions of just, like, what do you actually expect to happen? Do you think that AI is gonna get super powerful or, you know, not so much? And we'll leave any debate of that topic for another time, but I just wanted to at least give people a sense for kind of what your outlook is so that they can, you know, better contextualize the rest of your analysis.

Samo Burja: (4:58) I think that artificial intelligence is a fascinating test for people. It really whenever you talk about artificial intelligence, you end up having to look at basically all of the world models that 1 has, you have to employ them, you have to reconsider them, you have to check which assumptions hold and which assumptions should be dropped. I think it's fascinating how much people's expectations of the future of AI reflect their general thinking on technology, or even their general thinking on what is a mind, and ultimately, what human beings are and what is special about human beings. You might have people who greatly focus, say, on the lack of agency that the current AI models display, right? You might say that actually that rather than intelligence is the most valuable thing. And then others would brush that aside and say, you know, all you really need is a for loop on, you know, already there's generative intelligence. First off, that's shown to be not quite true. But it is a fundamental question of whether something much a very different process would be needed to produce agentic AI or whether it's just a small tweak that we're going to get through, through the normal application of generative AI. So first to get it out of the way, I think there's been real scientific progress in the early 2000s on just the computer science of artificial intelligence. The term AI scientist did actually make sense then. It still sort of makes sense. I think since the last 4 or 5 years, there have been new breakthroughs as the larger and larger models have been trained. However, to a great extent, we're merely applying these theoretical insights from the early 2 thousands. That's why there is a lot of disputation as to who ultimately should properly be credited for the so called transformer architecture that is now used to basically generate most of the tax that's behind LLMs. LLM is in a way a much less descriptive marker, right? It's a marker for an application rather than the technology itself. It's like talking about a car, when, of course, there could be an internal combustion engine inside the car, or there could be an electric motor. I'm certain that now that there is demand for basically generation of text with comes with generation of knowledge, that there will be architectural revolutions that will optimize for it even further than what transformers have done. So I do expect some continued progress in sort of the science of artificial intelligence, that is the theoretical side, the architecture side, and the discovery of what do larger and larger and different kinds of data sets do. We are in a way discovering it. We don't have strong theory behind what's happening. That's an an important thing that has to be stated. This is not similar to the development process of, say, supersonic flight where, let's say, 80 to 90% of the science is just resolved. We have our models of physics. It's not pushing physics that much forward except maybe some details of fluid dynamics, which are chaotic. We're just doing engineering on the basis of a theory. Here, the engineering has outpaced theory long ago, and we're continually surprised by what are essentially not just business ventures. They're not just launching an app. A training run is actually an experiment. I think we should think of it as if we've built dozens of large hadron colliders around the world. You know, the scientific experiment on the border between Switzerland and France at CERN, where they accelerate particles at great cost. Now we're kind of living in this world where, you know, $100,000,000,000 valuations have become boring for some of these new software companies. No one's that excited. They're interested when yet another AI scientist goes off to fund their own lab. But that seems like table stakes now. So these very, very large training runs are legitimate, in my opinion, scientific experiments. They're not just normal technological development. Now why did I talk so much about sort of the epistemology of this? Well, I think we have to be extremely cautious when making predictions. I don't think this is actually like Moore's Law at all. I think that we are continually surprised by what these systems do and do not do. There have been already so many deviations from the so called scaling laws that I think we have to admit that that was perhaps not quite the correct hypothesis. Right? To give an example, already, we're not seeing huge improvements in output for improvements in compute, but we're suddenly finding ways to reduce the amount of compute to get the outputs that we already had. So in a way, AI is becoming more compute efficient. That's not what people were predicting 6 months ago. That's not what people were predicting. Experts, by the way, or at least entrepreneurs, let's say, if we make a distinction between expert and entrepreneur 18 months ago. So truly, I think we do not understand the fundamental science of any of this. And I've tried, you know, in my own thinking, my research, both for personal interest and in a professional sense, deeply interviewing the best AI scientists I could find, and they have no consensus. They have no consensus with each other. So I think the enthusiasm over scaling laws was trying to invoke something like, Oh, we kind of understand the physics of semiconductors, so it's only a matter of Moore's Law as kind of like Intel's business plan. Right? And it's an economic and social and technological law. But here, we could encounter scientific barriers at any point. So, you know, there is no guarantee that simply throwing more compute will continue to work. We can expect more and more compute to be available. But perhaps as soon as there's enough data on what these training runs are doing, there will be some sort of scientific understanding of what this is. And I think that, you know, in computer science, there's always been this interesting question of how much useful computer science is there actually, and how much is it just engineering and just application of the computer science to new and new problems. Here, we have an open wide field of computer science. So this leads me sort of to the second point. Why is it important to carefully think about the theoretical underpinning, not just the empirical outcome of AI? Well, this is my theory of progress. I would literally not even track who has the most compute. I think that's important to know. You should have a little chart, little table with the best information you can find. You need to update it, not quite quarterly, but you need to update it every 6 months because it changes pretty fundamentally. And people are going in-depth, trying to find all of the compute they can, all of the GPUs that they can. They're trying to strike deals, favoritism, all of this corporate politics stuff, super interesting. At the end of the day, I'm not going to bet on the company that has raised most money to have the most compute. I'm going to bet on the company that has the best AI scientists. Right? So, Demis Hassabis, for example, at the end of the day, when you read what he writes, when you look at what he says, he ultimately wants to answer questions about the nature of intelligence. Ilya Sutzkevar, he almost has this religious air about him when he talks about the models. Because to him, this is about the beauty of the universe. This is about the beauty of intelligence. Right? This is about the discovery of it. And I think when you hear scientists talk this way about their craft, about their field, you realize that they have a very deep motivation. Now this motivation might be assisted by a financial motive, a profit making motive. But let's put it this way, if I see an AI company where everyone is just trying to make money, and, you know, it is possible to make money with AI. It's harder than people might think, by the way. Most of the AI companies are clearly losing a lot of money very quickly. So I expect most of these to fold within the next few years with the giants remaining because the giants just have way more capital to throw around. Like, whoever's working with Amazon, whoever's working with Microsoft, and, of course, Google itself, they're not gonna run out of money. Right? They're basically like Saudi Arabia. You know, Saudi Arabia can build a line in the desert. Right? That giant skyscraper that just goes to the desert, this pharaonic gesture. Well, Google can build, like, pyramids of compute, and it would still be among the most profitable companies in the world. Microsoft can build great cigarettes of compute, and they could still be a very successful company. So that's important to keep in mind, who has money to burn. But merely having money to burn does not produce the breakthroughs. The breakthroughs will be actually from who has the best AI scientists. And I do mean scientists. I do not mean software engineers. I don't mean data scientists. I mean people who you can figure out have both an intellectual excellence, an in-depth passion, and are not easily confused by marketing or social pressure. There has been in Western civilization a fundamental tension between science, engineering and commerce. The commercial mindset always involves talking up the current business plan, talking up the current approach to things. Science involves undermining it, right? Science involves questioning the current approach. Science involves disproving the current approach. So you can see how in a corporate culture, these 2 can be intention. And most people, also very intelligent people, if they go to elite universities, by default, they're trained in agreeability. The purpose of the educational system is to train you up in agreeability. We basically destroyed the Prussian Science University. It took a century for the destruction to happen. But when we made the Prussian research university model, it was adapted, adopted in The United States and adapted to scale to most of the population, it obviously can scale. So something that was designed to sort of protect your PhD autists ended up being a degree mill, and ended up being a necessary career step, not just here, but back in Germany, and in The United Kingdom, and in France, and actually now more and more so in China, If the talent distribution mechanism of your society is the same thing as the scientific mechanism of your society, that is also a contradiction, and that is a fundamental problem of Western civilization. We do not know what to replace universities with. So why am I talking about universities in the context of AI? Like I said, AI is a good test. You have to go deep. So many of these top AI scientists were poached from universities that had good computer science departments, but these weren't even first year universities. Can you name a brilliant AI scientist from Harvard? Has a background, did a PhD there or something. It's kind of interesting. You see that Canadian universities punch above their weight. MIT, well, that's pretty predictable. British universities, French universities, for some reason, French mathematicians seem to have a knack for it. Anyone who was trained in the former Soviet Union for some reason, you know, and by the way, that's not surprising. A lot of the mathematics underpinning modern models was actually developed in a compute starved environment as academic hobbies for mathematicians who are kind of trying to cope with the lack of good computers and trying to figure out new and more efficient ways to do things and working things out on paper rather than on huge training runs. Now that tradition has developed immensely in the 1990s and the 2000s after the fall of the Soviet Union, but it's notable that a lot of these, like, academic lineages go back there. So after the launch of ChatGPT, I think we have stripped Mind Academia for all the good AI scientists. I don't think you have any truly knowledgeable questioners of the nature of the universe, of the Ilya Sutzke Verteer left. I think they've all been struck mind. There was just too much money. There are mathematicians who, for ideological reasons, were not inherently interested in AI, but were scared of AI by basically Eliezer Yudkowsky and basically things around that. Arguably, Ternst Tau is in that category to some extent. And those mathematicians, I think, will make modest or decent progress. Because ultimately, if you have a holy terror of AI, which causes you to change your mathematics, what you apply your mathematical expertise to, I think it can make you a very good engineer, but it makes you like kind of a mid scientist. I think the curiosity, not the terror, is the actual underlying deep motivation, which causes these, like, very flawed human brains prone to various interesting cognitive flaws, like wishful thinking and so on, to actually end up locking on to truth. This means that if someone wanted to, they could compile a list of all the AI scientists that are likely to be paradigm breakers. You wouldn't get 100% of them, you could probably get 90% of them. And you can chart which company has the most. And at the end of the day, you can chart which company is likely to not be constrained by needing to develop a product. Let's remember that this breakthrough of application of ChatGPT came out of a company that people used to make fun of for releasing white papers instead of products. Like, this is important to remember before its memory hold, that OpenAI, you know, it wasn't just a corporate drama that came out of structuring it as a nonprofit. They basically gave the best AI researchers they could find tenure. Like, they basically were just publishing white papers and they were working very hard and they were motivated by this vision that AI will transform the world. They were recruited by literally Sam Altman and literally Elon Musk working together, like that's in the prehistory, including things like giving Tesla gifts to the early employees to encourage them, right, to go do that. And I think that cannot be recreated in the current environment. So it's sort of decaying. Right? And if you look at the key people like Dario Modet and like Ilya Sutzkever and, you know, maybe others in the future, this dream team has sort of split apart. Now, I think they have likely trained up significant human capital. So anyone who was working with them in that initial period before the entire planet decided OpenAI was a huge success. Those people, if they were AI scientists, not software engineers, I think they're underrated. And I think people should be trying to poach those because, you know, like I said, academia is strip mined, which places we're still producing and training people in the scientific mindset rather than an entrepreneurial mindset, a hype mindset.

Nathan Labenz: (21:29) Hey. We'll continue our interview in a moment after a word from our sponsors.

Samo Burja: (21:33) Let's be real. Like, I feel like generative AI is now as much of a buzzword as web 3 ever was. It's web 4, basically. Right? It's like what is supposed to save the balance sheets of all the investors and all the major corporations and actually BlackRock. And when you add it all together, you know, really The US has no plan for solvency except for an AI revolution. Right? Like that's the only way in which the Western world can achieve collectively 4 or 5% economic growth per year. Now, course, AI worked in this promised way, we wouldn't have 4 or 5% a year, we might have like 10% growth a year or 20% growth a year truly singularitarian numbers. But you know, if it works only the way it's working right now, I don't think there's any chance that this results in a 5 or 6% rate of GDP growth. There's no chance it automates labor fast enough to counteract the global labor crunch. And I don't think we are focusing enough on the science of artificial intelligence and asking the deepest questions and investing in the most esoteric mathematics. 1 way to put it is that imagine a world where it is easy to raise money to build particle accelerators and employ experimental physicists, but it is difficult to employ and protect for longer periods of intellectual development theoretical physicists. That is a world where we would hit a wall in physics very quickly. And I think artificial intelligence is much closer to physics than it is to, I don't know, fintech or Moore's Law or something like this. So that's my sort of contrarian thesis. We've undervalued AI science, we've overvalued AI compute, we overvalue raising huge amounts of money to like quickly enter the AI race and undervalued longer term plays to cultivate top AI talent that is intrinsically motivated by their curiosity to push forward this frontier of understanding. And I think it's just because a single AI product went viral. And here's the spiciest take I have. My spiciest take. Had OpenAI not gotten ChatGPT, had ChatGPT not been viral for whatever reason, probably 2 or 3 UX decisions could have killed it, to be honest. It would have enthused experts and some top programmers, but the general public would ignore it. I think we'd actually be making more theoretical progress because the dream team would still be together, and the intent would still be on the white papers, ironically. You should interrogate these 5 or 6 heresies because I don't think anyone talks about it this way. I've been, like, surveying the entire discourse landscape, and I see very few people arguing these points. So I'm happy to be challenged on them.

Nathan Labenz: (24:30) Yeah. I think there's you've touched on, like, 10 different things that could be full discussions unto themselves. Some of the ones that I think I agree with, assuming I'm interpreting you correctly, I always say the transformer is not the end of history.

Samo Burja: (24:45) Yes.

Nathan Labenz: (24:45) And it's important to remember that. You know, we've had for much of our lives the sort of end of history vibe, and, obviously, we're now shaken out of that. I think the AI community, maybe not the top thinkers, you know, not the the top scientists that you're talking about, but the community at large has experienced a somewhat similar thing where there's been this sort of, well, we just scale up the transformer and we'll keep doing that forever and that will deliver

Samo Burja: (25:09) Mhmm.

Nathan Labenz: (25:09) You know, the sort of transformative outcome. And I personally don't rule that out, but I expect that there will be more theoretical and like, we'll probably never find out because I think we will see more theoretical advances that will mean we're never gonna run the trillion dollar transformer experiment because there will just be other things that will be sort of better.

Samo Burja: (25:33) It's good to remember that diffusion models are what's generating all the images. It's not transformers. So what is it if transformers are fully general intelligence in the bedrock of it? Why are they so much worse to generating images than diffusion? I think that we are seeing the Cambrian explosion of the different definitions of intelligence, The Cambrian explosion with this event on prehistoric earth hundreds of millions of years ago, where suddenly there was a vast diversity of multicellular life that came to be. It's like visible in the fossil record. I think for most of our history, we have had only human and animal intelligence, and we added a tiny bit of computer intelligence, which I think is different in kind than human and animal intelligence. And now we have added even more forms of computer intelligence. Right? I think that already the Turing machine was already a form of artificial intelligence. I think we're eventually gonna have a periodic table of elements of different things that can be called intelligence that are fundamentally different processes. So I think we'll have qualitatively different processes that can produce outputs that we would call intelligence, but they will be fundamentally different processes with different properties, different strengths, different weaknesses. And I think that'll be surprising to us because we're used to thinking of intelligence as a single thing, because the evidence is pretty strong in human beings when you do psychometrics and cognitive science that humans are just kind of doing 1 thing super special. But, you know, maybe the universe doesn't work that way. Right? Maybe human intelligence was like the hydrogen atom, the first atom that comes into existence in the universe, but then very quickly the stars start making helium and iron, and then the supernova start making uranium and all these heavier elements that near star fusion can produce. So I really do think that we are going to be continuously surprised at the strength on reasonable effectiveness of various forms of intelligence as we discover them, but also surprised and shocked by their weakness. Right? Why do AI agents, when you put generative AI, when you put transformers on a loop, why do they melt down into incoherence and don't pursue goals and cannot actuate and cannot act on the world? Like, that's a surprising thing. I don't think people expected that. If you told someone 20 or 30 years the kind of verbal behavior this intelligence is capable of, they would assume it's trivial to use the words to command a robot and have the robot do things. Right? I think they would assume it's trivial, but it turns out it's not. What they would assume it's trivial to have a representative. Wasn't that the thesis of, like, a 100 startups 5 minutes after ChatGPT came out, the personal assistant, and they're just not you know, you can use them, but you can use them the way you can use a Calendly link. You can't use them the way you can work with an executive assistant, a human executive assistant.

Nathan Labenz: (28:39) I really agree with, and I think you're sharp to emphasize. I use the term Cambrian explosion sometimes as well. I also have started to use the term mixture of architectures era to signal sort of the end of the transformers era and the beginning. And I think we're just very much still in the beginning. But Yes. You know, folks who listen to my feed, you know, know that I have an obsession with state space models. And the big reason for that is that I see it as 1 of the first new core mechanisms that performs basically on par with the attention mechanism. But notably, as you said, does have different strengths and weaknesses, which you sort of have to drill down to a very small level to really distinguish. It's quite interesting that at the high level when they train, like, a state space model versus a transformer model at scale, they look pretty similar. Like, qualitatively, you can chat with them. They seem to have reasonable memory. When you drill down into the microscales, you find that there are kind of 2 sides of a coin sort of different strengths and weaknesses. And not surprisingly based on that, some of the most exciting new things coming out these days are hybrid structures where you have an integration of these things. When I just think about and I'm no expert in neuroscience, but when I just look at sort of a picture of a brain in an anatomy textbook, I sure think, you know, yes, there's, like, some parts of this that are kind of highly repeated. You've got your kind of cortical column that's, like, definitely a main piece that is repeated a lot and doing a lot of work. But we also see just a lot of other parts. And so it does seem like we're headed for I think a periodic table is a pretty good mental model for it. We'll have a lot of these different specialized, you know, not even full architectures, but components that you can kind of snap together in in all sorts of different ways. Going back to just the sort of how you know, let's say, Eric always asks questions in this way, so I'll do the same. If you're advising a national government, you know, president Harris comes to you and says, okay. You know, a lot of hype out there. Obviously, though, lot of undeniable progress too. How should I think about how extreme things could be? Like, I hear you that the agents aren't quite ready and that, like, GPT-four was still kind of a slog to implement and probably is not gonna be transformative, but there's a lot of buzz about a possible GPT-five and the GPT-six behind that. How would you kind of characterize, you know, for somebody that doesn't have time to get into the science, like, what the sort of range of plausible outcomes might be or, like, what they need to be prepared for?

Hey. We'll continue our interview in a moment after a word from our sponsors.

Samo Burja: (31:16) Well, I think the possibility of a compute race should be kept in mind, but a compute race is not a software company race. It is a hardware race. So if you want to preserve The US capacity to engage in a hardware race, I would advise the president and especially, you know, in an ironic twist, maybe former president Trump has already received this advice because recently he went on air saying, oh, we're gonna need to double the energy supply for compute. I'm like, wait. Where did you get that talking point? I don't think he read about it. I don't think he asked Chatuchipiti. Someone who kind of knows what they're talking about must have spoken to him. So I would say to president Harris, if Kamala becomes president Harris, I would say that you will have to pursue an industrial policy on semiconductors and an industrial policy on energy. And the thing that my libertarian friends will most disagree with me, I'll just say it will require government action. Not just the government getting out of the way, but the government buying compute in America. Even TSMC cannot run a chip fab in America and have it be as profitable. No matter what they do, the unit costs are 50% higher, like Morris is on record for having said that. So if TSMC can't do that, how could Intel? And if we're doing free trade even with our allies, this means the second largest chip fab might be in America, but not the largest. The free market will not provide. So I think the answer has to be something extremely politically difficult. I think we need to create a new Silicon Valley. Let's remember Silicon Valley originally meant Silicon. Right? It it meant hardware. Right? When I saw you know, there's a wired article a few years ago, it was titled Shenzhen is the Silicon Valley of hardware. I was like, I I felt like, oh, drop. I was like, oh, no. Oh, no. The Valley has no idea how much trouble they're in. And my friends at the time made fun of me because I was saying no, no, you don't understand. If they dominate hardware so thoroughly, they will easily reverse engineer software too. And they're going to have their own unicorns, and they're going to have their own software giants. Well, my friends don't find it so strange. My clients don't find it so strange either now that we are discussing whether to ban TikTok. Right? Like, obviously, China can build software startups now, and no one's wondering that. Obviously, Baidu is honestly it is about as good as Google, actually. They're, like, keeping pace not just with the stuff we did 3 or 4 years ago. They're keeping pace with things like self driving cars and driving their own progress just as fast. So, really, the hardware thing is going to be difficult. It's going to be difficult because you won't be able to do it like NASA. The political economy of NASA is that NASA has to do something in all 50 states because otherwise it can't protect its budget. It has to give something to every single state or states will go against it. However, that's not a good way to build rockets. Right? SpaceX doesn't have to have a facility in all 50 states. Right? SpaceX's biggest client, of course, you know, I think it remains the government, though Starlink is genuinely going to be a very different model. But I think there has to be a move to do the politically difficult thing of designating a zone, probably a port city, and it doesn't have to be, by the way, the Bay Area. Maybe California is actually difficult to deal with, and you wanna do it in a different state. There has to be a federal designation. Get every single regulation out of the way. You know, the Arizona plant and the delays there, you've heard of that?

Nathan Labenz: (35:09) I've heard a bit about it for sure. I know the Yeah. Daycare requirements and so on and

Samo Burja: (35:14) so forth. Yeah. Much of it is stuff that has nothing to do with manufacturing the chips. It's literally like, how do we comply with the building codes? I'm like, well, what if we didn't worry about the building codes and just build the chip fab as soon as possible? Right? It would take that kind of mentality where you create an American Special Economic Zone and you demand, you buy a lot of compute there. Honestly, given the density of data centers in Virginia, heck, maybe you do it in Virginia. Maybe you go retro, you put it on the East Coast. I'm not sure if Virginia would play nice with that, but maybe it would. Maybe the federal government partners up with that 1 individual state. And then we build full stack, somewhat subsidized, somewhat expensive chip fab set up there, and then we have the capacity to win a hardware race. Let's be real. We shouldn't be seeking a hardware race with China because we'll lose it. If it was actually an existential military exercise, China can destroy Taiwan. It doesn't need to take Taiwan. It can destroy Taiwan and can make it hell to get ships out of there with chips. And what are you gonna do then? Are you gonna invade the Chinese Mainland? Like, what the hell? Like, are you gonna nuke China? If you do that, China's gonna nuke you right back. The people who are calling, you know, shout out to Leopold. Right? Like, the people who are calling for this, like, massive buildup, I'm like, no. No, buddy. We we have to be much slower with this. If America wants to win an industrial race, it just needs some industry first. And America's industry is like, it's pretty good. But the fact that you know, let's put it this way. America is not the industrial peer of China. It is the industrial peer of Germany. So every single thing you can say about the German economy and how it sucks and how they've been deindustrializing, well, guess what? We can say that about American industry. It's just that America is not just a Germany. It also has like a giant software sector. It has finance, it has service sector, all of this other stuff that Germany has less of. So we sometimes forget that, you know, when we look at these GDP numbers, it's not really American manufacturing driving those. And American manufacturing does not outperform Dutch or German or South Korean or Japanese manufacturing. That's why ASML is in The Netherlands. Okay. That's why Samsung is like a Korean company, not an American company. So that's what has to be dealt with. So because of that, I actually think it's a hopeful message geopolitically that this might be yet a scientific race. However, I'm sure that if the intelligence explosion theories are correct, that there's like a key event after which there's a beautiful feedback loop that then results in kind of a single singleton style future where whoever starts this will just be so far ahead exponentially every few years, they'll be hopeless to catch up. We we have to preserve industrial capacity for that era because at the end of the day, it's not like it's easy to coordinate allies. So even French shoring does not fully work. And, you know, I'm not sure which president has the capital, the political capital to push for a functional industrial policy rather than, like, a pretend industrial policy where we have to do something in all 50 states.

Nathan Labenz: (38:37) Yeah. I wanna get into the chip ban in a second. And Sure. I definitely

Samo Burja: (38:43) By the way, the chip ban is, of course, a subsidy for American companies too. When I say subsidy, I don't just mean literally giving them 1000000000000 dollars, though, hey, maybe we should try that. Right? Do you remember that infrastructure bill? Can anyone point to a single bridge built from the trillion dollars of the infrastructure bill? A single bridge, right? We could have 1000000000000 dollar chip bill, and maybe even some chips would get built. Who knows?

Nathan Labenz: (39:09) Yeah. I mean so I'm very allergic to, I guess, the whole US China conflict. I mean, I we could do hours probably on this from kinda beginning to end. But in brief, 5

Samo Burja: (39:22) g 1 economic competition with China and stop emphasizing the geopolitical angle because that's really how China thinks of it. They think that they will win the geopolitics through winning the economics. Like, every single part Chinese party document that comes out, you know, they talk a little bit about Belt and Road, but they sort of know it doesn't work. What they really want is to be the economic hegemon. And because they're orthodox Marxists, if you you know, here's a funny 1. They were such orthodox Marxists that they decided to own the means of production. And they thought that results in global dominance by literally owning the factories that build all the stuff. And they weren't averse to using free market capitalism to get to the point where China owns the means of production. So I think that's their theory of geopolitics, and they don't think about it as this, like, imperial way that we in the West think of it. The Chinese approach to dominance, it's more like it's not that you go out to the periphery and you conquer all the periphery, which we could say is the Roman way of building an empire. It's more the you consolidate and unify China, and then you outshine and extract tribute from all of the periphery. That's the Chinese model of empire historically. And I think these different set of assumptions are there. Right? Like, for everything we can say about China is like, can we name a single country that China has invaded since their last war with Vietnam? Like, that's been over half a century ago. Right? Like, it's been a long time. The last place they annexed, you could argue maybe it's Hong Kong, but really, was Tibet. And from their perspective, Tibet, Hong Kong, Taiwan is just unifying all the Chinese lands. So I think the economic competition and having a smarter trade policy, I think that is the right approach to think about China and, like, militarizing it or, like, having a rhetorical point where you emphasize the national security angle, I think that is likelier to cause them to also build up military capacity. Right? And I think The US in the 20 first century should strap in for the long game of beating China economically over the next 50 years, and it will take 50 years. China will not implode in 5 years. It will not implode in 15 years. It'll take 50 years to beat China economically and once more manufacture high technology here in The United States rather than in East Asia.

Nathan Labenz: (41:58) So before going a little bit deeper on China, I just wanted to zoom out for a second and ask if you see any other live players on the global stage that you think we should factor into this analysis. I look around at The US. I agree with your general sense that, like, most of the second tier of AI companies, even those that have raised in the low billions, they seem to be numbered. You know, that that used to be sort of a spicy take to borrow your term. But increasingly, as these sort of acquihire type deals are happening, it's actually now becoming, I think, sort of consensus. That leaves us with maybe, like, 5 companies at the top depending on exactly how you'd wanna count. I would certainly still put OpenAI and Anthropic and Google in the top 3 just by the kind of sheer excellence of the science. Obviously, they've scaled, but they seem to have a know how that is kind of still cut above.

Samo Burja: (42:54) Mhmm.

Nathan Labenz: (42:55) Meta has definitely crashed that party in a meaningful way, although I haven't put them quite at the top tier. And then I would never count out an Elon company, so I definitely would put an x AI slash Tesla whatever in that same

Samo Burja: (43:09) Category.

Nathan Labenz: (43:10) Yeah. Same let's call that the top 5. When I look around the rest of the world, you know, then I sort of have China as, like, a bucket. And then maybe we could see a European champion. There's Mistral. You know, Europe maybe wants a European champion, but, obviously, they're a bit of a harder environment to do business in, and it's unclear exactly how that's gonna play out. Maybe somebody from India, maybe somebody else I'm not thinking about. Any other you know, before we zoom back in on China, which I think is inevitably where we're going, is there anybody else that you would say really has to be watched on the global game board?

Samo Burja: (43:46) I think not quite yet, but I think 10 to 15 years from now, not 10, maybe 5 years from now, 10 years from now, there will be a formidable Indian giant. I think that India is very likely to develop its own software ecosystem. And if they decide to eventually go the route of favoring Indian companies over Western companies, you know, that's 1 and a half billion users right there. That's large enough. That's the same sort of initial captive audience that Chinese companies enjoy and that actually even a Russian company, Yandex, enjoys. Right? Yandex kind of is the Russian software conglomerate. They basically do almost everything that Western software companies do, including having a self driving program of their own. They're not leaders in any sense, but they demonstrate that even if you have like a very small market of users, which honestly, like a 100,000,000 users is really not that much. But if it's behind a digital wall or a regulatory wall, well, then you can do very similar things to Silicon Valley even if you have much less money than Silicon Valley companies. You can do for a few billion, but in Silicon Valley is done for tens of billions or hundreds of billions. And I think the Indian situation will be very similar. Something will happen there. I think for Europe, I'm more bearish on Europe after Macron stepped down because I think Emmanuel Macron was personally convinced of the importance of artificial intelligence, had personal contacts with Silicon Valley, championed and fay pushed against German regulators, ironically, using French pull and wait to have French diplomats argue against regulation. You know, think how dire it is if if French diplomats are going around Europe arguing against AI regulation while the Belgians and the Germans are going wild and having fun restraining them. And it went as far as the French state was trying to do the best it could to recruit people into Paris. This was a smart strategy. With the exit of London from the European Union, Paris could have become the startup capital of Europe. Now if a future French government picks this up, and let me explain a little bit. France still has excellent mathematics education. They still have lots of field medalists and so on. You have the raw talent there. Right? So they don't need to poach from MIT. They have their own mini MIT. So they have a university there. It's a big city, so you don't feel like you're in a backwater. And if the French government gets off your back, you're not even getting in trouble regulatory wise. But in theory, you could actually do a startup if the French government got out of your way and gave you some tax breaks and all of that stuff, and also actively invested into seeding an ecosystem there. So I think it remains an option for the next 4 to 5 years for French government to continue this strategy. I'm just not sure if they will. Right?

Nathan Labenz: (47:01) Okay. So then that brings us back to US and China. I would love to get a maybe the brief version of how you understand why The US and China are at such odds in the first place. Mhmm. I don't have a great account of it. You know, start like you with a sort of naive observation that, like, we're on opposite sides of the globe. And it seems like, you know, neither 1 is a as you said, like, we're not gonna invade the Chinese Mainland, and they're not gonna invade North America. So we should seemingly be able to coexist each comfortable in the fact that we've got these huge ocean buffers. And yet all I hear, I feel these days, is out of, like, the DC establishment in general, China this, China that, China, China, China. And it's also now moving into the AI discourse in a way that I think is quite unhealthy where it's like, well, if we don't race to develop the most powerful technology that we can as fast as we can, then we're just seeding the future to China, and we're gonna have to live under Chinese values. And I swear to god, people who say that, I I always wanna know, like, what is your definition of Chinese values that you are, you know, expecting us to be living under? Because I think that's just often such a vapid comment. But, anyway, why do we have this? And I guess the next question that you can maybe carry on into is, is there any way for us to avoid the AI arms race or AI cold war with China? The chip ban, to put my cards on the table, seems like a step in the wrong direction in that it can only be interpreted from their side as, at best, an attempt to keep them down and, at worst, an attempt to create decisive strategic dominance over them. And, obviously, neither 1 sounds very good from their perspective. So, yeah, why do we have this? Is there any way we can avoid an arms race? And what would you you know, to to maybe go back to your strategic advising frame, what would you advise the next president to do about the chip ban?

Samo Burja: (49:08) You know, I think the chip ban is too little, too late. I think that had it been something that was pursued in, like, 2010, China could have been taken off the track of developing cutting edge chips. What the chip ban has done now, it's not sanctioned China really. It's given a subsidy to Chinese chip manufacturers. So if you can't buy the best NVIDIA chip or if you can only buy a nerfed version of it, well, guess what? A Chinese chip designer or manufacturer only has to compete with the nerfed version, not your cutting edge version. They just need to do the nerfed version better, slightly better. And a Chinese giant or company becomes viable because there is enough Chinese commercial demand for GPUs. Right? They have a huge tech ecosystem, like dozens of huge companies. They're equivalent to Google. They're equivalent to Amazon. They're equivalent to OpenAI. Right? They're equivalent to Waymo. They're equivalent of Meta. They have their own demand for this, and the government in China hasn't really even started buying GPUs at all. If they wanted to, they could make it a national priority. And we've basically announced, hey. It's a national priority for you to develop your own high end chips because we're not gonna sell them to you not now and not in the future. And the Chinese government is rich enough to afford that. I think people really don't understand that America is not the only country in the world that can drop 1000000000000 dollars on a serious problem anymore. China could drop 1000000000000 dollars on this problem if they choose to, or a 100,000,000,000 or 50,000,000,000. Right? In The US context, The US has not dropped 1000000000000 dollars on this. It's only dropped a 100,000,000,000. We dropped a 100,000,000,000 on it, and we get an Arizona chip fab. Meanwhile, we drop 1000000000000 dollars on building bridges, and it's not clear we built any bridges. So I find that super funny. Anyway, the future here will rely on a general industrial revival in The US, where I do think the energy angle is the right 1. I think we should seek to radically increase energy production, electricity production in The US, because then it makes sense to run data centers in The US. And energy is 1 of the few things where labor costs are not a big cost in producing electricity. Right? So I think that would be a good medium term advantage. And I think that also the scientific question is an interesting 1 as well. I do think we have a scientific advantage over China, but not a technological advantage. So I think something very good can be done in terms of smarter scientific grants, even if the Chinese inevitably copy whatever the science discovers or produces a few years later. I think that 1 of The US advantages has been DARPA. And having a good understanding of how DARPA works, well, maybe you should have a DARPA focused exclusively on artificial intelligence. And you have to staff it with ideologically diverse people. So don't just hire all the EAs. Even if Jason Matheny is listening to this podcast, you know, the current CEO of Rand and Holden Karnovsky and all of the EA royalty, they can certainly have a few program directors at the AI DARPA. They just should not be allowed to appoint all of them. So that my other advice to the president would be, you know, madam president or mister president, whatever it might be, you know, we have to have ideological diversity because this is no longer just a technology. People are pursuing their end of the world cosmologies here. It's like, you know, sure, the Manhattan Project can be 50% Marxist, but maybe keep it just 50% Marxist. So similarly, the AI DARPA, maybe it can be 50% effective altruist, but keep it at 50%. Okay? Not to pick on EA, but they're like a big you you also wouldn't want it to be more than 50 EAC, whatever EAC even is ideologically. Right? It's just the case that a lot of people will pay lip service to The United States, but ultimately, if you look at their ideology look. If you have a utilitarian ideology, you are not always a friend of The US. You are mostly a friend of The US, but you're supposed to be thinking globally and the future light cone and all of that stuff. You are incidentally not fundamentally patriotic. And I don't wanna overburden the patriotism because patriots can be dumb. Patriots often do very noble and excellent things, but it's very easy to weaponize patriots against free speech. Like, that's happened many times in the history of many countries where countries embark on a stupid course of action because proposing the smart 1 sounds like a little bit less nationalist than the alternative. Hey. That's why the empire of Japan failed. It was too unpatriotic to suggest that maybe Japan cannot beat The United States and maybe Japan should not bomb or destroy US ships. Right? That was just unpatriotic, and, you know, the outcome was not 1 that I think Chinese nationalists appreciated when the full course of history ran. So let's put it this way. I think currently The United States is it has to play a very careful game because it is not the world's industrial giant as it was. So the game has to be scientific, low energy costs, very smart government spending. We'll have to have much smarter government spending on strategically relevant science. Not more of it, smarter. Because the way we're currently spending scientific money is that we're literally giving the Wuhan Institute of Virology money to produce new plagues for the world. That's kind of insane. Right? That was partially US funded. That, mind you, is now well documented. It's long ago, left the Alex Jones or even Joe Rogan world. It's now just congressional testimony world.

Nathan Labenz: (55:21) So lot of trends there that I wanna pull on.

Samo Burja: (55:24) Again, I I invite pushback. I put evocative things there because we need to have real and honest conversations.

Nathan Labenz: (55:29) So I just so I I'm broadly very much with you when it comes to, okay. Let's get our own house in order. Let's, like, throw up some nuclear plants. You get the energy, you know, flowing more freely. All that stuff sounds great. When it comes to the I do understand that Japan in a similar way that, like, we're inviting them to develop a strong industry, and I don't doubt their ability to do it. It may take some time. I don't know how long it's gonna take, but, you know, I always say I watched them throw up a hospital in a handful of days, you know, at 1 point. So I don't wanna be caught underestimating their ability to create a chip industry. I also think there's just an interesting question of sort of do we want to be developing this technology in tandem, or do we want to be running on sort of parallel tracks and having, like, secrets from each other and all of the things that that implies? I don't necessarily know that you meant to imply this with DARPA. But when I think DARPA, I sort of think, like, secret technology development. Some of that stuff obviously finds its way into the broader world, but some of it is secret.

Samo Burja: (56:32) I think the US government kind of sucks at developing secret technology. I think The US has great strength in its commercial sector, which is by its nature public. Even if you have trade secrets. Right? You know, NVIDIA has trade secrets. TSMC has trade secrets. Secrecy in that sense makes sense. Secrecy in a top classified you know, top secret classified something. Well, let's say I think the World War II era US government could do that well. We are not dealing with a World War II era government. If you declare something classified and top secret, the result will not be a technological breakthrough. It will just be waste shielded from oversight. So I think that there can be clandestine projects. It's sort of like, well, how exactly we apply artificial intelligence to the NSA's work, okay, that can be secret, but commercialization and by the way, the intelligence world has done this through institutions like In Q Tel, right? Any technology you can declassify and hand over to Silicon Valley or America's commercial sector, you should. You should just do as much of that as possible, even if it means that China catches up to Silicon Valley 3 or 4 years from now because of what you declassify to Silicon Valley, because the true advantage of The United States is its vast wealth and the speed of its scientific progress. And when you make something secret, you stall scientific progress. You don't accelerate it.

Nathan Labenz: (58:03) I also worry that this combination of trying to deny chip access, which is not gonna work entirely, but, you know, may make for at least some asymmetry. You know, China might be looking at a situation where they're saying, jeez, we can only get to 20% the compute capacity that The United States can get to. And also they've got some secret military science programs that we don't have much of course, they might hack into those, you know, perfectly capably. Who knows? But let's say they are also, like, genuinely worried that there are secrets that they are not able to learn. It worries me that that might send them down a totally different branch of the technology tree. And when I think about, like, how do we get into an arms race that would be a really bad situation versus how do we avoid it? 1 key factor there seems to be, are we kind of on the same general technology path such that there is kind of a shared understanding of what everybody's working with versus to what degree do we look at the other side as kind of the great unknown that we can project our fears onto and then motivate ourselves to, you know, take higher risks, be more militaristic in our technology development.

Samo Burja: (59:17) Historically, that's always been the case. Right? Even when The US and the USSR signed a treaty banning biological weapons development in the late Cold War, Both side scientists were, of course, told that now you should get to work because, of course, the other side is in this treaty, and we don't know what they're doing. So you best develop the worst plagues you can think of. And even during World War II, the fact that Los Alamos scientists greatly overestimated the state of German nuclear research is what allowed the project to be completed before the end of the war and be used in Japan. I think the weaponization of AI is, like, troubling. And I think part of it is that, you know, scientists are like everyone else. On some level, they think weapons are cool, and they wanna develop cool weapons. So if they find a way to make physics a weapon, people get very excited. And I think this is 1 of the problems where we crave peace as a species, but we also crave violence. And I think a lot of the time, we should actually be very mindful of our motivations for developing breakthroughs. I think that it is best if artificial intelligence is a peaceful technology for 2 reasons. First, we do invite existential risk, like and human extinction into the equation if we're trying to develop literally weaponized or malevolent AI. Dude, Skynet is not just a movie. It's a Pentagon pitch deck at this point. Right? Like, if you could deliver half of what Terminator films depict, if you could deliver half of that, the Pentagon would just give you 1000000000000 dollars, like, right away. Right? Skynet, but a pitch deck would be you know, that'll probably work. But that's, like, kind of we we're developing the worst technology first when we think of it in a weapons context. We will not be prioritizing safety, and we will be teaching in artificial intelligence to win at 0 sum and adversarial games and to win at logistical and military games. And I think it's completely predictable that if you could have an agentic artificial intelligence that decides humanity is its enemy. Like, that's literally the plot of Terminator. And all of the AI safety people made fun of it for decades. But if you just look at how the military world is perceiving AI, you realize that that's, like, actually extremely bad. And if you make it a geopolitical conflict, the generals will ask you to just build Skynet, please. Right? They will just ask you to do that. Don't. Don't build Skynet, bro. Just don't. I know you think it's cool. I know it feels really good, but you're gonna regret it just as Oppenheimer regretted. Now after he got the high of the detonation, then he regretted it. And that's the hypocrisy of human nature. But, you know, hopefully, we can grow from that. So let's not make it cool that way. Let's not fetishize the Manhattan Project. Let's not do all of that. And instead, let us be curious explorers of the universe and figure out what is the nature of these new forms of intelligence we're discovering, and let's apply them to our economy to compete with the Chinese economy in who can get richer faster.

Nathan Labenz: (1:02:42) Yeah. I love that vision. Just to put 1 little bit, we're gonna have a full episode on this coming soon, but a little bit of meat on the bone on you know, because I think some people might be thinking, okay. Sure. But, like, what does that actually mean to have, like, weaponized AI versus non weaponized AI? 1 proposal for AI safety that I just have been reading through is to train AIs so that and in general, you have these loss functions, right, in the training process where there's some objective that you're being optimized, you, the AI, are being optimized toward. And a huge trend in general in the literature is to have a supplemental element to that loss function that in addition to, you know, so you have the 1 component of the loss function that's can you do the task and then there's some other thing that's like, but we also wanna emphasize this. And that that could be like sparsity for efficiency or for interpretability or what have you. So this really interesting 1 that I just came across was about minimizing self other distinction and trying to create an AI that had the same internal state regardless of whether it was referring to itself or referring to someone else. And this is a very early proposal, but they've come up with all these different scenarios in which they think there's really no need for the AI to have a different representation of these concepts just because it's referring to itself versus someone else. So let's try to minimize those differences. And they've been able to show in sim very simple reinforcement learning scenarios that applying this technique can be the difference between agents emerging that deceive other agents or that don't deceive other agents. And I think this is, like, 1 of the best ideas that I've heard in a while for how to create AIs that are sort of the, like, kumbaya, you know, always 1. We're all sharing prosperity sort of mindset that we might hope to have. But, obviously, that's gonna be, like, a real problem if the goal of the AI is to, like, distinguish between, you know, friend and foe on the battlefield. So or at least it seems like those things might be quite incompatible. So I I do think that there is, like, real we're so early in it. We've just kind of just got these things to work. Right? We've just ended the transformer for everything era, and we're just barely getting into the lots of different components to make tons of different architectures era. And we don't know that much about this stuff yet, but it sure seems like we could optimize for telling friend from foe and, you know, killing foes, or we could take some more time and be more creative, like minimizing self other distinction. And that just seems like such a much better idea to me that I would really love to see us go down that path. So the Leopold plan, I wanna kind of give you a chance to maybe articulate the Samoa plan. The Leopold plan, in brief as I understand it, is we should do everything we can to keep China down for the time being. So if we have a couple years lead, we should harden our defenses. We should get the national security establishment and the AI labs into some sort of public private fusion partnership. You know, we'll keep all the secrets from China that way. They'll be denied the chips. Then because we are the good guys, of course, we will use this lead that we have to solve alignment. Obviously, details will be filled in later. I I honestly don't really like it. And then once we've done that, once we can then approach China from a position of decisive strategic advantage, then because, again, we are the good guys, we can approach China with the offer of benefit sharing. And then we can sort of maintain American hegemony and American, you know, western civilizational values will prevail, and China will kind of recognize that better to take the deal at that point than to, you know, what alternative will they have. We'll have superintelligence. How would you what's your alternative vision for how leadership can conceive of a plan for the, you know, next however many years in AI that would contrast against that?

Samo Burja: (1:06:59) Let me start at the end of the plan and work backwards. The US cannot credibly offer benefits sharing of this kind. The US allowed Germany and Japan to rebuild their economies after World War 2 because it was interested in the containment of the Soviet Union, and the Soviet Union was a threat. Had that not been the case, perhaps Germany would have been deindustrialized in 1946 rather than 2024, which, mind you, is partially The US is doing through the destruction of Nord Stream. I'm just happy to say it's overwhelmingly clear The US did that. The second step. So the offer cannot be credibly made. Therefore, the Chinese will not accept it. A step before then, we solve alignment because who is we? Who is we? What set of institutional or political forces in the US government would result in an accurate scientific understanding of alignment, assuming that is a coherent problem, and make rational bureaucratic committee driven decisions in line with that. I mean, sure, if we appoint a benevolent dictator of AI, which, of course, if we take the premise is a benevolent dictator of The United States and of the planet. So, you know, so much for American freedom loving values. Perhaps that is the true convergence all along. And then we go a step earlier where it's sort of, you know, this this kind of hardening. The only way you can truly make the American technology stack secure against China is to make it secure against US citizens. So the idea of an open digital society, a creative digital society. I know open source. Know, people talk about open sourcing the weights, but I feel the vision of the social revolution that was supposed to happen because of the digital revolution is so lost at this point. I think we all want to have a closed Internet. We want to have a monitored Internet. We want to have a censored Internet. Like, that's what it would require. Do you scrub AI leaks? Leaks would happen. There would be an Edward Snowden of AI. Would you scrub that from the Western Internet, perhaps using AI to hunt down all instances and mentions of it? Like, our problem is without an open source, open access citizen participation vision of our Internet. If as soon as we abandon that, we actually go on the treadmill first towards being like the EU, and then even worse, being like China. I've written an article on this. Unfortunately, this article has aged well. It is called the centralized Internet is inevitable. And for example, the only break in the escalating trend towards social media censorship that is suggested, and by the way, this is documented by things like WikiLeaks and so on, There are various parts of the US government just to suggest, oh, man. Sure is a nice software monopoly you got here. It would be a shame if someone antitrusted it. Can you remove the social media post? Or can you unbank this person, this Alex Jones person? He's a problem. Do you remember that happened in 1 day? All the major social media networks decided to deplatform him. I don't like Alex Jones, but that should have been a red flag for the state of Internet freedom. And as soon as we abandon sort of the open source vision, I'm open to us developing a new vision of what the Internet could be. But let's not pretend that we're not slow walking into digital totalitarianism of our own kind. When I make a China comparison, it is a warning. It is not praise. Right? So I'm not praising The US when I say that The US Internet every year gets closer and closer to the Chinese Internet. Or to put it differently, if 1 were to speak their mind politically on socially taboo topics in The US today, I think your business gets in about as much trouble as it does in China. And perhaps the only difference is you personally are not arrested, you are just made unhirable, and maybe you're put on an intelligence agency watch list. And this happened to many concrete documented cases of US citizens in a way that's unconstitutional, in a way that should be corrected. So I'm just going to talk a little bit about that. So the hardening can't happen if the civilian infrastructure is built to be open. So I'm open to hardening the infrastructure. We should not be enabling, at least for the military side of things, to be accessed. However, the commercial side kind of has to be open or it becomes structurally tyrannical. So we have to figure out how to thread that needle. What kind of security will we have on the base layer of our economy, society, politics and communication? And like, how can we do that? And I know many people have many disagreements, but I'm talking here broadly more about the institutional incentives of the companies that provide us our very tools of thought. At this point, we are cognitively outsourcing as individuals. We're cognitively outsourcing a lot to corporations, and it is extremely risky to push a closer corporate government partnership than already exists because then who shall restrain it in a realpolitik way? Who will restrain these organizations and the government from being tyrannical in a way that is illegal? You can't just say, oh, that's illegal. You can't do that. Law has to be backed by power, and usually it's backed by a balance of different powerful actors where law is sort of the operating protocol between them. Right? So rule of law could be destroyed through significant centralization of the digital space. In fact, we saw this in Canada. In Canada, if you gave $50 to the truckers' revolt, you were unbanked. Your transaction was tracked, and your bank account was canceled. Okay? Like, that's a level of tyranny I've not actually heard of China do. I've heard China locking up hundreds of thousands of people, but I've not heard them tracing, like, individual bank transactions to find every single small time donor who gave money to Falun Gong or something like that. I've not heard of China doing that. I've heard of Canada doing that. So I think we give a pass to Western totalitarianism too often. Still, Chinese totalitarianism is worse, but that's not what we should be competing on. So that was a bit of a tangent, but let's go to the earlier stages of the plan. I don't think we can prevent the rise of China, and I think I probably just disagree on AI timelines. If what Leopold was saying on the earlier steps that we should try to slow down China, If you think superintelligence in this AGI sense is like 5 years away, then okay, maybe that makes sense. But if you think it's 20 years away or 200 years away, then it makes no sense. And I don't think it's 5 years away. I'm happy to bet with anyone that it's not 5 years away. So because of that, I think we should not be escalating geopolitical competition with China, and we should instead be pursuing a mercantilist trade policy. We should be thinking of it in terms of developing The US sector and US economy, and we should justify it in that language, and it should be a set of give and take negotiations with China over splitting the global economic pie. It should not be a set of give and take negotiations over splitting the light cone. It shouldn't. That invites the wrong kind of logic and invites the wrong kind of bureaucratic incentives. Like, it's almost like these people have never seen how bureaucracies work, either the large corporate environment, let alone a government environment. You have to be extremely careful. The first misaligned artificial intelligence was arguably bureaucracy. Let's prompt engineer it really, really carefully.

Nathan Labenz: (1:15:03) It's poetic. I like it. What do you think is the I have maybe 3 3 and a half more questions if you have that much time. 1 is gonna be, how do you understand the role of open source in this situation? That is at least at the moment, you know, a sort of irrevocable benefit sharing proposal among other things.

Samo Burja: (1:15:22) Yep.

Nathan Labenz: (1:15:23) Then though I also wonder let's say we do, you know, continue to see more and more powerful AI and we start to get to a point where it's like, actually, this is starting to look dangerous. Is there any sort of precedent that we should look back to or any framework that you would look to for how do we set up global coordination and containment of the technology development? Another 1 was gonna be on sovereign AI. This 1 probably save for the future, but if you're like any of the other 100 plus countries in the world beyond the 2 that we've spent most of our time talking about, there's this notion that like we should have our own models or we should take a base model maybe from Meta and train it on our bureaucratic information or whatever. I wonder if you think there's any value in that. And the final 1 personal angle is to what degree do you use generative AI in your personal work today and and what sort of value, if any, are you getting from it? So you can tap into 1 of your best or pick 1 or you just come back next time.

Samo Burja: (1:16:22) I'll answer. I'll answer. I check-in every now and then and use generative AI to see where it is. It's not really a big part of my workflow because it usually provides me the consensus or the general thinking in a field, and almost all of my work is trying to find spots where the economic or political consensus or industry consensus is flawed or overlooking things. So in a way, high schooler interns not get useful. Maybe I'll check-in when it can give me a Harvard intern who honestly, between you and me, are not that useful either. You kinda have to be even pickier than that if you're doing research. So I don't think it is useful for novel social science research yet. And because of that, I don't use it. And stylistically, sometimes it writes, puts things into words about as well as I do. The problem is it does it in a recognizable way. So I can't even use it in my personal writing because it can't really do my voice. And, you know, I can always tell since 1 of my part time 1 of the hats I wear, I say occasionally I'm an editor for other people's writing. Right? For a lady magazine, I sit on the editorial board. I can always tell when they use AI, and almost always, it makes the writing worse. Because just as people have become sensitized to so called AI slop in a visual sense, people have become sensitized to reading stuff that sounds like AI. I've actually had to change my own style to be a little bit different so that people aren't confused about how this text is generated. So, yeah, I think it's not even useful for all white collar work. Not at all. I'm sure it can write probably a good rescheduling email. I think that's about the level where if Google makes a feature where they, like, tweak or improve the UI with, you know, my inbox, like, get the bed suggested text a little bit better, maybe I'll integrate in my workflow there. But, yeah, not yet making use of the available intelligence. It seems too low grade.

Nathan Labenz: (1:18:34) Well, we will continue to watch that closely, and I'll share a couple ideas with you perhaps offline. I'm working right now on fine tuning some of the latest models to write as me. And I haven't completed this project yet, so I don't know how well

Samo Burja: (1:18:48) it's gonna work.

Nathan Labenz: (1:18:50) In all previous generations that I've tried to fine tune, it did not work. But there have been some notable improvements in the capability, actually, most importantly, probably the context window because I found in the fine tuning, it could pick up my style okay, but it couldn't learn the facts, and it it still didn't have

Samo Burja: (1:19:07) unique any ideas or theories and so on. And perhaps with the larger context window, it'll be capable of it. Yeah.

Nathan Labenz: (1:19:13) The hope is I can stuff all my stuff in there, and it might not have to hallucinate anymore for starters. I think I will still have to give it, like, the nudge the kernel of the original idea that I have in the moment. I don't think it's gonna, at the current generation, substitute for me there. Although, we'll find out. I'll definitely let you know if it does. But my hope is I can give it the kernel, and then I can give it, like, a lot of background and then train it on my style and get kind of interesting outputs. But I'll let you know if that starts to materialize. That's my current AI obsession. I know you gotta go. This has been fantastic. Lot of really interesting things to think about, and I will look forward to maybe getting into sovereign AI. And I'm sure we'll have plenty more developments on The US China I don't wanna say conflict because I wanna frame it at let's frame it as friendly rivalry. Hopefully

Samo Burja: (1:19:59) Competition.

Nathan Labenz: (1:20:00) Constructive. Yeah. But, yes, this will be certainly a very active area for continued strategic analysis. For now, Samo Burja, founder of Bismarck Analysis. Thank you for being part of the Cognitive Revolution.

Samo Burja: (1:20:13) Till next time.

Nathan Labenz: (1:20:14) Thank you. Great job as always. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.