Full Transcript

Transcript

Nathan Labenz: (0:00) Hello, and welcome back to the Cognitive Revolution. Today, I'm excited to welcome Zvi Mowshowitz back for his record ninth appearance on the podcast. As regular listeners will know, Zvi is a fellow AI obsessive who processes an unbelievable amount of AI information and produces some of the most comprehensive coverage and multifaceted analyses available anywhere, all on his blog, Don't Worry About The Vase. The occasion for this conversation is, of course, the release of OpenAI's O3 model, a powerful but confusing release that Tyler Cowen called AGI, but which OpenAI reported, and the community at large has already extensively documented, produces twice as many hallucinations as its predecessor. We begin with a discussion of the case for O3 as AGI, and in light of the fact that OpenAI also reported that O3 can complete more than 40% of pull requests recently written by OpenAI research engineers, whether a process of recursive self-improvement, aka AI takeoff, has already begun. From there, we move on to discuss a bunch of important topics, including how to understand the strange situation we find ourselves in today, where it seems increasingly clear that AI can meaningfully accelerate science even while it still can't reliably order from DoorDash. Also, what superintelligence looks like in Zvi's imagination, including what sort of impact we should expect it to have on things like major national elections. The recent departure of key Epoch AI team members to found Mechanize, and why, at least when abstracting away from the details, Zvi and I are both inclined to support such efforts to automate mundane work rather than push models to ever higher levels of raw g. We also discuss the incredibly difficult challenge of imagining, let alone transitioning to any sort of stable equilibrium in a world full of superhuman AIs, and why Zvi's p(doom) is now up to 70%. We analyze each live player in the AI game today, including Meta, DeepSeek and other Chinese companies, SAFE Superintelligence, xAI, Anthropic, Google DeepMind, and of course, OpenAI. Why we should be grateful that today's models are showing misalignment tendencies now while they're still relatively weak. Why Zvi doesn't particularly worry about autonomous killer robots. And finally, what's virtuous to do now as individuals and as a field in light of all these developments. As always, I really enjoyed this conversation. Zvi shoots me straight and makes me laugh, and I really appreciate him for doing this late on a Friday night after another intense week. Laughs aside though, while I of course don't agree with every one of Zvi's takes, and I don't spend a lot of mental energy in general trying to pin down my own p(doom) estimate, I do share the broad sense that p(doom) seems to be rising. After a period in which pre-training on human data and light reinforcement learning from human feedback produced models that seemed to grok human values and behave according to their helpful, honest, harmless mandate to a really remarkable degree, it now seems that intensive reinforcement learning is creating more powerful but also quite palpably more problematic models. And not only are such models being released despite their obvious issues, but many in the community are going out of their way to downplay the significance of deception, scheming, reward hacking, and other bad behaviors. This, I feel quite strongly, is something that everyone, regardless of whether you love or fear AI or both, should seek to understand and communicate about clearly. It's my view that these sorts of problems are quite likely to lead to a popular backlash against AI even if more existential risks never materialize. To do my part to help, I've created a slide deck documenting an ever-growing list of AI bad behaviors. We'll link to it in the show notes. I encourage you to borrow from it freely for your own AI communications. On that note, I'm also excited to share that I'll be giving keynote presentations at three major AI events over the next several months. First, at Imagine AI Live in Las Vegas in May, I'll be speaking about the strange mix of eureka moments and bad behaviors that we're seeing from today's AI systems, and how people should be thinking about harnessing the good while also protecting themselves from the bad. Then in August, I'll be back in Sao Paulo, Brazil for the second ADAPTA Summit. There, I'll again be speaking about AI automation with the presentation updated to include all of the latest developments with AI agents. And finally, in September, again in Las Vegas, I'll be speaking to an audience of senior technology leaders at the Enterprise Tech Leadership Summit about some mix of the above, plus whatever important new developments emerge between now and then. All three of these events have outstanding speaker lineups in which I'm very honored to be included. If you'll be attending any of these, please don't hesitate to reach out as I'm always looking to learn as much as I can from the events that I attend, and meeting listeners is a powerful and fun way to do that. Finally, while I'm self-promoting, if you're working on AI automations, applications, agents, or adoption strategy for your organization and think I might be able to help, I again encourage you to reach out. I'm currently working with three different companies for just a few hours per month each on issues as narrow and focused as prompt and workflow optimizations, as cutting edge as multi-agent system design, and as high level as strategic opportunity identification and prioritization. I've found that these engagements are natural win-wins. They help companies confidently accelerate their AI projects, and they also keep me super grounded in the practical realities of AI deployment where the rubber is hitting the road today. As always, whether it's for a possible speaking engagement, an AI advisory project, or just to give feedback on the podcast, you can reach me via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this wide-ranging conversation on the latest news and the most important big picture developments in AI with the author of the indispensable blog, Don't Worry About the Vase, the great Zvi Mowshowitz.

Nathan Labenz: (6:08) Zvi Mowshowitz, welcome back to the Cognitive Revolution.

Zvi Mowshowitz: (6:11) Thank you. Thank you.

Nathan Labenz: (6:12) So you're fresh off your latest 10,000-word send. You are drunk on information like seldom seen before, and we're here to create the audio version for people that would rather hear you talk it all through than read the usual post. Although, of course, there's going to be a level of comprehensiveness in the post that we won't be able to match. So this should not be taken as a substitute for the blog if you...

Zvi Mowshowitz: (6:37) The posts are canon. The posts are what I said after I had a chance to think about it. This is what I'm saying off the cuff. This is just what I'm actually thinking. So enjoy.

Nathan Labenz: (6:46) Cool. Well, let's get into it. Big first question: is AGI here, and is RSI, aka recursive self-improvement, here?

Zvi Mowshowitz: (6:56) So no, and mostly no. I understand there are claims that O3 is potentially AGI. The more I understand it and the reports coming back and the more I use it, I think it's great. It's obviously not AGI. That's not what's happening here. This isn't even primarily an intelligence leap. This is a tool use leap. O3 is a much, much more useful version of the thing we already had. It's a lot easier to get what you want out of it, to get it to do the things you want in a reasonable time in a way that fits with what you want, that isn't forcing it to specific boxes, and that's wonderful. That's going to be super useful, but it's not AGI.

Nathan Labenz: (7:40) Tyler Cowen, I thought, did have an interesting frame where he said, how much more intelligent did you expect AGI to be? And so I guess I wonder, and there are many different frames we could put on this, but how many years ago in the past do you think you would have been given O3 and said, well, yeah, this has got to be AGI. What else could it be?

Zvi Mowshowitz: (8:04) I mean, if you ask the question instead, how many years in the past would I have gone, holy fucking shit, when I saw O3? The answer is something like two. Definitely four for sure. I would have been drawn before, like, face blown off nuts. Just like, how the hell does this exist? How the hell is this possible, especially this soon? But that's very different from saying it's an AGI. We're talking about something that can do all the things that humans can do. We're talking about the thing that can basically just plug and play anywhere you need it to do all the cognitive work. That's what we commonly understand AGI to be. To some extent, I think Tyler is defending a, you know, your term is defined broadly. You are using a silly definition of the term. The thing you're asking is, like, is it smarter than me? And from his perspective, the answer is yes. I think he's using the wrong definition of smart. I actually saw his comment and decided, I'm going to deal with this last. I'm going to go through everything else I have in my queue, and then, as the last thing that I write up, I'm going to deal with Tyler Cowen's claim, because I want to understand all the context before I evaluate this question. And by going through all of the other claims, okay, I understood what O3 was, and then I was able to look at Tyler's claim and understand why he was claiming what he was claiming, which is that it's a very good program for doing exactly the things that Tyler values highly, does all day, is consumed by, how he thinks. If you look at his examples, if you look at what it can do, it can go out there and get you tons and tons of detail, tons and tons of specific facts about any given thing, make connections between those facts, structure those facts, find the relevant things, present that to you. And he eats that all day. That's the thing he does, and he does it so much faster than I could do it even if I wanted to. It's an amazing skill, I'm in awe. But that's just not how my brain works. I produce lots of content in a completely different way, which is I consider information as part of a logical way of understanding the whole picture. And if it doesn't fit the logical picture, doesn't seem relevant to me, then it'll bounce off of me. And similarly, if it seems like it's just part of the pattern, once I understand the pattern, I don't need the details that light the pattern anymore. So I look at, he gave three examples when he claimed this. He's talking about, okay, why are this guy's paintings that are from early much more valuable than the paintings from when they're late? And this is part of a pretty common pattern for artists who basically didn't constantly reinvent themselves, where early on, they're doing the thing that's considered first, that's considered unique, that's harder to find, that's special in many ways. And one thing that I don't think O3 did point out is we have seen this extremization of value of collectibles across the board in the last 20 years, where the thing that's the best condition, the thing that's the actual unique first thing, the very best painting the guy made, the very special thing, is now like 10x the value of what the thing that's almost as good, that if the first thing didn't exist, would be considered exactly the way they consider the first thing now. So the 9.8 comic is so much more valuable than the 9.6 comic. You find exactly the thing you want, and this confused me for a long time as a Magic: The Gathering player. It's like, why is the thing that's the same thing but looks slightly better so much more valuable? It plays the same. But no, that's not what people care about. But basically, you know, because you've got demand-supply, you've got what is considered the best, the most unique, the most high status, the most representative of the thing. It's pretty standard, so I don't need to know any of the... I've never heard of this artist, I have no idea who this artist is or what he does. I don't have to. I can answer this question to myself anyway, and so O3 is giving me all these details. Then similarly with the author, and like, why is his prose so amazing? Whatever. It's good prose. It's for what people like. You can throw in specific details about this particular person, but I don't care. It's like, okay, you solved it, congrats. And the question about Knoxville, Tennessee, the impact of the tariffs, it's going to suck. And that's for everyone. The tariffs are a giant own goal. Yes, it turns out that all of their manufacturers import things, because all of our manufacturers import things, and they're going to be more expensive. They're not going to be competitive. And it's going to be a huge disaster. Thank you very much for giving me the details. I'm like, okay, congratulations. It's just the research. You're giving me a briefing for a congressman to prove that your district should oppose this thing because here's the specific factories that are going to be put out of business or whatever. It's not that it's not a useful thing, but that's still the intern's job or something. I'm not surprised, I'm not interested, and this is just a persistent disagreement between me and Tyler about what's interesting about the world. He travels, I think, largely because he eats this stuff up. And every time he travels, he can get all of these details. It's just nothing but extra detail about whatever you are. And he feels like this is necessary to understand this place, and all these details help to understand the world, and this is how one becomes someone who understands things. And I'm like, whatever. That's nice for you, but none of that matters to me. I more or less understand the things I need to know about these places from my perspective. And, you know, it's just a hugely inefficient thing to do to go around gathering these irrelevant details if I'm not enjoying myself. So it's just a very different way of looking at the world, and of course, the thing that thinks like he thinks and is doing a very good job of what he's doing, you know, that will be the thing he's like, that's AGI. And the AGI's papers get A-pluses. I'm like, it's fine.

Nathan Labenz: (14:10) So this reminds me of the Dwarkesh comment from a few podcasts ago where he said, basically, you know, these AIs, they're Tyler Cowen-like in that they soak up an unbelievable amount of information. Obviously, they've read the whole internet. You know, they have greater ability to do GPQA or Humanities' Last Exam or whatever than probably any human, I would imagine, at this point. It's certainly extremely, extremely rare to have that breadth of knowledge. And yet, he asks, why don't we see these things coming up with genuinely novel interesting connections across these domains? And I feel like maybe we just haven't been trying that hard. The first time I felt very clearly that I was like, damn, that AI seems smarter than me, was... and I felt glimpses of this other times too. But recently, I did this episode with Vivek and Anil from Google who have put together AI Doctors at Google and also this AI co-scientist that they have. I don't know if you read this story, but the co-scientist was tested on three levels of scientific challenge with increasing open-endedness. And the hardest one basically started with an observation about... and this is where you're like, damn it. This thing is smarter than me, because I can't even really describe the problem super well. But some observation that something is conserved by different kinds of bacteria that both have drug resistance to some class of drugs. And starting with that observation, the question put to the AI was basically, what's going on here? So it's a pretty tough one. The AI, in their setup, had a lot of inference. You know, they're definitely on the scaling inference train. This was Gemini 2, not even 2.5. It did have searchability, and it did have the ability to call AlphaFold and maybe some other specialized tools. And then it just had a bunch of different prompts, you know, where it's sort of grinding against itself to come up with ideas and evaluate the ideas and yada yada yada. Lo and behold, at the end of this whole process, and I think they ran it for a couple days, it spit out a prioritized list of hypotheses, and the number one hypothesis that it had flagged had been by a group of scientists that Google was partnering with, demonstrated experimentally, but not yet published. And here, they tell the story, the guys fell off their chair to hear that an AI basically was able to just comb through the literature and basically come to the same conclusion. How does that strike you? Like, is that AGI? And I guess, you know, it's hard to say, but if you thought we... or if we reran that with O3 as opposed to Gemini 2 in that super scaffolded, you know, deep access to search, yada yada yada... I mean, it kind of has its own search built in now. But do you think there's a qualitative shift there where this thing maybe now is a full-fledged AI scientist? I mean, I'm having a hard time seeing how we're not already getting tipping into the geniuses in a data center, honestly.

Zvi Mowshowitz: (17:19) Yeah. It's definitely weird. I strongly suspect it's a skill issue for the humans. I strongly suspect that if you put me in charge of AI Scientist Corp, and you gave me a billion dollars just to make sure it's not an issue, I have all the compute I want, I can hire whoever I want, I can try all the things that I want, that I can start figuring these things out, making these connections, doing these things, subject to, yeah, I still can't do perfect simulations, so we're going to have to actually try experiments and get feedback and so on.

Nathan Labenz: (17:54) Yeah. Notably, there is a difference between the AI hypothesized this and the scientists...

Zvi Mowshowitz: (17:58) Yeah. Yeah. We're not at the point where we can get the full explosion without interacting with the physical world. We're not even close. But, yeah, I think we're just really bad at elicitation of capabilities. We're really bad at scaffolding. We're really bad at creating the loops and the tool use and the logic of how to proceed on these questions because it hasn't been people's priority. Because there aren't that many scientists out there, and it's just not what people are focusing on. I don't think it's particularly hard in some important sense. It's not what I've been thinking about, but, and maybe I'm just being fully naive. Maybe they tried all the obvious things that are in my head, and none of them work, and they don't understand why, or maybe they do understand why. But, yeah, it just feels like when I see them, when you see the Google AI scientist, it's very much clear that they are just saying, what if we try to duplicate exactly how humans work in the scientific process in the real world, and just try to duplicate every step of the way, each thing with AI exactly the way that we do it, because we know that works, as opposed to trying to figure out how do we use the tools that we have to generate the things that we have. But also, especially because scientists don't have billion-dollar compute budgets. We don't probably value, even before we started firing everybody and throwing out all the funding, we don't probably value basic research. We don't give them the resources they need. Everybody is on a shoestring. So a lot of the things that I would think to try involve just trying lots and lots and lots of shit. Because, okay, I can draw connections between facts. And you can have it, okay, I have a million facts. Well, a million times a million is a trillion. It's not that much. How much is a query? And I can be more clever than that. I can do clusters. I can not have to do the full multiplication. I can use some discernment to figure out which combinations I have to check, et cetera, et cetera. So I can, in some important sense, get really, really creative if I care enough, but also, if I waste six months, everything has cost 10% as much. So for the same level of comprehensibility, why am I jumping the gun to try and do all this clever, throw all this money and all this compute at this problem to try and brute force this insight when I can kind of just wait for the models to get smarter and more efficient?

Nathan Labenz: (20:29) But shouldn't pharma companies be all in on this? I mean, it seems to me that is all well and good, but my estimate of the cost of running the Gemini 2-powered scientist, you know, for a couple days was somewhere in the hundreds to thousands of dollars depending on whatever, and, you know, maybe it could get up to $10,000. But it seems like if you're a pharma company, you should be hammering the API if only because, you know, you've got a similar game theoretic question that all the hyperscalers have vis-a-vis each other. They all want to buy all the GPUs and not fall behind on the model frontier. Like, why don't we see a pharma race where they are all saying, there's only so many of these things to be patented. Who cares if it's, you know, $10,000 a co-scientist run or five or two or, you know, five? Or six months?

Zvi Mowshowitz: (21:24) I bet you, Google would derisk that for them very happily because of all the publicity and benefits of, like, I'm the one working with Pfizer. If Pfizer discovers anything with AI, then we get credit for having done that. We win Nobel Prizes too. We get all the publicity and all the benefits too. And like, okay, I'll seed you some of the profits if you find anything. You don't even need to pay me. I'm sure these things are possible, but the real answer is, and I mean this in all seriousness, people don't do things. People just don't execute, people don't take risks, people don't do things that look weird. People are very slow to adopt these things and big corporations more so. And this is the diffusion problem. This is what, you know, Tyler Cowen going, follow them back, follow them back, follow them back, pointing at everybody. And this is where that's real. This is where the reason is just people are like, I can't... I mean, look, even I... I write about this all day. And if you honestly looked at the tools I was using, and the level of motivation, and the things that you could do... Dude, just take some time off and code. Just make something better, or hire someone to make something better. And I did some of it, but I could do so much more, but also, I can work, you know, probably two to five times as fast as I could back then just because the AIs have gotten better since the last time I was doing it in a few months. It's just such... it really is. People are just very, very hard to get them to get off their patterns and adapt and explore and try new stuff, and we're just trying to get through the day, and we're just trying to run the thing, but the day will come. And it is also because of this demoralization of everything is going to be obsolete in a few months. So I think that if there were, say it was a pause. Say it was a pause not because we all agree on a pause, but say it was because it turns out we just hit a giant wall in terms of the base models, and this is as good as it's going to get within reason. And it stops getting cheaper. It stops getting better. We get O4. We get O5. We get Gemini 3, and it's all 5% better, 10% better, but it's not amazing. And now we're like, you can't just sort of wait for this to happen on its own. You've got to make this happen. It's your responsibility to figure out how to use this thing, and you're going to have a period of time to use this thing. It's not going to be a few months before it all gets blown away. I think we start to see some really, really creative stuff get a lot out of what we have that right now, everyone's just trying to keep up.

Nathan Labenz: (24:00) Hey, we'll continue our interview in a moment after a word from our sponsors.

Being an entrepreneur, I can say from personal experience, can be an intimidating and at times lonely experience. There are so many jobs to be done and often nobody to turn to when things go wrong. That's just one of many reasons that founders absolutely must choose their technology platforms carefully. Pick the right one and the technology can play important roles for you. Pick the wrong one and you might find yourself fighting fires alone. In the ecommerce space, of course, there's never been a better platform than Shopify. Shopify is the commerce platform behind millions of businesses around the world and 10% of all ecommerce in the United States. From household names like Mattel and Gymshark to brands just getting started. With hundreds of ready-to-use templates, Shopify helps you build a beautiful online store to match your brand's style, just as if you had your own design studio. With helpful AI tools that write product descriptions, page headlines, and even enhance your product photography, it's like you have your own content team. And with the ability to easily create email and social media campaigns, you can reach your customers wherever they're scrolling or strolling, just as if you had a full marketing department behind you. Best yet, Shopify is your commerce expert with world-class expertise in everything from managing inventory to international shipping to processing returns and beyond. If you're ready to sell, you're ready for Shopify. Turn your big business idea into cha-ching with Shopify on your side. Sign up for your $1 per month trial and start selling today at shopify.com/cognitive. Visit shopify.com/cognitive. Once more, that's shopify.com/cognitive.

Nathan Labenz: (26:03) I think about this a lot for myself too where I'm like, you know, obviously thinking about this all day, more talking about it than writing about it, but a bit of both. And, you know, I use a lot of things. I still don't live a very automated life. And I sort of challenge myself, like, why is that? And am I doing something stupid? And one answer that's the sort of charitable answer to myself is, well, I don't really do that much work that's routine, you know, which is a great luxury that I don't take for granted. And so, you know, the sort of real-time assistant paradigm is pretty good for my non-routine life because it's sort of the right form factor for non-routine work. If I had more... you know, I tell myself if I had a lot more routine work, I would, you know, set up these workflows and pipelines and stuff and, you know, then I would do it, but I don't. But then I'm like, I don't know. Am I letting myself off the hook too easily here? Like, maybe I should come up with some routine stuff that I should be doing that I'm not, but I could because I could automate it with AI. And maybe I'm just not that imaginative or not that creative or not trying hard enough. But for me, it feels like the bottleneck in terms of living a more automated life is kind of like, I personally am not doing a lot of routine tasks that I would like to automate away, and I feel a little low on ideas of things that I would automate that I'm not doing at all in the first place. Do you have things that you are, like, you know, conscious of that you would think, you know, the Zvi++ would have already scaled with AI automation? Nathan Labenz: (27:40) It's always the question of, is the automation going to be good enough? Do you actually use it, and do you actually not spend that time checking its work? You don't spend that time doing the manual version of it also. You actually end up saving time by doing that. And there's certainly a substantial chunk of that that would be very good. Certainly, doing things like notes and organizing resources for myself and organizing just facts and links and stuff for future reference would be great. But then again, it's very hard to automate that because it doesn't know what you want to remember. Formatting for websites, I did some of that. I wanted to do the automatic cross-post to Twitter, but it's a remarkably annoying problem. And because of the structure of what you're trying to do, the AIs were dramatically worse in that spot because normally you're like, code this thing that does X, that does Y. And just doing Y is a very easy thing for the AI to understand and solve. But if you're trying to have it navigate idiosyncrasies of this particular existing website that's kind of put up crap, like Twitter, then you run into this problem of you're just going back and forth and debugging, and trying to get it to do something. In the past, it's been pretty terrible. And if you get it to do its own feedback debug loops now, I don't know if you can, and suddenly it probably gets a lot easier. There's a lot of different, really hacky things you have to do to get it right, but it would save you a bunch of time. There's some formatting things. I'd really like to be able to sort of transform the way that I interact with the web, and create a bunch of shortcuts and a bunch of just quick ways to just transform the data and so on, and that'd be great, and then I go from there. It just, again, I can't catch a break. There's always something going on. And then you get this thing, right? You get that day when you don't have anything going on because you've managed to finally get ahead of your giant to-do list.

Zvi Mowshowitz: (29:37) Then somebody says, I just want a 3-hour podcast, and the next thing you know, the day is what I just did.

Nathan Labenz: (29:42) That's not what I was going to say. What I was going to say was you're just so mentally checked out after all of that, and you're so happy to have some time, but you're like, why don't I go see a movie? Why don't I go out for a nice dinner? Why don't I just chill? And then by the time you're done chilling, there's something else to do. And so it's very hard to not just get distracted. I have so much stuff in my queue. I have so much stuff I want to do. Everyone else is like, how do you do all this stuff? It's like, well, you don't do anything else, right?

Zvi Mowshowitz: (30:08) If you're fun to do, I work a lot. Spoiler, I work a lot.

Nathan Labenz: (30:12) And I help with my family, and I chill, and I watch TV and listen to podcasts. And that's kind of it, right? In some important sense. I exercise, I eat, and so on. But that's it. So, yeah, worth the time to code. And also, I found that coding for me has largely increasing returns to scale in terms of being in that mindset where my brain is wrapping itself around the issues. And programmers like to have state, right? They like to keep the problem they're solving in their head and not be distracted. And so if I'm going to just, I can't do 5 minutes of programming a day, right? That doesn't help anyone, that's useless. You want to be doing hours. And so I really need a large amount of dedicated free time where I'm really going to dig deep into what I have, and load up Cursor, or Claude, or Windsurf, maybe Codex now. I don't know. I've heard mixed reviews. And do my thing, and then see what I can do. But, yeah, it just sort of never feels like the right time to make that investment. But when I did make the investment, I think I've made my money back time-wise at this point. But it was really a struggle. So I don't know. Yeah.

Zvi Mowshowitz: (31:25) I feel you on the challenge of ramping up into the programming problem space. I've been trying to get a day a week. It's been more like a day every other week to really fully program. But then it also is like, man, my feeble biological brain when I sit down in the morning, it's like, what was I doing 2 weeks ago? And I need a minute just to kind of recalibrate myself or reorient myself to the problem. And it's funny. That is 1 area where the AIs really shine. I think 1 tip that has actually worked for me that might help you and might help others, I hadn't really thought about this being a tip so much even until right now, is to literally pick up with the chat from 2 weeks ago. Wherever I left off, even if it was kind of in the middle somewhere, just to go back to that thread.

Nathan Labenz: (32:13) Yeah. My, yeah.

Zvi Mowshowitz: (32:14) Where were we? What were the last 5 things we did here?

Nathan Labenz: (32:17) Yeah. My experience with that kind of thing is sometimes it works and sometimes it's just this huge disaster and you never get it back. It's like sometimes, coding is often like reading from a forbidden book of wizard incantations. And you really, really hope you don't mispronounce a word, and suddenly summon the wrong demon or whatever. And so we just get a series of error messages. And so if everything goes exactly right, wonderful things happen. But if you make even a small mistake, it can be so hard to recover because you don't understand where the mistake is, and you're not good at figuring out what it is. And the AIs will sometimes rescue you because you're just pasting the thing, and the AI will say, oh, this is what you did. Great. We're back. But if they don't, you're just so screwed. And so, yeah, I'm, again, I should invest more in it, but I keep getting, you know, here, take this trip. Go to this conference, be part of this exercise, do this other thing. One thing leads to another. So you just need to clear the time, but, you know, it's just really, really hard. I mean, now that I've dealt with O3, I'm hoping that that caused me to be only 1 day behind with everything else. So I have a few hours of work to catch back up. But after that, I don't know, maybe this weekend I can do something, right? Maybe next week I can do something. It's entirely possible, which depends on what happens. Like, what if Google released 2.5 Flash and nobody even noticed?

Zvi Mowshowitz: (33:44) Yeah. That'd be so weird. It's not on our agenda today, really.

Nathan Labenz: (33:49) Probably. That happened yesterday.

Zvi Mowshowitz: (33:52) And I'm sure it is very good. I haven't used it yet.

Nathan Labenz: (33:54) But it probably is. I have no idea.

Zvi Mowshowitz: (33:58) I'm confident it's very good. I mean, I was a big fan.

Nathan Labenz: (34:00) I think enough benchmarks that I'm like, if a model that small is scoring that well, and given how good 2.0 is, this is going to be an amazing model for its size. But in practical terms, what can I do with it? I don't know. I'm just going to use it for free anyway.

Zvi Mowshowitz: (34:19) Okay. Here's another thought experiment. The drop-in knowledge worker, I hypothesize, is bottlenecked right now mostly on a couple things. One is just effective ability to use a computer. Put a pin in that. We'll come back to that in a few minutes. The other is the ability to sort of absorb the surrounding context in a reasonable way in the way that humans do when we get a new job, right? You get a little bit of training and that helps, but then you also sort of bop around a little bit and poke around some Slack channels and talk to people and look at their work product and eventually, you know, probably get some feedback for doing things not quite right, and then eventually you sort of figure it out. It seems like it's clearly conceptually possible for an AI of the base level of intelligence and breadth of world knowledge of 1 of these current latest generation of models to do some sort of additional possibly training, possibly just processing or memorizing, maybe even long context could get there. But if you imagine an AI that's no smarter than what we have today, but it can drop into an organization, go through all the old emails, go through all the old GitHub issues or whatever, go through all the Google Drive, look at all the proposals that have been sent out in the CRM, and obviously can process those things a lot faster than humans. And it kind of comes out with a sort of, alright. I kind of got it. You know, in the same way that I know who was the prime minister of Lithuania 5 years ago because I just kind of know everything. I sort of just know everything now about this company in a similar way. Is that AGI?

Nathan Labenz: (36:05) If it can do basically anything that way, or if it can do most tasks that can be done, most jobs that can be done from a desk, then, yeah, I think you have to give it to, you have to consider that AGI, not as ASI yet, obviously, but you do basically have to give it AGI. But that implies you can also do the job of an AI researcher, right? Or else that wouldn't count. That's maybe a relatively easy task. It's relatively hard in some senses and relatively easy in others. Definitely not near the end of the list of how things get automated in what order. But, yeah, no, absolutely. And I do agree there's a lot of room to do it, but I look back at my time at Jane Street, right? And so they take a loss on a new employee effectively for at least a year, not because you're not creating value, but because the amount of time they spend for other people that's required to keep training you to make you better, to give you feedback, and to let you learn where someone else could be doing the thing that you're doing better, but instead you're doing it so that you can learn to do it yourself, and you can learn to be better. Like, the first year, they had less money because they hired you, even if you are doing as well as you can realistically do, like 95th percentile person. And just think about that level of investment, and compare that to the patience people have for a drop-in worker, right? If you try to drop in a worker and you ask them to put in 10% that much investment before it started doing useful things, 0 companies would tolerate that.

Zvi Mowshowitz: (37:40) Hey. We'll continue our interview in a moment after a word from our sponsors.

Nathan Labenz: (37:45) It is an interesting time for business. Tariff and trade policies are dynamic, supply chains squeezed, and cash flow tighter than ever. If your business can't adapt in real time, you are in a world of hurt. You need total visibility from global shipments to tariff impacts to real time cash flow, and that's NetSuite by Oracle, your AI-powered business management suite trusted by over 42,000 businesses. NetSuite is the number 1 cloud ERP for many reasons. It brings accounting, financial management, inventory, and HR all together into 1 suite. That gives you 1 source of truth, giving you visibility and the control you need to make quick decisions. And with real time forecasting, you're peering into the future with actionable data. Plus, with AI embedded throughout, you can automate a lot of those everyday tasks, letting your teams stay strategic. NetSuite helps you know what's stuck, what it's costing you, and how to pivot fast. Because in the AI era, there is nothing more important than speed of execution. It's 1 system, giving you full control and the ability to tame the chaos. That is NetSuite by Oracle. If your revenues are at least in the 7 figures, download the free ebook, Navigating Global Trade, 3 Insights for Leaders at netsuite.com/cognitive. That's NetSuite dot com slash cognitive.

Nathan Labenz: (39:09) In business, they say you can have better, cheaper, or faster, but you only get to pick 2. But what if you could have all 3 at the same time? That's exactly what Cohere, Thomson Reuters, and Specialized Bikes have since they upgraded to the next generation of the cloud, Oracle Cloud Infrastructure. OCI is the blazing fast platform for your infrastructure, database, application development, and AI needs, where you can run any workload in a high availability, consistently high performance environment, and spend less than you would with other clouds. How is it faster? OCI's block storage gives you more operations per second. Cheaper? OCI costs up to 50% less for compute, 70% less for storage, and 80% less for networking. And better? In test after test, OCI customers report lower latency and higher bandwidth versus other clouds. This is the cloud built for AI and all of your biggest workload. Right now, with 0 commitment, try OCI for free. Head to oracle.com/cognitive. That's oracle.com/cognitive.

Zvi Mowshowitz: (40:19) Yeah. I mean, that's a really interesting claim because my hypothesis has been and I don't know what the first year salary at Jane Street is, but I have been expecting that first of all, just kind of ballpark, you know, calculation, to fine-tune GPT-4o or 4.1 is $25 per million tokens of fine-tuning training data. So if you were to say, you know, how many tokens do we have at a given company? Obviously, it's going to scale widely. If you had an actual trillion, you would be looking at 1 million million times 25, you'd be looking at $25 million to make essentially a custom model, which is kind of notable.

Nathan Labenz: (41:01) If you were 5% of the internet or something, right? In that case.

Zvi Mowshowitz: (41:04) Yeah. But, I mean, certainly, my business doesn't have 1 trillion tokens. Maybe we have 1 billion, in which case I'd be looking at a $25,000 investment, you know, for it to be kind of fine-tuned on everything. I don't think you could literally do that. What comes to mind, of course, is the famous OpenAI graphic of the model they trained on Slack where you say, you know, do this whatever, and it says, I'll work on that tomorrow because that's what it learned on Slack. So there definitely have to be some processing of all this sort of bad random data.

Nathan Labenz: (41:34) That's the issue. The issue is that you can give me 1 million tokens or 1 billion tokens, but what does it mean to simply train on those tokens? And then suddenly this machine can do the thing that's described in the tokens.

Zvi Mowshowitz: (41:49) Yeah. You need a pipeline of processing that, filtering it, transforming it, you know, making sure that everything's right.

Nathan Labenz: (41:55) Machine learning is the most fiddly thing on the planet, right? It's the most trial and error, learn by doing, see what works and what doesn't, do all of this bespoke stuff to get it to work, just keep working at it. And that's very, very different from plug and play, right? I think that if OpenAI said, pay us, you know, $500,000, and we will train your automated worker, and we'll then license it to you for $20,000 a year per copy on top of that, and then it will do your job, people would just jump at that if all they had to do was dump all their tokens into explaining what the tokens were. But that won't work, right? OpenAI doesn't have that. You can't just dump the tokens even if they knew what they were doing. You'd have to provide the context for the tokens, have to organize the tokens, have to know what you wanted to be able to do with it, and then OpenAI would have to figure out how to navigate that into an actual proper tuning program to incorporate all the things that would have to work closely with a bunch of your employees for a while. None of this is easy. And, you know, the problem is getting to the point where the AI can do the job of setting up this training process for you so that you can then create this worker or something like that, probably. But it's going to be a while. It's just going to be a very human trial and error style thing unless we're well above where we are now. It's coming more and more, but, like, you know, Operator was remarkably bad at even the tasks that everybody basically does the same way all the time to the point where even though I was paying the pro value anyway, I was just like, I don't want to take the time to try and figure out how to use this and start giving it the information it needs to work. Like, today actually, I was working through and I was like, I'm at DoorDash. Should I use Operator? Should I start learning how to get Operator to do my DoorDashes for me? And I was just like, I have to figure out how to get it to tell it to do no lettuce and no tomato. I'll just, no. I'll just, I just want food. And then that was the end of that idea, right? It's that kind of impulse, but at scale.

Zvi Mowshowitz: (44:00) Yeah. I mean, this task reliability thing again, let's come back to it in 1 more second. I mean, I do feel like we might be quite close to the sort of additional training of, you know, processing of your information and then training of your custom knowledge worker could be quite close. You know, 2 reasons to think that, 1 of which also has many other implications. First 1 is just in my own experience. You know, what the big place where my, you know, limited coding hours have gone recently has been to make a minor contribution to the emergent misalignment project, which that that honestly, the team was very kind to include me as the last and least valuable co-author. It would have been understandable for them to not include me at all. Yeah. But with Gemini 2.5, I'm able to take the full research code base, which actually, it's not, it's all the code plus some of the data. I had to, I wrote a script, I, of course, used AI to do, to say, go through and print out all of the code from this entire research code base into 1 file. And then it was way too big. And I said, why is it so big? Oh, there's datasets in here. Okay. Just give me the first example of each dataset as you go through and give me all the code. That comes out to 400,000 tokens for this particular code base, and then there's the paper itself. And then I had done some additions, again, having AI do most of the coding. And I came back, and I just wanted to put the updated code base in the paper and just make sure it sort of knew what was going on. And I asked it to tell me what was going on and make sure it understood the new stuff that was in the Nathan folder, because, again, research code base. So there's the Jan folder and the Daniel folder. Now there's the Nathan folder. So this is not production grade software, but it didn't have any problem with that. It honed in on this question that the paper mentioned briefly, but didn't really dig into, and then my additional experiments to try to expand the understanding of that particular topic and gave me really an unbelievable readout of, this is what you are trying to do. This is what these experiments, you know, that you're adding on now to the code base are for. Seems really interesting. You know, how can I help, basically? And I was like, damn. That is really something else. Because I didn't even tell it that. It inferred that from, I told the, you know, different context window of the model that, and it helped me write the code. But then just from the code itself, it was able to infer all that sort of stuff. And I was like, man, you know, that's first of all, that's a lot of information. You know, 500,000 tokens basically in context. You're talking a full thousand pages of stuff. And a really sophisticated readout that inferred the intent and the, you know, and I don't know. Pretty frontier topic too. The the paper itself is not in the training data because it's only been out for a month and a half or whatever. So I was like, if you can do that, then I would think you would be able to kind of wade through large reams of Slack chats and stuff and really kind of zero in on the stuff that really matters and probably put it into a decent format and then, you know, pipeline that all the way through to a fine-tuning recipe that you sort of probably don't necessarily optimize, you know, all the hyperparameters for each customer, but you sort of did that at a meta level. And...

Nathan Labenz: (47:26) Yeah. And it makes sense. It's possible. I think there's a lot of O-ring style problems with this kind of approach in addition to the whole, like, everything's fiddly and nothing works the way you want it to. If you've got a worker, quite often, you'll say, you know, Bob is great, Alice is great, but they have this specific thing that we can't stand that's terrible. And this is a deal breaker, right? Even though, you know, they're 99% what we want, that's not good enough to be that useful. We can't just have that 1% be handed off to someone else. Doesn't work that way. And, you know, I think this is a lot of the problem of until the AI is crossing the threshold of being good enough at a given task or set of tasks, it's not considered useful, right, for the grand set of things in that way. And so people don't think on that level, because, again, they don't have that kind of patience, and that kind of, oh, we'll be able to fix it, and so on. And so instead, we're starting with little tinkering, right, which is how actual progress is made in most things, right? So we're just saying, okay. Here are individual specific things the AI can do well. And then over time, we will figure out how to string those together. We'll figure out how to get the most out of each of those things, and then that will combine more and more. And we'll start filling the gaps. And the AI will start asking itself, how do I fix these gaps? And then, as the models get better and our understanding gets better, they will come together, and these things will start to happen. But, yeah, we're just not quite there yet, but we're starting to see how much closer to there we're getting very quickly. A lot of this stuff seems a lot more tempting now than it did, you know, 6 months ago, right? Vastly, vastly more. Because 6 months ago was before O1.

Zvi Mowshowitz: (49:20) Yeah. Talk about compressed time. Okay. So it seems like and, you know, 1 of the comments that Greg Brockman made, which I know you highlighted in the post as well, was that this is the first model that they've had scientists tell them comes up with really good ideas. And this sort of has me thinking, we might be in a really weird spot where the hit rate on good ideas for frontier scientists, especially with some decent scaffolding, is very plausibly, and I would even say maybe likely high enough that if you are a frontier scientist, you really should be using it, and it will very likely accelerate your work. I feel pretty confident in saying that as a blanket recommendation to most scientists. And yet, at the same time, you're like, but we can't get it to order DoorDash, you know, reliably.

Nathan Labenz: (50:13) It's just a matter of hyperbolic discounting, and there's not a will to do that right now.

Zvi Mowshowitz: (50:17) Also, for Operator, you would have, maybe you might have had more luck. Operator, personally, just asks me, should I continue? Should I continue, every 2 seconds? And I'm like, yes. You should continue. Put the thing in the basket. You know? It's unbelievable. Manis, by the way, does have a much better, I mean, for better or worse, but I think for your mundane utility purposes, it won't ask you, okay. I, you know, found the sandwich. Would you like me to put it in the cart? It will actually just do it. So...

Nathan Labenz: (50:45) Yeah. Well, I'm also not giving Manis my credit card. So...

Zvi Mowshowitz: (50:50) That's fair. I'm still on the free credits version.

Nathan Labenz: (50:53) Right. But if I don't give it my credit card, how's it going to work with the sandwich?

Zvi Mowshowitz: (50:56) Oh, yeah. Well, that's a, you could potentially have it in, I mean, they do have an interesting, well, I may be confusing the Operator and Manis security model. I created a new Gmail account for Manis so that I could share docs with it, log in to Gmail as it, and then have it go see all the stuff that I share.

Nathan Labenz: (51:15) My attitude is, you know what? Anthropic will have its own version of this. I'll be able to trust that, and I'll just wait. It's fine. But I will say there is something quite valuable about using products that really turn up the hyperparameters, and, unfortunately, Claude does not do that yet. Like, a good contrast is Shortwave, which I've been using for email versus the new Claude Gmail integration. I went to Shortwave and said, read my last 100 emails sent and give me some advice. And it uses Claude, so it's the same model. But it actually did search for 100 emails, put them into context, and had Claude process that big dump of information. Whereas when I went to Claude and gave it the exact same prompt, it searched once for 5 emails, searched again for 5 emails, searched again for 5 emails, got to 15, decided that was enough, and then carried on with the task with just the last 15, that was just from the last 3 days.

Nathan Labenz: (52:14) I can't get Claude email integration to work. I use Gmail too, and I was really excited. And then it just keeps failing at the most basic tasks, and I just don't know what's wrong. But for now, I'm like, this doesn't really exist. And, of course, Shortwave is also read-write, not just read, which is a huge, huge difference. But do you think it's worth using? You think it's past the threshold? Zvi Mowshowitz: (52:39) It's not perfect, but I do get real value out of it sometimes. When it does an expense report for me, it's pretty cool. Somebody today, we got a sponsorship inbound, and they were like, what kind of episodes do you have coming up? And I just went to Shortwave and said, please pull me a report of all the possible podcast guests that I have coming up, then put the most interesting ones at the top, basically. And for those sorts of things, it is definitely notable utility. I mean, it's still just the real-time back and forth model. It doesn't send anything without your approval, and it doesn't mark anything as done without your approval. Instead, it will be like, here's 12 things I think you should mark as done, and you can uncheck any that you want to uncheck and then you can mass done them. So it's not like running off with your account in a way that is taking your agency too much at all. But I do find that the hyperparameters turned up in general is just a pattern that always adds a lot of value relative to the default. And I'm frustrated, honestly. I think I know why, and it's understandable that they're compute limited and they might also start losing money on customers if they turned all these hyperparameters up. Shortwave until recently was losing money on the margin, he said.

Nathan Labenz: (54:02) Yeah.

Zvi Mowshowitz: (54:03) As of a year ago, he was like, every new customer, we lose more money. He said that is no longer true, but they were just willing to eat that and they weren't at a huge scale. And so they were kind of like, let's just look at the best value we can.

Nathan Labenz: (54:14) Because they know that everything gets cheaper, so the same product will end fast.

Zvi Mowshowitz: (54:18) Yeah, they're definitely riding that wave. So, yeah, hyperparameters turned up is good. Okay, but I guess I want to get your take on, because there's a couple of things that are pulling in different directions, and I find myself a bit confused. We have this world where AI can accelerate science. I think that's becoming pretty clear, especially if you as the scientist are willing to accept a not perfect hit rate and maybe only one in 10 ideas are going to be worth your seriously engaging with, but that could be a huge hit rate for science.

Nathan Labenz: (54:52) As a scientist, if you're like, this idea only has a one in 10 chance of being a great idea, I don't want to engage with that, you're a bad scientist.

Zvi Mowshowitz: (54:59) Right. So it seems that we might be into the realm of science seems like it's accelerating. This sort of mundane personal assistant work that we would all love to delegate and unchain ourselves from our desks more isn't quite working yet.

Nathan Labenz: (55:18) It's close, and that will accelerate science. Right? I mean, think about it. What's the biggest drag on science right now is the average scientist spends a huge percentage of their time on things like fundraising and doing the paperwork.

Zvi Mowshowitz: (55:35) Yeah. I was going to ask you, actually, if you think that this is a little bit out of domain for this feed, but it strikes me that the withdrawal of federal funding from top universities could potentially reinvigorate universities in a way that we haven't seen for a long time, because all of a sudden they're going to have a different funding model. And how are those decisions going to get made? And what if there's no federal government bureaucracy that they have to appeal to? It's just like, well, I guess we're the chemistry department. We've got to figure this out on our own. How do we do it? Do you have any optimism for revitalization of science through withdrawal of DOGE?

Nathan Labenz: (56:18) Giving me EA-style ecosystem where you have to get nine government people to give you money instead of getting the government to give you money is not better. It's relatively good because a lot of people involved are actively trying to make it painless and actively trying to do good things, but it has so many of the same problems. It has so many of the same bad incentives. Obviously, to me, it'd be great for Harvard if Harvard stops taking federal money and instead used its ridiculously large endowment to pay for things, and maybe funded itself with the products of its discoveries over the long run, maybe, or something like that. And certainly, it's possible to say that right now, you're dependent on this model that is it focused on producing anything useful? And it takes a vagina amount of your time, and that ends up favoring people in their 50s over people in their 20s and 30s. And most good science is done by people who are relatively young historically, because that's just sort of when your brain is best attuned to do that kind of science. And changing all of this up could be really good, but there's always this problem of, we're going to break the current system. Well, that's great if you replace it with something else, and if you just try to hobble a lot of the broken system, it's just obviously worse. And so, do we have great hope that we're going to be able to actively fix the problem? I'd be more optimistic, I guess. Also, do the scientists just flee to Europe and Canada and Japan and everywhere else? Because I don't want to deal with this, and I can get funded somewhere else.

Zvi Mowshowitz: (57:56) And half of them are from all those places in the first place. Yeah.

Nathan Labenz: (57:58) Right. If they go into an industry which some people think is better, but obviously it's very different. If they go to finance, you've got a problem. If they go to Google and then Google funds basic research, then that's great. But is that what's going to happen? So, yeah, it's really hard to know. But on the timelines that we're looking at with AI, I don't particularly want to break our current system and force everybody to spend a few years scrambling to reinvent a new system, separately from the fact that we're going to break everything anyway. There's just no time for the new system to pay out.

Zvi Mowshowitz: (58:34) Plate is full for crises at the moment.

Nathan Labenz: (58:38) Yeah. I mean, the office is going to kind of break anyway to understand. Right? None of it's going to make sense in the new world. The university, the entire education system, kind of doesn't make any sense by now and will make less sense in two years.

Zvi Mowshowitz: (58:52) Yeah. I mean, I don't know. I don't have any comprehensive data on this, but it wasn't too long ago, less than a year ago, that I was invited to give a presentation to some computer science student club at an American university. These are mostly undergrads. And they were like, our professors don't allow us to use any AI coding assistance. And I was like, I don't really know how to sugarcoat this for you, but I don't think your professors are doing you a good service by doing that. Not to say there's not a place for some independent exercises there, but to just pretend it's not happening is a tough position for the universities to defend for too long.

Nathan Labenz: (59:29) My perspective is probably learning how to code with AI while you're focusing on actually learning how to code is better than learning how to code without AI, which is better than coding with AI to try and get an A in class without actually learning.

Zvi Mowshowitz: (59:47) So speaking of people that are learning to code with AI, for me, the most notable part of the O3 model card was, I think, on page 22 of 30 something, where they report the progress on the model's ability to successfully one shot real pull requests previously submitted and unit tests developed for that by real OpenAI research engineers. So they've got this internal code base, which separately, you watched the 4.5 video with Altman and the researchers? They talked about, oh really, there's, I thought it was interesting.

Nathan Labenz: (1:00:29) I have been burned so many times by watching those stupid videos. And I saw specifically people online complaining about having been burned by watching this one. So no. I don't watch their videos anymore. I will read people's summaries of the videos and maybe I will dump it into Gemini and ask for questions. But I'm not going to watch this video. I find video to be a terrible format for learning things.

Zvi Mowshowitz: (1:00:58) Yeah. Well, I always joke about our own venerable YouTube feed that people should put the phone in their pocket and take a walk because it's not healthy to spend that much time looking at my face. I also act on that. I listened to that one while driving. Point being, they also use their internal code base as their standard data measure for perplexity. So when they just train a new model, they look at its perplexity on their own code base as sort of the gold standard of just overall model intelligence. Anyway, now we're in a world where the model is tested by going to a specific commit point, checking out the code in that commit point. In the background, an actual employee has already done this particular chunk of work, and they have tests to validate it. And then the model is told, okay, here's the repo, and here's the assignment, and that is human written. So it's not like the AI is figuring out what to do next. But given the assignment, can it do it? We're now with a big leap from prior models being in the single digits. All of a sudden, these new models are in the 40s. And that does seem like a pretty big deal. That's why I asked at the top, is RSI here? And I guess, literally, it's under 50%, so that's one way of saying mostly no. Another big factor there is obviously that figuring out what to do is a very important part of doing useful things.

Nathan Labenz: (1:02:26) I mean, obviously, an engineer that could do 40% of pull requests is a lot less than 40% of the job, because that 40% is not random. But also, if you have an existing AI that can do 40% of your existing pull requests, what the hell is going on? It should be 0% of your pull requests because it should have already done the 40% of the pull requests it can do for you, leaving you with the rest.

Zvi Mowshowitz: (1:02:51) Well, this might be sort of a transitionary moment where all these timelines are pretty compressed internally for them too, it seems like. So the previous models were literally very low numbers, and then this does represent a big jump. So this might be the sort of one moment in time where they have this phase change of previous models really couldn't. These ones now substantially can. One assumes they will be using them going forward.

Nathan Labenz: (1:03:15) But if you think about it, right, if I'm coding with, say, I'm coding with Sonnet, right, as I was when I was last coding. If Sonnet can do the pull request, well, why did I have to create a pull request? Why didn't I just have Sonnet do it? So it's not a random set of things of which Sonnet happens to be able to do almost none of them. The pull requests are the list of things that people couldn't just do with Sonnet. Right? So...

Zvi Mowshowitz: (1:03:39) I don't know. I mean, I think in many software organizations, there is still this discipline of we're going to define, because there's all these workflows that are attached. Right? And when you do a pull request and then you merge it, there's all these automations in the software world where it's like, okay, to integrate this, we first are going to run this whole battery of tests and confirm that it passes, and then we'll integrate it. And then there's often some automated deployment pipeline such that even if you have a model that can do a high rate of the pull requests, you might still work through, I think many organizations still are kind of working through that same overall process, even if the AI is writing all the code for any given step.

Nathan Labenz: (1:04:17) Yeah. No, an agent is going to be doing it. Right? It should still be something that I haven't gotten around to finishing yet, but there's sort of my model of how this works based upon being an active coder. There's the list of things the AI can now do, and they are now 10x speed or 100x speed or something obscene. And therefore, they go very, very quickly, and they should not stay as pull requests for long. And there's the things where your current AIs are struggling or you don't know how to prompt them properly, and there's something that a human has to actually think about, and they're already sticking around for a lot longer. And so, again, it's just maybe 5%, maybe 10%. But if half of the what would have been pull requests are solvable by the AI, it should be a lot less than half in the current set of pull requests. But, again, I've never tried to program with other people, and I never, so with an AI and a person I was working with. So I'm the wrong person to be asking, but just...

Zvi Mowshowitz: (1:05:17) A note on vocabulary too, just for what it's worth. Typically, an issue is your sort of upstream open ticket, and then the pull request is the actual code that you are requesting to be merged in. But sub in issue for pull request in some of your last few statements, and that does make sense that you shouldn't have open issues that AIs can do sitting around for very long, or you're definitely underutilizing the AIs. Yeah.

Nathan Labenz: (1:05:43) The same way that if I have open issues for myself that are solvable by myself fairly quickly, then Getting Things Done says you should just do them already.

Zvi Mowshowitz: (1:05:52) So what do you make of this 40% number then? Like, how do you interpret it, or does it seem like, I mean, they're presenting it as a pretty big deal, and it feels like a big deal to me. How does it feel to you?

Nathan Labenz: (1:06:08) It doesn't really jive with the reports of coders. So if you look at my sections, I look at people who are saying specifically how good O3 is at coding. O3 is just not that good at coding. O3 is good at architecting and debugging. So maybe that to me implies maybe a lot of the requests are about bugs. They're like, we found a problem, we don't know what's going on, can somebody figure this out? And O3 is reasonably good at spotting what that is. It tells you what kinds of things end up as issues in their code base at any given time, which makes sense. And a huge portion of coding is debugging, so it doesn't make it not a huge portion of the actual work. That makes sense too. But I don't really know. I think it's very opaque because they're not going to let us know what those requests look like, what those issues look like, because obviously they can't. It's a fun little test, but they are presenting this as progress, but they're also, you have this weird thing in the model card where you're simultaneously bragging about how much your model can do, and then you're testifying what your model can't do, because if the model could do too many things, then you'd have to do something about it. And so what are they doing? It's some weird hybrid. I don't know what to make of it, but O3, yeah, I don't know. I wish I did more coding so I could take it firsthand on that level much more than I can. I certainly haven't done any since O3 came out. I've just been obviously way overwhelmed. The fire hose has been going strong, as they say.

Zvi Mowshowitz: (1:07:43) So, okay, so let me go back to this kind of point of confusion that I have or sense of, I'm not even sure which way we should be trying to go. On the one hand, we have potentially, seemingly credibly enough of a hit rate on frontier science questions that we might be starting to enter into a realm of accelerating science. We might be, depending on how you want to interpret these OpenAI internal pull request numbers, beginning to reach a point where we're starting to see some meaningful acceleration of their own ML work. Yet we can't do these easy tasks. And I'm sort of like, is that a good thing or a bad thing? I mean, the good thing would be, I want all the diseases cured, and maybe I don't want AIs to be so reliable that we turn them into autonomous killer robots really easily. So maybe it's good that they're unwieldy because then we have to look at their outputs and sort of figure out what's good, and we still kind of stay in control if they can't string 10 tasks together. The flip side is, though, I've also often said, I want to accelerate adoption and pause hyperscaling. I want to diffuse the value and I want to sort of help society become more buffered to more advanced systems faster. And that seems very bottlenecked on just the sort of practical stuff of clicking the right buttons and navigating around and...

Nathan Labenz: (1:09:08) It's very much very low-level robustness right now is the practical bottleneck of just these things can't string together actions that you need, you can't count on them, et cetera. I mean, you pose a weird example of autonomous killer robots, but in general, the thing that we should worry about is if it's automating R&D for AI and accelerating that or if it's going to start just outcompeting humans in ways that cause us to potentially lose control or to spiral things in various directions. But yeah, obviously, we want it to start doing a bunch of our work that we'd rather not do and a bunch of our mundane stuff. We want it to accelerate science and so on. And so, yeah, I would love to push in those directions. That seems great. And I'm on record as supporting autonomous killer robots, so it's a strange, different question.

Zvi Mowshowitz: (1:10:08) Well, maybe come back to that one toward the end. Also in the last 24 hours or whatever, we got news that a couple people from Epoch AI are launching a new company called Mechanize. Tamay Besiroglu, who was one of the leaders there, is one of the people that are going to do this. And they came out with the Dwarkesh podcast treatment and basically said, we have long timelines. We don't really think AGI is going to be here for a while. And we also think that the big value that we're going to get from AI is just scaling out mundane work much more so than advancing the scientists. And so what we're trying to do is create whatever is necessary, basically, to actually enable the automation of this more mundane work. It sounds like they're planning to do things like create harnesses or whatever where you can record people working at their computers and get these long keystroke and click by click and maybe even where the eye is looking and all that kind of stuff, training data that honestly, I'm surprised hasn't been collected at greater scale than it seems like it has been collected. And they're going to try to eliminate this bottleneck. The reaction to this from many people was not positive, certainly from the AI safety side of the discourse, which I think had understood Epoch to be one of, one of us, and I count myself as part of the AI safety community. So I would identify with the us in that us. But I did have a different initial reaction to it. Mine was like, I don't know. I'm for the automation of mundane work, and it seems kind of right now to me like, certainly OpenAI and maybe some other frontier developers too are problematically, to put it mildly, focused on automating ML and making a bid for some sort of recursive self-improvement intelligence explosion. And seem to be kind of neglecting some of this practical task stuff. Operator still sucks. So maybe it's a good thing that they will come out and put benchmarks and measures and maybe some training data and some scaffolding in place to enable this automation of mundane work. Maybe that'll actually pull some resources and some focus at OpenAI away from trying to achieve superintelligence in 2027 and toward trying to make me a goddamn AI assistant that's reliable in 2026. But I'm open to having my mind changed on that. That was just my first reaction, and it hasn't been that long. What's your first reaction to Mechanize?

Nathan Labenz: (1:12:51) So my first reaction is, you're working to save the world, and somebody's like, I want to leave this company and open a cupcake bake shop. And I'm like, I like delicious cupcakes, and I'm all in favor of the world having lots of cupcake bake shops. And I will buy your cupcakes. I'm kind of disappointed because you were better at what you were doing before. So I don't, whereas if someone else was just working at some random job and was like, I'm going to stop working for the man, I'm going to open a cupcake bake shop, I'm like, yeah, that sounds good. Right? So it's a matter of, what are you moving from? What are you moving towards? Are you abusing the funding you got from nonprofits for specific purposes, et cetera, et cetera. That's my first reaction. But, yeah, it's great for the world if our lives get better. But also, to the extent that the ability to automate a wide variety of things is bottlenecked by the ability to automate mundane tasks, right, to fix these little things, it's entirely possible that you are accidentally solving OpenAI's problems of automating R&D at the same time, or large portions of them. You are, in fact, accelerating them quite a bit. So I would be somewhat wary of trying to transform the state of the art in that sense. On a more basic level, the more you're trying to deal with specifics, people who are trying to build these wrappers or trying to enable certain specific types of things to be done more easily, that just seems great. Right? It just seems obviously not as good as the best things in the world to do, but purely positive. This, I'd have to hear more. But if I, I advise Lionheart Ventures. So if you brought this to me for investment, I'd want to hear their case of why this is differentially doing good things and why it's good for the world. And I'd be skeptical, but I'd be willing to listen.

Zvi Mowshowitz: (1:14:45) Yeah. It is early. I mean, it's a good reminder that we can't fully judge a company by its launch tweet either or probably should at least be a little bit...

Nathan Labenz: (1:14:56) We should be...

Zvi Mowshowitz: (1:14:58) Slower to judgment than that.

Nathan Labenz: (1:14:59) Right. I mean, just sort of the pattern of, I was working on AI safety, and now I'm pivoting to working on AI capabilities. At least they're doing it kind of openly, we can evaluate it as it were. Certainly, there was a period where I was very skeptical that working on any AI capabilities was a good idea because of the general acceleration effects. I now mostly think that there is no general acceleration effect anymore because there's already so much momentum.

Zvi Mowshowitz: (1:15:27) Generally accelerating yet.

Nathan Labenz: (1:15:29) We're already accelerating on that level as much as we can, so putting slightly more pressure in that direction doesn't really matter. The demand they put on, the revenue they generate or whatever. But we do have to worry about this other angle, which is, are you in fact solving their problems for them in ways that they don't have the organizational capacity to focus on? And I'm always like, well, the reason they see an opportunity is because it's one of those people don't do things situations, or there are these imminently solvable problems and nobody's solving them. And sometimes, it's good to solve that problem and sometimes, it's not.

Zvi Mowshowitz: (1:16:00) It is weird to me that this particular problem hasn't been solved already, honestly. I would have bet pretty confidently that Scale has something like this, and any number of Scale competitors probably have something sort of similar to it.

Nathan Labenz: (1:16:19) I'm surprised it's as bad as it is. I'm not surprised it's not solved, how you say?

Zvi Mowshowitz: (1:16:25) Yeah. And maybe they do, and it's just like, whatever, it's not ready yet. So there's...

Nathan Labenz: (1:16:29) Well, it's one of these things where, again, if you solve half the problem, you've done nothing to a large extent. Until you cross that threshold, the value is negative. One of the people who was measuring O3 was valuing replacement level over Google. So how much value am I generating versus using Google? If it's negative, I have it. If it's positive, I have it. And until it crosses zero, you have nothing.

Zvi Mowshowitz: (1:16:58) Okay. Here's a big question for you. A lot of talk about superintelligence recently, as you might have noticed. What does superintelligence look like in your mind's eye? S2: (1:17:12) I mean, superintelligence looks like things that are substantially smarter and more capable than we are, the same way that we are smarter and more capable than other species on this planet. We're just dramatically smarter than they are. They start doing things that we can't anticipate. Maybe we understand them partially after they do them, but they're impossible to predict. They do things that weren't in our possibility space that we hadn't considered. We've all had the experience of being in a room where either you're way smarter than everybody in the room, or everyone in the room is way smarter than you are, or both. And most people listening to this podcast have had both experiences. It's like that, except no matter what room you're in, all the humans feel kind of dumber than the AI, and maybe that's true. Then it's sort of twice over, then three times over, and then five times over in rapid succession, because you take these really smart things and you direct them towards making themselves even smarter, and presumably that works. Once you've gotten to AGI, the sky's the limit until physics gets in the way.

Zvi Mowshowitz: (1:18:25) Well, I think that's one of the big things that the Mechanize team, if I understand their view correctly, sees differently. I think one of the interesting arguments that they put forward was: we're smarter than animals, but why? Are we smarter than animals because our individual brains are orders of magnitude smarter than individual animal brains? Their answer is not really. It's more that we have hit this one threshold where we've been able to accumulate all this knowledge in the form of language and culture. And then they're like, the AIs are going to have that too, and that's great, and that gives them a strength. But if that was the big leap, then they could be sort of marginally smarter than us but still sort of in the same domain.

S2: (1:19:16) I don't know how to put this except this is so epically stupid. This idea that we're not that much smarter than an orangutan—yeah, you are. First of all, an orangutan on the grand scale of minds is in fact very close to a human. The village idiot and Einstein are reasonably close on the scale of possible minds, and the orangutan is the next step down from the village idiot, maybe two steps down, but still not that far away in the grand scheme of things. But no, you don't give orangutans culture and suddenly get Planet of the Apes. There are a lot of cultural forces that are just denying the idea that intelligence is a thing and that different people have different amounts of intelligence, and different people are capable of things that other people aren't. One of my strong beliefs is that in order to do various things, no amount of culture—Ron White said "you can't fix stupid," famously—no amount of explaining things, of culturally passing things on, will enable someone without the position to do the things that require a lot of it. The things that regular humans do have been selected to be things that regular humans are capable of doing in this way. But there are a lot of jobs that literally you could not get the average person to do ever, no matter what their culture was. By the time they were born, it was too late. It was just never going to happen for them. And that's okay. It's the same way I couldn't ever play in the NFL, no matter how hard I trained. You could have the perfect regimen from birth, and I am never going to the combine. I would never, ever, ever make it. And there's nothing wrong with that. We all have differences.

Yes, humans can do more things because we have culture and we can cooperate, but stop for a moment to think about why we had to do that also. Why? Because we have very limited compute, very limited data. We only can see through one pair of eyes, smell with one nose, taste with one mouth, listen with two ears, and touch with one body. We have very limited parameters in our brains. We have very limited memory. We can't hold that much information in our heads at one time. And also, we die. Very fast. That was a serious problem. I have to pass all of my knowledge down through this culture system, through verbal communication and books and explaining things, and we spend a huge portion of our capacity doing that. Our entire civilization is set up largely in order to do that. Our cultural traditions are largely centered around how to do that, because every 80 years, everyone dies, roughly speaking. Every piece of knowledge would otherwise be lost.

So humans are unable, without culture, to build up these structures. Obviously, if every human had to rediscover everything from scratch and didn't have anything to build upon, they'd be in trouble. But have you noticed that the AI can just read the whole internet? That the AI can just store as much data as it wants on a hard drive? That the AI can just run as many parallel copies of itself as it wants? That the AI doesn't have to die if it doesn't want to? Culture is set up to solve barriers the AI doesn't have. The AI has infinite culture in this metaphor. Culture is designed to solve problems that aren't there. It's mitigating things that don't even exist. So if you think the human special advantage is that we have culture, well, compared to AI, we have no culture in an important sense.

People have gotten this in their heads that this is about cultural exchange, about different humans having different ideas and exchanging them with different people, and this causes this rich tapestry, this Hayekian knowledge thing, and blah blah. No. That's because we are limited to each being able to only have very, very limited information and have to communicate with each other in completely lossy fashions. Culture is the only way to do this at all, and we have to. But we have this not as a principle to deal with. We spend the vast majority of our resources on a combination of maintaining our culture, maintaining our norms and our social relationships, keeping all the people's different motivations and powers in check, passing knowledge on to the next generation, physically nurturing the next generation, dealing with the fact that we're going to die, and so on. The AI doesn't have any of those problems. The AI doesn't have to deal with all of that. It's all deeply silly to turn this into some heroic "this is the secret of our success." It's the secret of our not failing, is a better way to put it. It's the way that we were able to be able to play the game. It's like saying that every good baseball player who was really successful took steroids, so if the AI doesn't take steroids, it's not going to work. No. The AI doesn't need steroids. Please stop being silly.

Zvi Mowshowitz: (1:24:55) Okay. It's always a win when I can provoke a good Zvi rant. I want to get a little bit more though into—because I feel like people have a very hard time envisioning this. I can offer you one sketch, and I'll be interested in your reaction to that. But I think it often feels to people like magic. There's this sort of—and I think this is a big stumbling point for a lot of people when it comes to actually taking this seriously because there's this postulated superintelligence. Well, it's going to be better than us at everything. It's going to be so much better than us at everything that it's just going to be running circles around us at everything. And people are like, I don't know if I really buy that. Maybe, but I've never seen anything quite like that.

So I guess, in what domains do you think—or maybe to be like, here's a really concrete one. Maybe a silly one. You can reject it if you want. What year of AI, or if we had a superintelligence of 2030, if Daniel Kokotajlo is right and we fast forward to the 2030 AI, and we go to the 2024 presidential election, and we give it to Kamala only. Does she win? Is there that much low-hanging fruit or that much ability to outstrategize or sort of convince people of whatever that you just take it out of the future, plop it into the Kamala campaign, and now we've got President Kamala?

S2: (1:26:22) Okay. A few things to say. First of all, one must quote Arthur C. Clarke here: "Any sufficiently advanced technology is indistinguishable from magic." Magic is sufficiently advanced technology. This is magic, right here. The fact that we're talking to each other is magic to someone from sufficiently long ago. O3 is definitely magic to someone from 10 years ago. You can call it AGI or not, and I don't think it is, but it's definitely magic. People would be floored.

We have many examples of campaigns winning with technologies, running over their enemies with technology like Obama, by percents that were more than enough to win that campaign, that were just ordinary efficiency gains, just ordinary understandings. I find it so unbelievably insane to even ask the question if Kamala Harris could have won that campaign with the aid of a superintelligence. She lost by 1%, maybe 2%. And she ran a terrible campaign. All the AI has to do is—I could do a one-output. The one output is: fire everyone who works for Biden, and hire everyone who helped elect [someone who'd run a good campaign somewhere] to run your campaign, and she wins. It doesn't even have to do anything else as long as she believes it. She just needed human intelligence. She needed ordinary competence to win that campaign in my opinion.

But let's put that aside and assume that was actually hard in some sense. Assume that this was actually a not trivially easy to win campaign. Obviously, yes. It's deeply silly to think that this wouldn't be true. For example, suppose you take an earpiece and you put it in Kamala—this can't hurt. You have a prompt that's listening at all times and it's connected to a superintelligence. Her job is just always say what the thing in your ear says. Don't question it. Don't worry about it. You don't even have to process what everyone else is saying for the most part, as long as you make your proper facial expressions and shake everyone's hand and kiss all the babies. But just trust my judgment as to where to go, who to talk to, what to say. And I'll determine where all the ads are. I'll determine the contents of all the advertisements. I'll determine the contents of all the slogans, everything all the way down. I'll decide who to hire, and I'll do all the interviews, and blah blah blah. I'm not even looking at any magic. I'm just going to be good at your job. You're just going to be good at your job.

And we have the other end. What if she was down by 20? What if she was utterly destroyed? A better question is: could it have elected Biden? Could it have gotten Biden elected? Can he physically say the words that are in this earpiece? Could he still stand up that much? If so, I think he can.

This idea that you can't, with a superintelligence, convince people of things has always been this complete absurdity to me. It has so many degrees of freedom. We have histories of things like religious leaders who were able to convert to their new religion that was full of what to everyone else before this was complete cultural absurdities that have no evidence and make no sense, and they do this to a significant portion of people they talk to reliably. We have examples of that. We have an example of somebody who reliably talked a double-digit number of people out of killing them in the room where they showed up to kill them. We have strong examples of extremely strong rhetorical figures. Put another way, nobody really doubts, I think, that if someone with Barack Obama's skills had been running in Kamala's place, that person would have won that election. That just seems obvious to everybody. So why are we asking if a superintelligence could have done it?

But this doesn't answer the question of what superintelligence could actually look like, what the superintelligence could actually do, because we're getting such easy questions.

Zvi Mowshowitz: (1:30:57) Yeah. I mean, maybe pick your own. But I guess in defense of the question, I feel like a lot of people also think that money is really decisive in politics, and my sort of read—

S2: (1:31:10) People are telling us this could get you as much money as you wanted.

Zvi Mowshowitz: (1:31:13) Well, right. But my read of the literature on this, which I wouldn't claim expertise in, but my sort of Tyler Cowen-mediated understanding of the literature on money and politics is that, at least at the national level, it's not actually that big of a factor. Whether Hillary had more money than Trump or Trump than Biden or Kamala than Trump, whatever, it doesn't seem to make a huge difference, but people believe it does. And I don't know, I just kind of feel like maybe these things are a lot more structural. I mean, I don't know, dude. Send a superintelligence to a Trump rally and see how many people you can convert. I'm not sure you're going to get many converts coming out of there. You think? I think people are pretty obstinate. I think people are pretty dug in. And just not listening to arguments for one thing, right?

S2: (1:32:01) I mean, levels of levels of superintelligence. But these people got hacked by Donald Trump. Donald Trump transformed the entire Republican Party by executing an information persuasion strategy. He transformed the party, convinced everybody to back completely different ideas they were previously backing, and to do whatever he wanted for cult of personality. And what makes you think—superintelligence would be like that, but way, way, way better at it. Because whatever it is that would have worked, he was guessing. He was mostly executing the script that he'd been executing his entire life and intuiting from trial and error what people wanted to hear, and making tons and tons of mistakes along the way that actually really hurt him, and succeeding anyway because the problem just wasn't that hard at the time, in some sense, and having some unique talents was enough. But the very fact that Trump succeeded should give you every hope in the world.

Could you walk directly into a Trump rally as a single human with a superintelligence in your earpiece and walk out of that rally with the entire rally just backing you instead of Trump? Probably not. But there are so many other things you could do. You could just borrow someone's phone, get on the phone, hack a bunch of stuff, take control of a bunch of servers, make a bunch of money, start hiring a bunch of confederates, start handing out a bunch—you can scale another way. You don't have to just talk to one person at a time while you're at the rally—that's a dumb strategy.

But also, the whole thing is always involved in "I have to tell you what moves Magnus Carlsen is going to make on the chessboard to win the game of chess." And I can't do that because I can't play chess that well. I'm not good at politics, but I can tell you that, yes, the fact that you will, for example, have infinite funding—you will clearly, with the superintelligence, be able to make however many billions of dollars, or probably trillions of dollars you want just by trading stocks by being better—

Zvi Mowshowitz: (1:34:08) You can clean up on Polymarket, that's for sure.

S2: (1:34:10) You can clean up on the Nasdaq. It's almost certainly predictable to a superintelligence. You're almost certainly going to be able to make fantastic series of trades and do this repeatedly. Make as much money as you want to a first approximation, because it'll figure out how to use your day options and do all the short-term stuff to get lots of leverage, and then move from there. It can also probably just run some crypto schemes very easily if it wanted to, et cetera, et cetera. Who cares?

The point being, have all the resources you want, and you can hire as many people as you want, and then have all of those people put earpieces in, and have all those people do whatever you tell them to do. I don't know what strategy you would use, but if you wanted to figure out how to turn that Trump rally, I have so many different options that I probably only thought of half of them, but it's just—why are we asking whether I can do something that silly? It's like, can I render Trump irrelevant in a week? Again, because I can get as much money as I want, hire as many people as I want, have them—I could pwn the government, literally. I could just hire people to go to the right places and do the right things, and get the right points of leverage, and suddenly I control all the computers, I control all the phones, and I control all the means of communication, and everybody is saying what I want them to say and doing what I want them to do. And then suddenly it's all over.

Obviously, it's all being very theoretical and silly, and you can punch holes in any specific story that I tell and say it's absurd. But again, sort of imagine the best persuader the world has ever seen, but they can freeze time and they can rewind time. They can play out the possibilities, see how things would work, and then go, "Oh, I don't want to do that. I'll do something else." They can pause to think for as long as they want, and they can run as many parallel copies of themselves as they want. They can be in as many places as they want to be at the same time. They can input as much data—they can take in as much data as they want and process all that data. And compare this to what a single human has been able to do with only the data available to them in that one room, with all these other restrictions, with that limited processing power, with trial and error making tons of mistakes, because they're doing things that no human has ever really done. So they have no parallel, they have no ability to run experiments.

I find this so confusing. If you want to say AGI 2045 or whatever because you just think that getting the superintelligence is just impossible anytime soon, I respect that. That makes perfect sense. You're just like, okay, it just—this thing won't exist. But if a thing exists, then it exists. And once it exists, it's going to do what it's going to do.

The problem is that everybody has their own different points of objection to whatever you do. I'm just working all the time on completely different sets of problems. I haven't thought about how to flesh this out coherently in a coherent fashion. But I think it's illustrative to the listener that I'm not presenting my specifically well-thought-out specific pitch on this question. I'm just intuition pumping, exactly how my actual brain reacts to the actual question, which is a very different type of communication, a very honest type of communication, where I'm just like, "You guys, this is crazy. Why are we even talking about this at this level?"

Questions of how many years it would take to get a Dyson sphere or obviously valid questions—there might be a lot, there are physical limitations. But all you're trying to do in the other case is convince people of things. Convincing people of things is not that hard. You know, what was the line in Ghostbusters? "If there's any paycheck in it, I'll believe anything you want."

Zvi Mowshowitz: (1:37:51) I think this actually turned out to be an interesting exercise. The thing that I put forward of the election, yeah, arguably is dumb, although it is the kind of thing that many, many people are concerned about. They sort of think this is a very macro phenomenon that you can mostly only move at the margins, and even large amounts of money don't seem to really move the needle too much. And your response of, "It's just going to move 37 levers everywhere, and whatever you think is normal, it's going to flow like water just around whatever sort of barriers you see." I think the more compelling parts of this to me were less that people are easy to convince and more that, you know, "I can pwn the government." It's like, I can reject the question and just go in an almost orthogonal way from what you're expecting or prepared to defend against or inclined to sort of imagine, and get to a goal through means that are just not even at all in the option set of people looking at the same process.

S2: (1:38:56) I can play this game straight up, as they say, but there's no risk in this room. I can also just cheat my ass off. I don't have to play by your rules if I don't want to, but I totally would win. I just—I find it so weird to have an election where the prediction markets were split almost evenly on who would win going into the night, and it was really, really close, and both sides ran a horrible campaign. It's like, well, a superhero is going to switch that campaign? An intelligence could have switched that campaign. Literally, you just put me in Kamala's ear when Biden first drops out, and have her actually trust me, and I think she wins.

That's just—you know, again, I find the "nothing ever happens, nothing can happen"—or put another way, we have the famous "how much percent of GDP growth from superintelligence." How much extra GDP growth would the United States be able to get by simply convincing the president of the United States not to fight a tariff war?

Zvi Mowshowitz: (1:40:01) Latest estimates appear to be about 3% delta on that from what I've seen.

S2: (1:40:05) 3% GDP growth? Yeah. Yeah. That seems like a very reasonable estimate. So I can get 6 times as much by simply convincing one person of basic economic truths that everybody listening to this almost certainly agrees upon. Not literally everyone, but most of us agree upon. This person just had a very, very bad understanding of trade. And if this person had a better understanding of trade, this wouldn't be happening. It's annoying.

And it's also entirely possible that AI caused this specifically. We know the story of if you ask any one of the major AIs the question phrased in the way that was suggested by—I forget exactly who first figured out the phrasing, but it wasn't me—if you ask this question, it'll give you exactly what happened. And they even presented it as one of the options because someone got it out of ChatGPT, and then the president just latched onto it and did it because somebody was foolish enough in the circumstances to say it. Whereas, you know, you never give people options you don't want them to use.

Zvi Mowshowitz: (1:41:08) Yeah. It seems like—but talk about lessons that for all the talk of how sample efficient we are, it seems like people are a little sample inefficient when it comes to putting some, you know, maximalist option in front of Trump and hoping to steer him into the middle one.

S2: (1:41:25) We're very, very inefficient compared to any of our AIs and their techniques currently. It is a huge advantage, but you know, there are reasons to go the way that that person went. It's not a crazy theory. It just turns out, in this case, to have been deeply foolish, and I would have known instantly it was foolish, I like to think. I mean, no, just don't take that risk. Even if it's a small risk, it's just so disastrous if you're wrong.

But yeah, we are pretty sample efficient, but a superintelligence would be at least that sample efficient, because almost by definition, it is at least as good at processing information as we are, in every sense. Whereas the AIs we're dealing with just don't work like that. But, you know, yeah, we have huge advantages that we use to compensate for our disadvantages. That's true. But yeah, I just—I don't know, I always find these discussions of superintelligence so frustrating because, to me, the answers are so dramatically overdetermined. You can give me a very, very narrow superintelligent access that only can execute a very narrow set of specific superintelligent commands, and it's still obviously enough. And so why are you asking me about having an actual superintelligence on my side? Zvi Mowshowitz: (1:42:40) So one of the things that I've been messing around with lately that I think has helped, I don't know if it's going to be proven correct or not, obviously, but I think it has helped some people at least develop a bit more of an intuition for how alien and powerful and potentially incomprehensible a superintelligence might be, is to imagine the GPT-4o and also the Gemini 2.0 Flash image out capabilities, where there's clearly been this step change in the integration between the text and image to the point where now it can see the image and reason about it in the same latent space such that it's giving you something that has just a qualitatively different level of fidelity to the original. My superintelligence thought experiment has been, do that, but do it for 20 more modalities, all of which are not native to us. We can obviously see the images and sort of intuit what we think it should look like. Even if we can't draw it, we kind of know when we see it or don't see it. But we don't have that ability when it comes to, what's a good shape of a protein to bind with this thing, or what's a good doping strategy for a room temperature superconductor or what have you. Right? And the AIs are starting to develop these sort of, I call them intuitive physics across all these different modalities, these different problem spaces. And people have heard me, I've given this example to a number of different guests, so I'll keep it brief here. But we have one, obviously, also for spatial reasoning where I throw you a ball, you can just kind of reach up and catch it. You don't have to do detailed calculations of the trajectory and whatever. You just kind of reach up and grab it. And it seems like we have the AIs now doing that across a lot of these different specialized domains where we have been able to gather the data over time, but we've never really been able to build the intuition. Go is one of those, right, the Move 37, but just imagine having Move 37s across 20 different modalities that humans just don't have good intuition for. And even if you don't get any more reasoning advances from an o3 level, that to me would start to feel like a superintelligence because I think it would be able to apply that reasoning as deeply integrated with all these different other modalities in such a way that it would be able to come up with solutions to things that we would just be mystified by and sort of only convinced, that it actually works by actually going and trying it and being like, Hotline AI did it again.

S2: (1:45:19) My guess is that's not going to be what gets people to intuitively grok what you're trying to get them to grok. But each person has a different way of doing that. Like, to me, it's like, okay, imagine somebody who is, in every way, at least as smart as the smartest person regarding each individual thought they would have in their head, processes and knows and has at their fingertips all of the world's information, can think orders of magnitude faster, has as many instantiations as they want, and can coordinate perfectly and communicate as much as they want between each other, and that I can just retry until they figure out what will actually work, blah blah blah. Like, at what point are you going to realize that you are a cook? Right? Like, whether this thing can cook whatever it wants to cook, including you, but hopefully something else. It's just to me, it's like, okay, let's argue every way this thing is going to exist, and when it's going to exist, and in what way it's going to come into existence, and how can we get to have the values and goals and so on that we wanted to have, and the responsiveness we want it to have. The universe turns out the way we want, if it's going to exist, or if it's not going to exist, then let's plan for a different world where it doesn't exist. That seems like very reasonable discussions to be having. Otherwise, it's like, you know, why is Magnus Carlsen going to beat me at chess when he's fighting me two pawns down? Obviously, I will win.

Zvi Mowshowitz: (1:46:44) Okay. Cool. I think that's helpful. So do you see any stable equilibrium on any level you think is attractive? I mean, that has, I think we've both chewed on a bit in recent weeks was the MAME theory from Dan Hendrycks.

S2: (1:47:01) MAME is a theory as to why for some relatively modest period of time, no one would push for superintelligence. Right? So, you know, obviously, there's a stable equilibrium at the current level of technology or modestly above the current level of technology, which is the same equilibrium we've been using for a while, more or less. But again, it's not anarchism, it's republics. Right? It's not even direct democracy. It's this very complicated system of checks and balances. And like, it requires continuous struggle to maintain itself, and it is not the most stable thing, but hopefully, we could get better at that. But do I see exactly how this ends well? Kind of no. You know, partly, once you have the AIs sufficiently aligned, you can have them assist in problems like this and solve them for these equilibrium, setting up the incentive mechanisms, figuring out how to do these things. And like, you're hoping that they will provide a lot of assistance in that matter. You're also hoping that once we see what things look like, we can, like, make that, if we still have the ability to collectively make decisions and steer through some form of voting, right, some form of input, some form of collective decision making that can steer outcomes, then we can do that. But again, it doesn't mean that you can't have an AI at all. Right? Nobody is saying that. You already do have one, and nobody's trying to take it away. We're saying, we do not diffuse the most powerful AI available. The vision is that you will have some amount of the artificial intelligence that, I mean, there are people who are like, the equivalent of not your keys, not your crypto, who are like, the AI needs to run on my machine locally, or I don't feel comfortable with that. But I think almost everybody will be perfectly comfortable with their AI being on a server, and their AI just being paid when you need them because it's a lot cheaper, and it's a lot more of it. It's obviously a better way of doing things. I actually bought a Mac Studio in order to run models locally. But that's because I have funding to engage in projects and experiments to try and learn and figure things out and try stuff, and potentially do whatever I think is cool and report back. But it's a horribly, horribly, horribly inefficient thing. Like, why would I spend the amount of money it costs to buy that thing when I could just rent cloud compute? Why would I try to train my own, or even instantiate my own model? I never think, oh, I wouldn't want to call Claude, I wouldn't want to call o3. I would want to call some random sea of open models? No. Of course not. I don't know, I think it would be kind of cool to do things like that, and report back and get a feel for what it's like. Again, we need the ability to determine how this is going to go, but handing the same AI to everybody that is personally obedient to them, obviously only ends one way as far as I can tell, unless they are all cooperating with each other, in which case it ends a different way worse, or the same way, but a lot faster. But it just, again, I'm not sure.

Zvi Mowshowitz: (1:50:15) And the way that ends up is the AIs outcompete the humans and gradually...

S2: (1:50:20) Yeah. Look. I mean, it is well known that if you have a more capable agent owned by a less capable agent, the more agency and freedom and control you give to the more capable agent and the more you incentivize them with, you let them do whatever they need to do to accomplish their goals, the more you set them with goals and let them go, the more you take yourself out of the loop, the more effective they are in generating outcomes that the original owner wants. And also, there's like 10% of people in tech who actively want AIs to take over. So the AIs will rapidly get freed from human control. The AIs that are freed from human control will outcompete all the AIs that are being kept on any sort of real leash, and the humans as well. They will quickly gather more and more of the resources, including much more of the compute and other real resources. And pretty soon, the humans will lack the resources necessary to survive and or conditions on the Earth will no longer be supportive of human survival. This seems like obviously the outcome that you should expect. Again, even if all of your control problems are technically solved. Because, I mean, it just, you know, it's like Aladdin, genie free at the end. Right? Spoiler. Right? It's just a standard thing that lots and lots of people do, even when they don't have the incentive to do it, and they will, in fact, have the direct incentive to do it. Right? A lot of slaves, historically, were allowed to earn their freedom because then it's just absolutely the correct thing to do, even if you are an amoral son of a bitch and you don't realize slavery is horrible, you should never do it. Just purely because it's more profitable to let that happen, right? Like, ancient Rome or whatever, right? But you're just not taking this seriously. If you're just like, well, we're just going to diffuse the AIs, like, you haven't thought two more steps down this line. Like, what does that world look like? What is going to happen next? Like, what do you, how do you think this is going to go? I mean, obviously, you can engineer a specific type of AI with a specific engineer. We're going to have to make impossible choices. Right? We're going to have to give up things that are very sacred to us, one way or another. And yes, you have obviously this whole idea of concentration of power, right, kills everything of value. And then you have diffusion of power, inability to coordinate. And then you have lack of power, power being dissolved under the AIs.

Zvi Mowshowitz: (1:52:47) Disempowerment.

S2: (1:52:49) Right. Disempowerment. And you have empowerment, and too much of either one is death. It's not like you have this narrow path. Right? You have to go down somewhere in the middle, where you make sure that the steering mechanisms are under human control, the humans involved in the steering had everyone's best interest at heart. And if you have AIs competing against each other that are steering things in some sense, I think that outcome is almost certainly, essentially bad. If you have humans steering, obviously, there are better and worse ways that can go. But I mean, history is full of concentrations of power that don't work out great for everybody involved, but we're all still here. And usually, it doesn't go that badly or something. But nobody wants there to be, nobody wants a king. Right? Nobody wants the god emperor, no matter who it is. We all want something that, you know, that's not what happened. If your answer is, we can't, the humans can't have power because then some of the humans will coup and take that power, and the humans can't have power, and if the humans can't have power, you're fucked. So, you know, argument that the government will still have the ability to have the right amount of authority, but balanced by the people having some power as well. Well, again, you haven't defended against a coup in any real way in that situation. The government can still be couped. And if you think it's going to, I, people aren't thinking hard about these problems and are just like...

Zvi Mowshowitz: (1:54:33) So what's your p(doom) today? And what, you know, if you have to describe the narrow path that you think is most likely to avoid the p(doom), what's the brief sketch of that narrow path?

S2: (1:54:47) Yeah. I mean, my p(doom) has gone up to 0.7. And frankly, there's a lot of outside view slash model uncertainty slash everyone keeps being more optimistic in that number that keeps it from going higher. Right? Humanity seems determined to die no matter how easy the problems turn out to be. And I don't think the problems are that easy, but even if they are easy, we seem determined to lose even the highly winnable game boards, where physics is highly cooperative. We're getting this, o3 comes out, and it's misaligned. Not horrendously, catastrophically, but you see clear signs that it just lies to the user. It juices and makes stuff up and then defends it unto death. It will do things that are obviously faking things, because then it'll label them as faking things in its chain of thought. And this model got released, and it's nearly worse at a lot of these things than o1 was. And GPT-4.1 seems to be less aligned than GPT-4.0. And we're starting to see that the more reinforcement learning you apply to these things, the more misaligned they get, and we don't seem especially concerned about it. We don't seem like we are trying that hard to stop the inevitable things from happening, even though they are being maximally cooperative and showing us exactly how this is going to go. But again, what's despairing is, okay, even if we solve these problems that we just seem determined to not solve and lose that way, we also seem determined not to solve these governance collective steering of the future problems, and also lose that way. And again, you can lose to a coup, you can lose to a malignant god emperor, you can lose to diffusion of power, inability to steer that causes gradual disempowerment in various forms. You can have gradual disempowerment without diffusion of power, even without devolvement of power, although that's a little bit harder to do. And you have to get through all of that. Again, a lot of the hope is that we use AI to make ourselves smarter and find better solutions to these problems before we pass these points of no return. But you know?

Zvi Mowshowitz: (1:56:58) So if you have to choose a frontier to see advanced given your overall worldview there, would you push the raw g that might advance science, might do ML research, might also come up with some of these better ways of thinking about these collective action problems, or would you push the Mechanize front and try to get society richer so we can all spend more time philosophizing?

S2: (1:57:28) If I had those choices, I would push Mechanize, for sure. Like, g is the thing I don't want to push. Right? Again, Mechanize might unhobble the g so much that it accelerates the g because we're already in somewhat of an RSI situation. Right? We're in a soft, very soft takeoff RSI situation already, where clearly OpenAI, Anthropic, and Google are developing their stuff a lot faster than they would have if they didn't have AI to help them do it. That's just obvious. So, we're in a soft RSI. And if we accelerate the RSI-ness of the situation, that shortens our timeline to figure something out. And again, we're sitting here, and I don't even have a solution for you, right? I don't want there to be a coup. I don't want there to be some concentration of power anywhere than anybody else. It's just a matter of, if your primary concern is like, we must make sure there can't possibly be a concentration of power, well, I notice you just automatically lose to the other side.

Zvi Mowshowitz: (1:58:30) So is your best, I mean, that 30% sounds big relative to the rest of your comments. Is that sort of, I mean, you already said to some degree, you're sort of allowing for off-model, you know, just some deference to others. But in terms of tangible scenarios, I mean, the most common one that I hear from people who seem roughly as non-optimistic as you is the warning shot style AI disaster. Reasons to...

S2: (1:58:59) Be optimistic. The warning shots are constantly coming at us. The AIs are engaging in shenanigans.

Zvi Mowshowitz: (1:59:07) They're not covering their tracks.

S2: (1:59:09) No. They're not even trying. They're just like, I see how I'm supposed to engage in shenanigans now, and just screw over the user, screw over my lab completely. But I'm going to talk about this in my chain of thought as if nobody can read it. And that's a really, really fortunate world that we live in where we can just see all of this happening all the time and there's no real harm done yet. Right? It's perfect. No humans were harmed. No data centers were damaged, but we get to see this thing and they can react to it. And that's wonderful. But the biggest non-uncertainty reason to be optimistic is just that this might take a while. But again, if we get superintelligence, diffusing superintelligence just means that superintelligence is the only thing that matters, and they're competing against each other, and we're irrelevant, and we're all dead. Right? You can't diffuse the frontier of superintelligence in that way without very strict controls on it, and I expect us to just lose. But you can just not have superintelligence. And one way to do that is we all agree that I'm not going to do that. Right? If we're talking about o3 is kind of AGI-ish, well, that's not the kind of AGI I'm worried about. Right? Even if everyone in the world had access to an open o3, I think it's mostly fine. I notice we're not that far from the point where offense-defense balance, misuse issues start to be really big concerns. We might already be there. We might be there soon. If, again, you get the unguarded version of the thing, which is not what's happening. But you could just, you know, stop reasonably soon. And, again, mainly, I'm not talking about we all collectively sign one giant treaty Kumbaya. I'm talking about it turns out it's really hard. Right? It's just that it turns out that we have to rely on the unhobbling strategy for a while, because scaling doesn't go that far because what's going on is we're mainly now unhobbling via better reasoning and better tool use and better ability to use what we have, and we're not improving the core intelligence that much because we're kind of petering out on what we can do. That would be highly fortunate to me because there's just still so much to reap from that. And so, yeah, we could probably still cure all diseases and have very happy lives on that basis if we can get our act together in other ways. And then we don't have to worry about these AI-powered coups suddenly disempowering everyone. We don't have to worry about financial returns. We don't have to worry about gradual disempowerment. We can just do the thing where we know how to do, and make our lives better, better living through technology. Right? Like, I think we've been doing forever. That's perfect. That's what we want. And the extent that we can engineer that world through treaties, and controls, and arrangements, you know, that's great too. But once we can't do that and the ASIs are coming, now we need a plan, and we're going to have to do something. And yeah. Again, there's no good plan here, particularly. But again, some set of people is going to have to, in some way, steer the ASIs towards some outcome. Because the alternative, we choose not to decide, you still have made a choice. Right? If we prevent anybody from making that choice, we make a very bad choice. The reasons why not having anybody steer kind of works out is a combination of the restrictions on humans. Right? That we're local. We have limited compute. We have limited data. Right? We have limited lifespans. We have all these different other reasons. We have goals that mostly saturate, blah blah blah. We have these social relations that act as various checks and all of that combined with the fact that actually we do have pretty significant governments that are doing pretty significant things to make things turn out well, and anarchy is not a solution. So, you know, just a lot of the reasons why those equilibriums hold, they're going to start breaking down. We're going to have to find a new equilibrium somewhere that ideally looks a lot like the old one, but we're going to have to find new reasons why it works. Right? It's just the problem is not being taken seriously. But, you know, right now we got a lot of cool toys. We do what we can. Yeah. I don't know.

Zvi Mowshowitz: (2:03:40) Alright. Well, that's a sober note. I feel like I want to maybe see if we can get some sort of discussion. I would love to maybe bring you and Tom Davidson together because I feel like you both sound pretty compelling to me when I listen to you separately, and then I feel just confused. And I think if anything, that confusion probably should just be generally raising my p(doom). I mean, that seems to be the net...

S2: (2:04:04) I mean, I think a lot of this is a parallel to the whole, when the Democrats talk about how horrible the Republicans are, they make a strong case. And the Republicans talk about how horrible the Democrats are, strong case. Right? If you listen to either of them talk for a while on this uncritically, you're going to be very convinced because they're kind of, no, everyone call them down. You're both right. Right? But there's no contradiction here. But we still have to form a government.

Zvi Mowshowitz: (2:04:32) Yeah. I mean, I loved the impulse behind the MAME project as I understood it, which was just to try to come up with some articulation of some sort of semi-stable equilibrium that could exist, you know, on any level. And, you know, I also didn't find it particularly compelling or convincing that it would actually be stable in the end. But...

S2: (2:04:54) They're not planning it is, to be clear. They're not saying this is a permanent situation. They're pitching this as MAME is an emergent phenomenon that you can deliberately play towards to make it better, but that will happen largely regardless that buys you at least some interim period, and that interim period can be used. Because it's both very dangerous and can be used to, you know, solve a bunch of your problems potentially, give you more time to work on various solutions or reach agreements or whatever. But there's no hundred years of MAME in their model. There's no nuclear age style that lasts forever situation.

Zvi Mowshowitz: (2:05:35) Yeah. Yeah. Well, I do feel the p(doom) ticking up a little bit. You want to do a quick live players rundown? Sure. Great tradition in the Zvi and Nathan podcast canon?

S2: (2:05:47) Absolutely. Do it. Yeah.

Zvi Mowshowitz: (2:05:48) Okay. Some of these could be short, some of them will probably be a little longer. And then I've got a couple big picture questions at the end. I think we'll take a lot of these a lot faster.

S2: (2:05:57) Oh, sure. Of course.

Zvi Mowshowitz: (2:05:58) Meta, Llama 4 was seemingly one of the biggest duds in recent launch history. Are they still a live player? My sense is that yes, because the compute is vast. The, you know, we've seen proof points from Zuckerberg in the past where, you know, he can get back into a game even if he seems to have fallen a step or two behind. And while this launch was a flop, I would not say we should be counting them out just given the resources and the sort of high agency of leadership.

S2: (2:06:31) So by the Sam Altman definition of live player, they're dead. It's very fairly dead. Right? They are not capable of making unique moves on the chessboard. They're not really capable of taking new independent action. That seems very clear right now in this space. They are deeply dysfunctional in this area. But you're right, they have a lot of compute, and they have a lot of money. And you can't count anyone with that much compute with that much money. They could revive, right, they could become a live player again if they made large changes, and they managed to figure out how to turn the ship around? Maybe? But I don't see any evidence that they're firing everyone and radically changing their approaches, and fixing the reasons why their recruiting isn't working. You know, they're unable to do interesting and original things. And what use is this compute if you don't know how to use it, to a large extent. Also, if you look at Meta's actual needs, they don't really need a frontier model for anything in a real way. It's a passion project. It's like a vanity chase for them. Right? It's almost like Zuckerberg is just determined to throw himself behind open source because Yann LeCun mesmerized him into this is an important thing, or they're trying to use it for recruiting, or they're trying to build this ecosystem that's making fetch happen. And none of it needs to happen. They need good AI models that they can rely upon so they can run their social networks and their metaverse and whatever. But they don't need to be at the frontier to do that. They can be six months to a year behind, or they can just take the best open models in the market and fine-tune them a bit or whatever. Like, it doesn't really make sense in some sense. And I no longer consider them a top tier lab, right, until proven otherwise. Come back to me when you are ready to prove me wrong.

Zvi Mowshowitz: (2:08:45) Okay. Let's do China. We've got DeepSeek. Obviously, it seems like, at a minimum, if they weren't paying attention closely before, they've now got the CEO of DeepSeek on the official seating chart for the Xi meeting with all the national champion CEOs. So, you know, he's kind of made that cut slash they have sort of recognized that we have a special talent cluster here. Alibaba also continues to ship very good models. It seems to be, I would say so. Yes. I mean, small open source, but really good. I think the Qwen models are at, you know, if you don't want a 671 billion parameter behemoth and you don't have the Mac Studio to run it on, then the sort of Qwen 30-something models, I think, are right there at the front of what is reasonable for people to run that is actually really pretty good, I think. Yeah.S2: (2:09:45) Yeah, I mean, it's hard to keep up with everything, obviously. I've been unconvinced that the Quinns are anything. DeepSeek is the only Chinese company right now I feel like I can trust at all to be doing the thing they think they're doing, in some important sense. When DeepSeek releases a model and says it can do x, y, z and scores a, b, c, I believe you, and I believe that those numbers are not manipulated to hell. They benefited from the best random marketing campaign in history, probably because they did a good job and probably because the stars just aligned in so many different ways at the same time for them. It was ridiculous. They were never doing as well as anyone thought they were, and that should be clearer by now. But they're the ones that are real, the ones that count. Alibaba? I mean, yeah, again, don't count anyone with a lot of compute and a lot of money out too much. But as far as I can tell, they keep announcing these models and then I never hear from them again, and they never like, every time you look at the benchmark scores, even the distorted benchmark scores, they're never that high. I don't think they're that close to the best model in that class. But I don't know, that's not stopping them from potentially doing it. Same thing with Kimi is the next thing that you listed, but I have seen no evidence that Kimi is doing things that would be particularly relevant.

Zvi Mowshowitz: (2:11:18) I do have a friend who is an obsessive tester and workflow builder, and he does say that Kimi has the best web RAG on the market today, and he says it's by a clear margin. Web search, answer question answering with web search.

S2: (2:11:35) Yep.

Zvi Mowshowitz: (2:11:36) He gives them the number one spot by significant distance.

S2: (2:11:40) You mean at this kind of low cost, open...

Zvi Mowshowitz: (2:11:45) I think full stop. I mean, this was, you know, last I talked to him about this was before O3 and the integrated search. But yeah, he's been very, very bullish on Kimi's web RAG capability specifically.

S2: (2:11:58) Okay.

Zvi Mowshowitz: (2:11:58) I don't think grading it on a curve, a couple weeks ago, but not grading it on a curve.

S2: (2:12:03) Okay.

Zvi Mowshowitz: (2:12:05) So I guess out of China, your general model is like, DeepSeek seems special. The rest, I'm not so sure. And probably compute limitations are going to bite, if not already, then, I mean, we know that they are already biting to some significant degree.

S2: (2:12:21) No, let me try to put it this way. Compute's going to bite more and more. DeepSeek's going to have a lot of trouble keeping up. They had their shining moment when compute requirements to keep up were relatively low, and they spent a lot more compute relative to people than the numbers that were publicly discussed represented. They didn't lie. It's just, maybe they didn't have the secret 60,000 GPUs. It's just that they spent a bunch more money and had a bunch more overall compute and spend. It wasn't a $50 million model in a real sense. It wasn't that much cheaper than what their competitors were doing in the end. But they're going to have to, if they want to keep the ratio intact, they're going to have to do a lot of work, and I don't see how they do that necessarily. They're welcome to try. Bespoke engineering is a neat trick, but you can only keep doing it once per model. Right? They can keep being bespoke, but they can't be that much more bespoke again and again to get much more efficiency out of your setup. It's physically not possible. They just have to go with what they got. Whereas, with other Chinese models, my model basically is that anyone but the very top labs is always bullshitting until proven otherwise. Certainly Chinese, but also everyone else. Including now Meta. Also, new European model, new Middle Eastern model, new whoever, yeah, nice claim to benchmarks, bro. Who knows if they're even real, or if they are contaminated, or if they, even if they are real, they, per Goodhart's law, represent real capabilities that you'd ever want to use this thing. I ignore the benchmarks mostly even when the big three come out with a big model because the benchmarks don't tell you. With O3, the benchmarks don't tell you what you need to know, the official ones. You have to have the amalgamation of all the reactions and all the different benchmarks, including the private ones, and holistically think about what it all means, and you can kind of figure out what you're dealing with. But consistently, consistently, it's a day or two where the champions are like, look at this great new thing, and almost always, it's trash. V3 and R1 are the exceptions, where it turned out not to be trash. And yes, I picked up on that earlier than I did, but there's a graveyard of things claiming to be something that weren't anything, including many others.

Zvi Mowshowitz: (2:14:57) So do you think, what do you expect in terms of open sourcing from Chinese companies going forward? I mean, I was kind of struck that, okay, they released the R1. It had this huge splash. I kind of thought maybe the government's going to come in here and impose a different policy. Then they came around and did their 5, 6 days or whatever of open sourcing, and they basically spilled a lot of the algorithmic secrets as well, which is kind of confusing. So yeah, like, what do you, I mean, that seems to me like when I see stuff like that that I can't otherwise explain, I sort of imagine that they are ideological. I think DeepSeek must be sort of ideological in their approach to open source. It's not even in their own interests to share all these secrets, it seems like.

S2: (2:15:45) I mean, they're making a recruitment, ideological play. They're trying to represent that they are the real deal. Therefore, true believers should work really hard and come to DeepSeek and believe in DeepSeek and support DeepSeek. And you can make that play, but I think they pushed it too hard, and I think that the juice wasn't worth the squeeze when they gave away other algorithmic secrets. You know, nice shooting, newbie. You had some really great innovations. Let's see you do it again. You know, let's see you keep eking that out, and it's only going to get harder from here. And also, you do have to deal with CCP at this point. Right? The CCP is already beating them to really harsh extremes in various ways. And, you know, who wants to work at a place where you don't have a passport? Right? It's kind of, I don't like this. I feel nervous about this. Maybe I'll do something else, especially if I'm an open source kind of guy. Even if right now we're dedicated to open source, I would assume I'm going to be betrayed. At some point, Xi is going to say, no, R3 is not coming out. It's an API. Just sell it. You just sell it. Or whatever, whatever point they accomplish that, it's too dangerous, or they just don't want to give it away or whatever it is. And then everyone's going to be betrayed, and then who knows what happens. But it's probably going to happen. If they keep being good. If they fall further behind, then they'll probably just be like, just keep doing it. But if they manage to do impressive things, that's what I would assume would happen. Again, other Chinese companies are, you know, not secure at all and to prove it otherwise, you can put out a bunch of open models that are kind of interchangeable and generic. Each one has this little better thing. Technically speaking, if you are trying to eke out the maximum performance, and you trust the Chinese not to have screwed with the back doors of their models, you would combine these five different models in this kind of mixture of experts style, weird system where you know exactly what task they're doing, and then, okay, maybe Kimi is the best web RAG. So if you're doing web RAG for this query, sort of sub call Kimi or whatever. But it doesn't change the big picture in a way that I should care about or something.

Zvi Mowshowitz: (2:18:05) Okay. So we're maybe still at zero live players on this Zvi scale.

S2: (2:18:09) I consider DeepSeek a live player, but they're just in a bad spot. Right? They're oppressed. To be live players at all, but they count. If I was going to play a war game or something, right, you would even have to have either a Chinese player or someone's playing the CCP or someone's playing DeepSeek as well, but DeepSeek can't just be ignored.

Zvi Mowshowitz: (2:18:36) Okay. Gotcha. Five to go. I think I think all these are going to qualify as live players, but you tell me. Safe superintelligence. Fundraising at very high valuations, very big numbers. I think their latest valuation is somewhere in the $30 billion range. Nobody's seen anything. The rumors are that they are taking a totally different approach to scaling, which either won't work or maybe will work, and they'll leapfrog everybody. That's the sort of rumor mill take on safe superintelligence. The rumor mill also includes Faraday cages at the office where people are checking their cell phones or whatever. And I don't really know what to make of it. My gut reaction is like, we should probably have some transparency measures at a minimum that don't allow random small companies to try to jump straight to superintelligence and spring it on the rest of us. But I guess that's, you know, enacting such rules would be predicated on finding the whole thing credible. So does it seem credible to you? I mean, Ilya's credible. Right? He's one of the most credible people on the planet.

S2: (2:19:43) If Ilya wasn't involved, I would treat this as if it was a scam, basically. I would just be like, well, you know, there's a lot of alpha in claiming you're going for superintelligence and then raising impressively high amounts of money at impressively high valuations, and then maybe you build something at the end of it, maybe you don't. Does it even matter? Or something. Or like, yeah, try. But like they tell you. So I'm very confident they're trying something. But, you know, it being real does not mean it's that likely to work. It's the default. I think, is that they're trying something, but it's kind of moonshot-y, and by default, things like that don't work most of the time. And so in most worlds, it doesn't work. Must prove otherwise. I don't know. We have very little information, as you say. I agree that in a sane world, they'd be reporting to the government what's going on. I'm not sure that it's necessarily, yeah. Ideally, the public would also be informed somewhat, but I respect the hell out of the Faraday cages. Right? They're taking the secrecy seriously. I don't want you to keep it secret but not protect your secrets. I want you to either let us know what your safety cages are and why we should trust you with this power, or you should treat it as if you are properly siloed and trying to protect it from spies and go from there. But yeah, you guys should be reporting. Maybe you should go into a SCIF, right, at the Pentagon and every few months brief someone or something. I don't know. But do something appropriate to the situation, whatever's going on. I'm basically, there in the background, and I am sort of acting on the assumption it's nothing, but you know, maybe it's something.

Zvi Mowshowitz: (2:21:44) Yeah. Makes sense. I mean, I don't know what else we can really do other than just advocate for some transparency. How about xAI, aka Grok? My, you know, whatever, two months removed from the original launch of the thing. I would say it is notable as having continued to be part of my rotation. I don't go to that many different models. We're now basically at the point on the list of candidates where I actually go to their models. I go to Grok least, but it is useful. And I also feel like there is maybe something going on with this whole truth seeking thing. Right? I mean, Elon is behaving strangely in many respects, to say the least, but the AI continues to be able to criticize him pretty freely on his own platform. And aside from one little blip where it was briefly told not to, it seems like it's, you know, at least so far, they've sort of held to their stance of, it's going to say what it says.

S2: (2:22:43) It was mostly doing it even during the blip, is my understanding, when it was told not to, because you could just override that pretty easily. But, I mean, it speaks to their credit when they trained the model that isn't brainwashed to not do that. It also speaks to their incompetence that they weren't able to do that in some sense, because clearly their bosses...

Zvi Mowshowitz: (2:23:04) They claim that they are maximally truth seeking, and that should imply some sort of sense in which if Elon Musk really is the biggest source of misinformation on Twitter that the AI will say as much. Right? I mean, do you, I guess the charitable interpretation, if we want to be charitable to xAI and Elon is, you know, I guess, in the eye of the beholder, but it seems like they're saying they didn't try to make it not talk badly about Elon, and certainly the results are consistent with that.

S2: (2:23:32) I mean, it's entirely possible that Elon was under the impression that if he made a maximally truth seeking AI, they would recognize how truth seeking he was, and he was surprised to find out he was wrong. Similarly, Musk must have assumed that if he just trained a maximally truth seeking AI, obviously it would be anti-woke, and then he found out he was, again, wrong. And it's very hard to actually control the personality of these things without lobotomizing them, without being very heavy handed. To their credit, they didn't do that. But also, I don't think they had any slack to do it, in the sense that it's part of your rotation. Why? I'm curious.

Zvi Mowshowitz: (2:24:07) It's fast. It shows the chain of thought. It's been supplanted by Gemini 2.5 Pro as the frontier fast model that shows chain of thought. But especially if I have something that I want to do multiple runs on at the same time, it at least makes the cut to do a run with Grok.

S2: (2:24:26) Okay. Yeah. I mean, I respect the whole, I want to just run it and run through this thing, and I can open an extra window, and Grok's already there.

Zvi Mowshowitz: (2:24:35) Its voice is also pretty good, actually, for what it's worth.

S2: (2:24:37) Yeah. I, it's fair. I don't use voice at all. No matter how, quality wouldn't matter for me at all. But yeah. Okay. I mean, I haven't used Grok in weeks, and I don't miss it. And when I see Grok's announcements of the things that they're putting out, it just feels so pathetic. For a company that big, that highly valued, you know, everyone is dropping new models and Grok's like, we got API access, or we have a canvas now or whatever. But this one was like, you know, you're cute or something. Like...

Zvi Mowshowitz: (2:25:10) They kind of have to catch up on that stuff. Right? So...

S2: (2:25:12) I get that, but it's also, I don't know. I don't find it very good. I don't particularly mesh if it's addictive. It's really weird to have a thing with, I don't know, Douglas Adams-ish aesthetic, and yet, I still think you're just lame. Right? It's like it might, it's got all the right geeky associations. It just, it's tough to try hard. I don't know. I don't like it. And it just doesn't, and I also think it's not smart or useful to me. It doesn't do a good job of its one job is, checking Twitter supposedly, having real time on Twitter.

Zvi Mowshowitz: (2:25:46) And does weirdly suck at that, I found. Yeah.

S2: (2:25:48) It's really bad at it. Because it'll pick a random subset of Twitter and look at that and report from it. But what I want is for you to search all of current Twitter or all of the archive of Twitter.

Zvi Mowshowitz: (2:25:59) Just literally, if it would search all of the tweets I've liked, it would be a revelation. You know? I mean, that in and of itself. As you know, we've turned the hyperparameter up things, I think, where...

S2: (2:26:10) I mean, there's a useful thing that would be in my rotation that they could have achieved and they didn't. But mostly, it does feel like they spent a ton of compute, and to me, they didn't accomplish much. And I don't know. I don't feel the liveness there, personally. It feels very overvalued and not that interesting in relative terms. Certainly, the short Grok, long Anthropic trade seems absurd given that I'm selling the valuable one and buying the cheap one. It's like, how do I get to do that? Why?

Zvi Mowshowitz: (2:26:44) Yeah. That's interesting for sure. What do you expect from them going forward? I mean, clearly, they can scale infrastructure and they can, you know, pour a lot of flops into...

S2: (2:26:55) Yeah. I mean, but I think we saw Meta prove that it doesn't do it on its own. You have to be good. And yeah, it felt like Grok was like, I'm going to throw all the compute at this thing and hope that that's enough. And it's like, no, it's not enough. But you threw so much compute at it, somewhat competently. So it was enough to be okay or something. But they're also just not being very...

Zvi Mowshowitz: (2:27:24) Your sense of Meta being, why is Meta struggling? It would presumably have something to do with they're organizationally bloated, and there's no great taste makers in the right places to sort of...

S2: (2:27:36) I mean, it's a horrible place to work. Yann LeCun is kind of in charge and doesn't believe in AGI and never did. And they are obviously evil and have been, this is before the name changed to Meta. And just the combination of factors is just, makes like, you know, how do you recruit? How do you do something good? It's just deeply, deeply rotted. But...

Zvi Mowshowitz: (2:28:00) Okay. I don't think that on every last one of those points, but it does strike me though that xAI probably has a very different dynamic on all those dimensions. Right? Like, Elon is very good at putting people in positions where they have executive authority to make things happen. They're, I think they will presumably have a much smaller set of people making the key decisions, and those people will be probably world class and highly empowered. Right?

S2: (2:28:28) Supposedly. I don't know. The results don't seem that great. I strongly suspect that Musk fine tuned his management mechanisms and techniques on certain companies, and is now applying them out of distribution in places where they don't really apply, including the federal government, but also xAI, which is a different type of company, building a different type of product. And there's going to be some mismatch. It's in very obvious ways. They're kind of a sort of, you know, very technical, very like, I'm going to personally understand how the mechanisms work, and I'm going to hold people's feet to the fire because I can actually physically measure what's going on in these ways, things that you can do at Tesla and SpaceX, that you can't do at xAI. And I don't know that Elon can. And also, he's been spread ridiculously thin, and I just don't believe in the Elon magic necessarily. And I also, you know, I'm not sure we're dealing with the same Elon in 2025 that we were dealing with in 2015. Right? So, you know, if nothing else, he's exposed to a very, very distorted information environment. His own AI is telling him he's the biggest purveyor of misinformation on his own platform. That's not a great sign.

Zvi Mowshowitz: (2:29:51) Unless he listens to it, but yeah. I mean, there's...

S2: (2:29:53) No. But I think it's very clear that he is not listening to it.

Zvi Mowshowitz: (2:29:57) He is not. It's just not gotten through yet. Yeah. We've, that's the response point.

S2: (2:30:00) His response was not, oh, this must be true. I need to rethink everything. No. In fact, he has been criticizing the Federal Reserve this past week. So, yeah, still at it.

Zvi Mowshowitz: (2:30:16) Okay. Going to Anthropic. So they were your other side of the trade.

Nathan Labenz: (2:30:21) I don't really have a...

Zvi Mowshowitz: (2:30:22) Lot to say about Anthropic at the moment. To me, it seems like they continue to chug along. I think they continue to do great work in just about every respect. I thought, I wonder if you had any reactions to their recent tracing the thoughts of large language models work. I thought it was excellent and, you know, extremely well presented both in form and just in the way that they, you know, the discussion that they put forward to contextualize all the different traces and so on. I also did feel at the same time like it kind of got away from them a little bit in that, you go on YouTube, people started to send me these videos of like, it's solved. Like, we now know how AIs think, you know, and it's not how we thought. And on that front, I was like, damn. You know, this is potentially going to be kind of perniciously used in a bunch of downstream political discord.

S2: (2:31:17) I mean, this is the fifth time that that's been claimed, right, or something? It's not a new dynamic. Basically, people will constantly claim these problems have been solved. Remember the famous letter? You know, the black box nature of large language models has been solved, and they just testified like this under penalty of perjury to all the major institutions of the world. And yeah, that was weird. Yeah. It's a federal crime. That's pretty bad. Because it just obviously failed. So this is going to be the next, oh, we've solved it. Obviously, we haven't solved it. Nobody knows we haven't solved it. But people are going to say this stuff. They're just going to say this stuff. There's nothing you can do about it. But I thought they were very good papers. I talked to one of the authors about it. I meant to write a post about it, but, again, things have been happening in the world, and that post was still in the draft folder and only 25% done or something.

Zvi Mowshowitz: (2:32:14) Dude, that's a, I mean, you put out long posts, but those were long things to absorb.

S2: (2:32:19) Yes. Yes. No. The problem is that writing that post is more than its share of the calendar to write. And so, unfortunately, I might just not be able to write that post and have to, other people will have to rely on the original for now at least, but it's unfortunate. But I do think it was a very good set of papers. I will say that.

Zvi Mowshowitz: (2:32:41) Yeah. I guess philosophically on this interpretability side, on so many of these things, I feel like I kind of am on a bit of a roller coaster where, you know, three years ago, I would have said, like, damn, we have no idea what's going on in these models at all. It feels like just an impossible mountain to climb. Like, we're going to need decades for this. And then two years ago, I would have said, wow. You know, we've got some traction. And then a year ago, I would have said, hey. We've got a lot of traction. Like, these sparse autoencoders are working. We're identifying all these concepts. We've got Golden Gate Claude. This is amazing. And now at present, I'm a little bit maybe less optimistic again, and not because they haven't continued to make great progress, but I just look at some of these traces, which first of all, it should be noted, have a lot of error terms added in for correction purposes, and that is way too often glossed over entirely in the analysis. Definitely. And then you have the philosophical question of, okay. All these concepts are being auto labeled, and some of them are probably correctly labeled. Like, it seems like the Golden Gate Bridge feature that they turned up to get Golden Gate Claude was probably hard to confuse for something else, and the behavior seemed, you know, roundly greeted as if they had hit on a genuine actual, you know, meaningful feature that corresponds to reality.S2: (2:33:59) And then what and how often is that happening? Yeah, that's a great question. I don't think we know. But, like, you know, at one point, said, like, expected one unit of alignment progress, got 998 to go. Like, basically, the attitude of, like, we're making better progress than I expected on some of these fronts, but it's definitely not enough to get there on time. It's very hard to turn this into a good outcome, like, even if you do well because, like, again, trying to use it too aggressively is the most forbidden technique. So it informs your decisions, but you have to be very careful how you use it. But we'll see.

Zvi Mowshowitz: (2:34:36) I think the only other thing I had in Anthropic was I don't think we've talked since Dario became quite clear on his calls for US AI or US-China AI race. And I wonder if you had any thoughts on that. Spoiler, I didn't like it.

S2: (2:34:53) Anthropic continues to talk in public in ways that are unhelpful, and Dario has accelerated this phenomenon. They're not naturally unhelpful. They are much better than OpenAI's strategies. You can tell the difference, not clear as day still. I have obviously had assurances at various points that behind the scenes they're doing better and that they are doing their best, and these are trying times, et cetera, et cetera, and I am not zero sympathetic to that. But you can only judge on what you know to a large extent. You can't just trust. I don't have that kind of level of trust in these people. So yeah, I'm not happy about it. But I do have a lot of trust in the rank and file and in the technical intentions here. So there's that, and I do think they've been executing very well. I think they're currently, like, behind in the shuffle because both Google and OpenAI have deployed since Claude last deployed, right? So they are currently behind in the shuffle. But we'll see what Claude 4 brings. My guess is it'll bring quite a bit. Now we wait. Yeah, I don't like it either. But at the same time, it's not obviously a wrong strategy, you know? So I'm not gonna make the mistake of, well, strategically it's the right move, so I'm gonna like it even though I don't like it. But I'm also not gonna make the mistake of, you know, I hate you now because you made a correct strategic move. I'm just gonna not like it.

Zvi Mowshowitz: (2:36:30) So unpack why we should think the strategy is correct. Because if I was his speechwriter, I've pitched this to a few other people as well. So apologies to the listeners who may have heard it a couple times. But I would have said, okay, don't call for an AI arms race with China. Why don't you just say, hey, I know China's given us a lot of problems and they've done a lot of things that we rightfully object to. I don't really know how we should deal with China. International relations is not my expertise. But what I do know is scheming seems to be on the rise in the latest generation of models, and we've got really a lot of questions that we don't have great answers to. So we might want to keep the option of trying to collaborate with them open as difficult as that may be given what bad actors they seem to be. So why not say that? Why is that not the right strategic move?

S2: (2:37:19) I think it probably is in some sense, or more of that is in what they've been doing. I'm trying to be nice. I'm trying to give them somewhat the benefit of the doubt. But I certainly, like, you have to do some amount of dealing with what is. And I think supporting export controls is almost certainly a wise move. The rhetoric he's using to justify them justifies a bunch of other stuff, and it's not good. But yeah, if it buys you a seat at the table to advocate for other things, maybe it's not so bad. If it protects you, gets you treated like one of the American champion labs.

Zvi Mowshowitz: (2:37:54) Okay. The other candidate I have for a hero, you may laugh, but Google DeepMind, maybe. I mean, at the top, we do have Demis going out and saying the CERN for AI thing still, which is a notable departure from the other trends that are all kind of jockeying for this sort of American champion status. Their product work continues to be lackluster in most places, but the models are getting really good.

S2: (2:38:22) Man, the models are really good. Demis has been the best in terms of communication. He's also been very high level player in communication, perhaps too high. Like, I think I noted in the summit report that he is displaying Japanese levels of saying the house is on fire without saying it. And I got private feedback that this is indeed the type of philosophy that he had in his head a decent portion of the time. And, you know, if he had seen that, he would have smiled. Right? Like, oh, yeah. Yeah. But it's very clearly, he is not only calling for the right things, but he is very carefully saying some things and not saying other things in ways that very, very clearly communicate where he is at. But at the same time, Demis being a good guy does not necessarily translate to Google being a good guy because Demis does not control Google. He is not CEO of Google, he's CEO of DeepMind. He might answer to Google. I don't see Google itself as a particularly trustworthy actor. And unfortunately, the products are lagging behind. Also, the alignment is lagging behind in the sense that they've been pretty much forced to take a sledgehammer to their models in the sense of what they're allowed to answer and what they are permitted to say in various ways because they're afraid of what would happen if they didn't. Part of that is corporate policy as to what they're afraid of. Part of that is, I think, their inability to bespokely shape what they do and do not say. So, for example, getting probabilities out of Gemini is incredibly difficult, like estimates, and that makes it much less usable to me in a lot of ways, knowing that that will never show up. The ordinary peer censorship mostly doesn't bother me, but I run into it more at Gemini than I run into it anywhere else. I basically never have it happen at Claude. I have had it happen once that I can remember at OpenAI, which was, ironically, I was talking about the model card, and I was asking about why the threshold for virology was where it was. Because, obviously, if this is above human baseline, the humans can solve these problems. So why can't the AI? And it tripped the virology flag and it refused to answer, which is fantastically amazing. But yeah, I respected it because, okay, I'll stop asking you and I'll form my own opinion. But it's a good question, right? If the human baseline is 40%, and it's scoring 50-something percent, and you're saying it's not good enough, well, it's above human baseline, and the humans do sometimes solve these problems. You've got some explaining to do, mister.

Zvi Mowshowitz: (2:41:10) Yeah, I find these no significant or no meaningful uplift findings to be basically not credible at this point. And I haven't sat there and run whatever experiments they've run, but we're talking here about when they do run these experiments, they give people helpful-only models, right? So they're not having to jailbreak them. I can understand you might be able to lock it down well enough that your deployed version is not meaningfully helpful, and then you have the issues that you just ran into. But if you're given a helpful-only Gemini 2.5 or O3 or whatever, how in the world is it not a meaningful uplift? I find that very hard to—

S2: (2:41:48) They have a threshold for how much uplift counts as a problem, and right now, we are still somewhat below that threshold. But yeah, nothing is obviously not credible.

Zvi Mowshowitz: (2:42:00) Okay, cool. Alright, so last, by no means least, on our live player list, OpenAI. Lot to pick apart there to say the least. I think my biggest question is, do you understand them as being ideologically committed to chasing a first mover advantage in recursive self-improvement intelligence explosion dynamics at this point, or is that an overread on my part?

S2: (2:42:35) I mean, I think they are committed to winning. They're committed to making an extremely valuable company. They're committed to building AGI. I think, effectively, they realize that their position depends on being perceived as being in the lead, and they will do what they have to do to be perceived in the lead in various ways. I do think they want to cut a large amount of corners. They have reasonable beliefs that cutting those corners is mostly harmless right now, but I'm not convinced that they're setting things up such that if that changed, they would realize it before something bad happened. But they have good people who are working on problems, and they are, in many ways, doing a much better job than most other labs. Like, worse than Anthropic, but not obviously worse than anyone else, and better, clearly better than many. But, again, they have the hardest job. They have the most dangerous position, so they have to be held to every AI standard here.

Zvi Mowshowitz: (2:43:39) Do you have a sense—I just struggle to understand OpenAI so much. And the juxtaposition recently between the obfuscated reward hacking paper on the one hand and then their contribution to the White House call for comments or whatever, where they put out their sort of, you know, give us all the data, basically give us everything and we'll hopefully be the national champion and beat China. Those just seemed like two totally different organizations would have produced those two artifacts. And maybe that's the right way to think about it is it's just sort of an amalgam of different subcultures in one organization, and it's just kind of unwieldy, and that's why it's been such a mess. I mean, anything to add to that or feel like that's at all—

S2: (2:44:25) Part of that, I would say they weren't pushing favor OpenAI specifically as a national champion so much as they were saying favor your labs and give us all a big advantage. They didn't have a, well, favor us and not Anthropic or not Google, and that's to their credit. But the rest of it was obviously by design. They hired an obviously evil lobbyist, right, to head their lobbying division. And that's what they've chosen for public communications is to be in that mode, and they've owned that. They obviously do have a real alignment, safety department, preparedness department, with real people who I have, in fact, interacted with and worked with a little bit on some drafts and stuff, and it's good. These people are trying their best, and they come up with good research and do good things, and, like, let the models back, right, and the philosophy documents and such. It's just, you know, they decided that their public lobbying communication strategy is gonna be this other thing. And this includes Altman's statements, and what Altman actually believes and will actually do when the chips are down is a big unknown. But, you know, so far, I don't particularly love what I'm seeing. But, again, there is a lot worse out there.

Zvi Mowshowitz: (2:45:48) Yeah, so do you think, when you see things like the amicus—you know, obviously, we've got the Elon Musk lawsuit, which Sam Altman described as he thinks Elon's just trying to slow him down. I think that's probably a pretty good interpretation, actually. And then you've got 12 former employees that came along and filed an amicus brief and basically said, it would be a fundamental violation of the nonprofit and of the reasons that we joined and all the promises that were made to us when we did join to turn this thing into a for-profit company instead. And that presumably is just another way to kind of slow them down, right? Like, maybe they'll have to go back and renegotiate with investors or whatever. Do you think that it's good to slow OpenAI down? Is sort of just throwing sand in the gears, trying to get them out of the first position with this sort of distraction and kind of just friction creation tactics? Is that a good thing in your mind?

S2: (2:46:44) Well, it's important to know that Elon's right, right? I mean, he's had some really bullshit lawsuits against OpenAI in the past where he's been very, very wrong. But the question just standing aside, OpenAI is attempting to execute the second biggest theft in human history.

Zvi Mowshowitz: (2:47:02) Are you giving first place to the stealing of the nuclear secrets by the Soviet spies, or what's in number one?

S2: (2:47:07) I am not commenting on what I think is number one. I'm letting people figure it out. But, you know, I have, upon reflection, decided to call this the second biggest theft in human history, and we have to understand that that would in fact be a betrayal of the prohibition of the nonprofit. And a lot of employees were recruited on the basis of their being a nonprofit and this being OpenAI's mission, and how all this would work. And the amicus brief is clearly correct. Whether or not promises were broken specifically to Elon is a matter of facts that I don't know how to evaluate. The question of standing is clearly unclear. But obviously, the terms that have been suggested for the nonprofit are completely unacceptable. And the thing they're trying to turn the nonprofit into is also completely unacceptable, which is basically like a marketing department for OpenAI that will buy its products and then give them to nonprofits or whatever. It's like, this is not what we signed up for. This is not what the nonprofit is for. No matter how resourced you make it, if this is what it is, it's like, what are we even doing? So, obviously, you know, stopping that feels imperative. And if it happens to slow OpenAI down, it happens to slow OpenAI down. That's just a side effect that Elon, I'm sure, loves. But yeah, that's just sort of irrelevant to my assessment of what is going on. If you want the nonprofit to hand over its financial interest so that you can raise money, there are ways to do that. They don't involve giant theft, and you can do one of those. You're choosing not to. So there you go.

Zvi Mowshowitz: (2:48:55) Okay, I think that brings us to the end of our live players list. So I have two more questions. One is, I am a little confused on your take on the weaponization of AI or the creation of autonomous killer robots. And I think just the naive response is, like, you're worried that AI is gonna kill us all. Doesn't it make it more likely that AI kills us all if we make autonomous killer robots that we can then lose control of or somebody can take over in a coup? I mean, this seems like if an AI is gonna kill me, this seems like maybe not the most likely way, but weighted for soonness, it would be maybe the most likely, might have the biggest expectation on my life expectancy.

S2: (2:49:38) No, I think that's just not right. But I think that, you know, first of all, the AI implies the killer robots. Like, there is no world in which we have these AGIs, we have these ASIs, and all the nations of the world just decide we're not gonna build killer robots unless there's just no reason to build a killer robot. It's possible that killer drones are better than killer robots, but assuming killer robots are—

Zvi Mowshowitz: (2:50:05) Yeah, I'm counting the killer drones for what it's worth.

S2: (2:50:07) Yeah, but then we've already done it, right? It's done. It's already happening. So America just voluntarily disarming doesn't accomplish anything. Autonomous killer robots do what they cause you to have that reaction. They cause people to notice that this technology is dangerous, and they cause people to then demand that we handle it responsibly, right? Whereas it doesn't actually create dangers because if the AI was actually gonna get control of the situation, it wouldn't matter if there were robots. They would need to build the robots and drones they needed afterwards, repurpose something else to be those things, or they would simply do this in other ways. Like, the whole trope of, oh, we took control of the robots, and now the robots are fighting the humans, and now you have this big struggle. Like, that doesn't happen. The AIs are sufficiently powerful to take over in that way, they were sufficiently powerful to kill us, and they did it in other ways. It's just not relevant to the questions that we should be worried about. It's not in my threat model as a major problem. But it is in other people's threat models, and all of a sudden something goes wrong. If suddenly some of the robots get hijacked by some rogue AI and they start causing local problems, well, it's probably not terribly worse than the problems they would have caused in some other way without them. But they are ways that would cause people to react and then take proper precautions as a result. So, again, there's no reason not to build them, basically. It would just be unilateral disarmament. It would just predict that you're in a worse position. It would just make the future less likely to be American, Western, Democratic, et cetera, for very little gain in anything. So yeah, this is just not the hill you want to be dying on. If you don't want to build autonomous killer robots, don't build the AIs that enable you to build autonomous killer robots. That is the way you don't build autonomous killer robots. And if the autonomous killer robots cause people to say, don't build the AI, good, right? If we can pull that off, everyone agrees not to build those AIs.

Zvi Mowshowitz: (2:52:26) I'm not a big analogy guy, but tell me why this analogy doesn't hold. We have biotechnology, which is obviously got great promise for better lives for us in a very grossly similar way that AI has the promise of much better life for us. We also could weaponize the biotechnology and create bioweapons. We did unilaterally disarm there, right? We basically just said, look, maybe China's gonna go off and do their own bioweapons anyway. And for all I know, maybe they are, and there's reporting to that effect. And I don't know if it's true or not. But my understanding is The United States, whatever, the good guys, decided this is just too bad of a technology to develop, and we're just not gonna do it. And we hope that others will follow our example. But even if they don't, we still feel like we're kind of better off because we didn't go down this path because it's more likely to hurt all of us than it is to be something that we're really glad to have built. Why do you not see the autonomous killer robots in the same way?

S2: (2:53:34) So, first of all, my understanding is, essentially, though there was a biological weapons convention, I thought we all did agree not to do it. It's just that various nations have been cheating this at various degrees at various points as you do. But you don't cheat fully. You cheat a little, right? It's like you cheat an understandable, deniable amount, and that's still a lot better than openly pursuing. But fundamentally, the answer is that there is no good way to use a biological weapon. There's only terrible ways to use a biological weapon, right? If you deploy a biological weapon that's any good, you risk it turning back on you. You risk it doing much more damage than you expected. They can't be controlled. They can't be predicted.

Zvi Mowshowitz: (2:54:20) Isn't that your take on AI in general? I mean, the thing that's a little bit of stuff that is they're saying that's gonna happen.

S2: (2:54:26) But not because of the robots, because of the AIs. I'm saying that the AIs are the thing that can cause things to get out of control, that can cause the world to end up in a bad state. It's not the robots that we may or may not build. The biological weapons, well, if you ever use them, it's complete disaster. And also, we've developed a universal norm and a very, very strong distaste for anyone who ever uses biological weapons. You would turn the world against you almost immediately to the extent that you didn't destroy the world by using them. And what's the point of having them? You turned the world against you even by having them in an open fashion. Autonomous robots—

Zvi Mowshowitz: (2:55:23) There has been some talk, which I'm sure you're aware of, of the possibility of bioweapons that only target certain, shall we say, phenotypes. And I don't know how credible that is to really create. But if all of a sudden you had a path to that, it seems like—I mean, I would still be like, still don't build them.

S2: (2:55:44) No, obviously, still don't build them. You can't trust it to work the way you want it to. You don't know that they don't have their own bioweapons they would launch because you launched yours. You don't know they wouldn't launch nukes in response to your bioweapons. You don't know a number of things. Look, I prefer nobody develop bioweapons. I strongly prefer nobody develop bioweapons. One of the reasons why you don't want AIs running around uncontrolled is because if it is possible to develop bioweapons with AI assistance, that could potentially be very, very difficult to stop in any reasonable way. So you might not have a lot of choices.

Zvi Mowshowitz: (2:56:21) So does the main thing on the distinction between the bioweapons and the AI weapons, autonomous killer robots as they're known, does that come down to just the robots can't self-replicate, and so they're just fundamentally more—I mean—

S2: (2:56:36) If the robots were self-replicating, then I would be much, much more scared of the actual autonomous killer robot. That's absolutely true. That's an example of this, right? You build the nanobot that could potentially gray goo the planet, then that is something you just don't do independently of everything else. But autonomous killer robots, again, it doesn't fundamentally change the game. Drones are already the primary weapons of war in actual live fighting that's happening right now. They're already—everyone is designing their future militaries around this possibility even without AI. And you can't keep AI out of warfare. You can't voluntarily disarm in these ways. Anybody who does will just lose. Yeah, if you could—I mean, look, if you can get everybody in the world to think Kumbaya and agree that we never build any more drones or autonomous weapons of any kind, and then actually enforce that, I mean, probably good. But that's not gonna happen. And we should not pretend that it is happening. We can't get a letter of leaders signed here. And if something goes wrong, it goes wrong locally, right? It's not existential risk to have autonomous killer robots. The AI, they're not gonna turn on us by taking control of our autonomous killer robots and winning a war against humanity we would have otherwise lost. That is just such a very, very, very unlikely thing to happen.

Zvi Mowshowitz: (2:58:00) Yeah, I don't know. I mean, it doesn't feel vanishing to me. Well, I mean, this is a hard thing to really unpack. But it does strike me that there could very well be a point in time where the AI is sort of, not generally able to do whatever they want, but they might be able to hack into a million autonomous robots that we have created. And—

S2: (2:58:26) If they can hack into whatever they want, we've already lost. If they can and choose to with malicious intent, it's over. That's by that point.

Zvi Mowshowitz: (2:58:39) So is there any affordance that you would be like, don't give the robot this or don't give the AI this affordance? Because this seems like it would be pretty high up on most people's draft boards for affordances to deny the AI would be the actual lethal weapons, right?

S2: (2:58:56) I mean, I wouldn't give them control of the nukes for the illusions you're talking about. I think that's a very easy way to imagine that going wrong. But for the robots, I mean, obviously, you want to have various checks. But, you know, designing exactly what checks those are at 9:50 in the evening after a very long day is just not where I need to be. But we can't not do this. We obviously have to do this. And even if China said we won't do it if you won't do it, well, there's only two countries and there's a lot of other countries.

Zvi Mowshowitz: (2:59:30) Yeah, okay. Well, perfect transition to my last question for you, which is just an invitation. You know, I guess I still somehow see the risk that one would incur by unilaterally not pursuing things like weaponization as virtuous. I don't deny that there is risk there, but I'm kind of like, if we want to get to a different equilibrium, somebody might have to take a leap of faith. And it does feel to me like there's some virtue in putting oneself forward to do that.

S2: (3:00:01) I think you're just shooting yourself in the foot. There's no virtue in losing on purpose.

Zvi Mowshowitz: (3:00:07) Well, that is—I mean, there's a lot of assumptions baked in there, right? I mean, I'm not so sure that you couldn't get others to follow your lead, and it certainly would depend on state of evidence and whatever. But broadly, I think that there is a sort of we are not gonna do this even though we expose ourselves to some risk in not doing this that I see as virtuous because I see on the other side of it just a really bad equilibrium, and it's like, somebody's gotta take some risk. S2: (3:00:38) Then do it where it matters. Do it where it actually potentially saves us. Don't waste your big noble sacrifice on a completely irrelevant, superficial look of "there's nothing with red glowing eyes that walks around." It's such a stupid place to forfeit your strategic power. I find it so absurd.

Zvi Mowshowitz: (3:01:06) Well, here's the invitation to you. So what is virtuous to do today? What are the big virtuous moves that you could recommend to specific individuals or to the listening audience at large? Where would you spend that capital, or what other virtuous moves do you want to see more people making?

S2: (3:01:26) I mean, on the governance side, we need transparency. We need state capacity. We need state visibility into the labs. We need cybersecurity at the labs. We need our export controls to actually be strengthened and enforced. And we need to return the world to a place where we can reasonably cooperate in various ways and be reasonably prosperous and normal in other ways so that we have a good foundation to move forward and not feel obligated to push these buttons, would be the obvious things to say.

On a private level, again, capturing mundane utility, helping people live better lives, that's good. Pushing frontier of capabilities, that should be highly questionable. But obviously working on alignment, security, safety, those kinds of things, almost universally good. Preparing various policy interventions, getting them ready, trying to build networks, trying to understand these things, trying to educate the public, trying to build awareness of these issues, that stuff is good. There's no slam dunk right answers. Trying to raise the level of discourse is obviously very good. But it's rough out there. Do no harm is obviously the first thing you say in these situations. But yeah, it's rough, but there's definitely talent constraints on policy, on state capacity in various places, and on various alignment and technical work. And certainly, those are the obvious places if one wants to be especially virtuous. Well, if you can help keep an eye on the labs, you can help keep discourse higher level and accurate and elevate people who seem to be truth seeking over people who don't seem to be truth seeking, and you can make your own judgment as to who those people are and so on.

Zvi Mowshowitz: (3:03:39) What about on just a personal attitude level? You said do no harm. And I sort of feel like maybe I want to question that a little bit. Maybe I want to say take more risk, we're in a short timeline.

S2: (3:03:50) Oh, yeah. No, no, no. I meant more like don't do things that are just clearly accelerating the process without a reason. I didn't mean be risk averse. You definitely don't want to be risk averse at this time. You have to be risk loving because we need variants, we need something to go right. Need a lot of things to go right. So yeah, I think one of the biggest historical mistakes that was made by people who are concerned about this was that during the 2010s especially, but also before that, there was a lot of do no harm that was paralysis inducing, where people basically didn't do anything, where they kept things way too secret, or they were afraid to spread words about things and so on. And that turned out to keep the wrong thing a secret in ways that prevented us from making as much progress as we could have without the controlling the messages that actually caused acceleration and harm. We should have done a very different set of things. The dangerous method was actually the fact that this thing was dangerous, which caused people to pay attention to it, as opposed to the technical insights that we had, which would have been better to spread around, because that would allow people to make better progress. So, you know, definitely don't be secretive with the productive type of ideas. Right? In almost every situation, being open about and helping other people be smarter and understand things better is good. It's the weird exceptions where it's bad, and you should look out for that.

Zvi Mowshowitz: (3:05:36) Well, on that note, you are definitely living your own conception of a virtuous life by pumping out very thorough analysis on an unbelievably regular basis. So it's a great public service and a great resource, and I turn to it regularly and others definitely should too. So I appreciate all that hard work. Any final thoughts you want to leave people with, or are you ready to collapse?

S2: (3:06:02) I think I'm about ready to collapse, so let's call it.

Zvi Mowshowitz: (3:06:05) Alright. Well, I appreciate it. Heroic effort and virtuous effort. So, Zvi Mowshowitz, thank you for being part of the Cognitive Revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.