Lindy AI Founder Flo Crivello on Biden's Executive Order and AI Safety

Full Transcript

Transcript

Flo Crivello: 0:00 So I was afraid that the regulation would go something like, if you install Microsoft Office in your AI, then you have to make a report. And it's like, what? Like, it's happening again. They're doing it again. And it's like, chill. I totally get it, but this time is really different. Like, this is really something special that's happening. Not just in the markets, not just in the economy, not just in the country, in the universe. Like, there is a new form of life that's being built, and we're in new territory, and we need to be careful right now. I think if you zoom all the way out literally from the birth of the universe, the evolution of the universe has been towards greater and greater degrees of self organization of matter. You are such a leap compared to an atom, right, or compared to a bacteria that there is no reason to expect that there wouldn't be another thing above you that is as much more complex or bigger than you as you are to the bacteria. Like, there's nothing in the universe that forbids that from happening, for a being to exist that is about as big as a planet or a galaxy. But I don't expect ASI to take more than 30 years. So I expect that you and I, in our lifetime, we're going to see ASI.

Nathan Labenz: 1:06 Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz joined by my cohost, Eric Tornburg. Hello, and welcome back to the Cognitive Revolution. Today, I'm speaking with Flo Crivello, founder and CEO of Lindy, which describes itself as your AI employee, who was first on the show back in March in just our ninth episode. Flo

Flo Crivello: 1:44 and his

Nathan Labenz: 1:45 team are still building Lindy very actively, and Flo is often posting updates about their performance improvements on Twitter. But for now, the product remains in a super closely held private beta. And to be honest, even I haven't got my hands on it. If you wanna hear more about Lindy, go back and listen to episode 9. In the meantime, Flo has become increasingly outspoken, always articulately and often very insightfully on the importance of existential risk management in the context of advanced AI systems. So with the entire AI community working to make sense of the recent White House executive order and the statements from the AI Safety Summit in the UK, which notably do include some meaningful participation from China. And meanwhile, with the culture war heating up in the background with VC manifestos competing for airtime with Cruise shaming and Cruise self flagellation, I wanted to get Flo on the show to discuss the AI safety and regulation landscape from the big picture point of view. We start with some of the areas where we are both quite enthusiastically accelerationist before considering whether thinking about AI as a new life form is just another analogy or perhaps a proper literal description. And then we explain why on the defining question of our time, the potential development of a general super intelligence, Flo and I are agreed that something like the precautionary principle absolutely makes sense. As always, if you find value in the show, we'd appreciate it if you'd share it with a friend. Send this one to the techno optimist libertarian in your life who hasn't properly considered the full potential ramifications of AI. Now I hope you enjoyed this conversation with Flo Crivello. Man, it's a tough time for somebody that tries to keep up with everything going on in AI. It's gone from, in 2022, I felt like I could largely keep up and wasn't missing whole major arcs of important stories. And now I'm like, yeah, I'm totally let go of AI art generation, for example. And this policy stuff is really hard to keep up with, especially this week. Of course, it's hitting a fever pitch all at once. But, you know, I love it, so I can't really complain at all. It's just, at some point, you know, got to admit that I have to maybe narrow scope somehow or just let some things fall off. I'm just kind of wrestling with that a little bit.

Flo Crivello: 4:13 Which I think is just like a natural yeah. I mean, you

Nathan Labenz: 4:15 know, I hear you. I think it's just

Flo Crivello: 4:17 a natural part of the industry evolving. It's like imagine, you know, talking about keeping up with computers, right, in the eighties or something. It's like I'm sure at some point, it was possible to keep up with computers at large, and now it's like keeping up with tech. It's just like, okay, dude. It's like half the GDP or whatever. Right?

Nathan Labenz: 4:34 You're doing all this in your second language, right? This is I assume English is your second at least.

Flo Crivello: 4:40 I have an excuse. Yeah. Second. I'm actually getting my American citizenship. I had the interview just yesterday.

Nathan Labenz: 4:45 Wow. Congratulations. That's great. I know it's not an easy process, although maybe it's about to get streamlined. I haven't even read that part of the executive order yet, but I understand that there is kind of an accelerated path for AI expertise. Have you seen what that is?

Flo Crivello: 5:03 No. But generally, those good stuff being done in immigration. Like, they're relaxing a lot of requirements. They're closing a lot of loopholes. They're doing a lot of very good stuff.

Nathan Labenz: 5:12 Yeah. I've been thinking of just as kind of a general communication strategy, if nothing else, calling out the domains in which I am accelerationist, which are in fact many. I think you and I are pretty similar in this respect where it's like, on perhaps the singular question of the day, I am not an accelerationist, but on so many other things, I very much am an accelerationist. And, like, streamlining immigration would be one of those. You know? I would sooner sign up for the 1,000,000,000 Americans plan than kind of, you know, the build the wall plan, certainly. And I just did a right before this, I did an episode on autonomy and, you know, self driving. And that's another one where I'm like, holy moly. You know? I don't know if you have a take on this, but the recent Cruise episode, I find to be, you know, kind of bringing my internal Marc Andreessen very much to the fore where I'm like, we're gonna let one incident shut down this whole thing in California. That seems crazy enough. But then the fact that they go out and do this whole sort of performative self I mean, whether it's performative or not, maybe it's sincere, but do this whole self flagellation thing and, you know, shut the whole thing down nationwide. I'm like, can we where is our inner Travis on this, people? You know, somebody has to stand up for something here at some point.

Flo Crivello: 6:40 Totally. I agree. I think it's just the natural order of things. Right? It's like, don't know if you know that piece of history about when the automobile came about. There was this insane law that said you needed to have someone walking with a flag in front of the automobile at no more than 4 miles an hour. Right? So it's part of the process, man. It's infuriating. I hate it, but in some way and maybe it's cope, but I make peace with it. I'm like, it's part of the process. You can't really stop progress. It's going to do its thing. So it doesn't really matter anyway because the self driving cars are not really deploying at a very large scale. And so I'm like, ugh. You know, it's not a bottleneck anyway. I don't think it is.

Nathan Labenz: 7:19 I guess I have 2 reactions to that. One is, like, it feels like if they if nobody kind of fights through this moment, then there is this potential for kind of the nuclear outcome where, you know, we just kinda get stuck, it's like, sorry. You know, the standards are so insane. You've gotta be you know, we do have a little bit of a chicken and egg problem where, you know, if you had a perfect self driving car, they'd let you deploy it, but you're not gonna get to perfect unless you can deploy it. And, you know, to me, this technology is just an incredible example of where, you know, the relative risk is already pretty low. As far as I can tell, they already do seem to be as safe or marginally safer. You know, maybe as much as order of magnitude safer already depending on exactly what stats you look at. And I would just hate to see us get kind of you know, as we're kind of close to maybe some sort of tipping point threshold, whatever, to get stuck in a bad equilibrium of, you know, never get and then, you know, maybe get stuck and never get out of that chicken and egg thing would just be so frustrating. I drive a 2002 Trailblazer that I have sworn never to replace unless it's with a self driving car, and it's becoming increasingly difficult to keep this thing going. You know? So I'm like, how long do I have to wait? My other take on this is I think Tesla's actually really good. I borrowed a neighbor's. I don't know if you've done the FSD mode recently. My grandmother came up for a visit. It was fun. I actually took, you know, my 90 year old grandmother on a trip back to her home, which is a 4 hour drive there, and then I did 4 hours back all in one kind of big FSD experiment. I set up my laptop in the back, put a seat belt on my laptop. So it was recording me and recording us, you know, driving so I could look at the tape later. And I was like, man, this is really good. I had no doubt in my mind coming out of that experience that it's a better driver than other people I have been in the car with, you know, for starters. So I'm thinking through my personal life of, like, yeah. I'd rather be in the car with an FSD than this person and that person and this other person. You know? And I'd be definitely more likely to let it drive my kids than this other person. So I felt like it was really good. And then the other thing that was really striking to me was the things where it messed up I mean, there weren't many mess ups for one thing, but the few mess ups that we had. There were a couple in an 8 hour thing. It was like, if we actually had any mojo and we went around kind of cleaning up the environment, we could solve a lot of this stuff. Like, there was one that my neighbor who lent me the car said, you know, you're gonna get to this intersection right there on the way to the highway, and it's gonna miss the stop sign because there's a tree in the way. And I was like, you know, for one thing, probably people miss that too. Like, let's trim the trees. You know? And then there's another one where you're getting off the highway, and there's a stop sign that's kind of ambiguous. Like, it's meant for the people on the service road, but it appears to be facing you as you're coming off the highway. And so the car saw that and stopped there. And that was probably the most dangerous thing that it did was, you know, stopping where people, you know, coming up the off ramp do not want you or expect you to be stopped there. But that's another one where you could just go put up a little blinder, you know, to just very easily solve that problem, and I imagine people must have that problem too. And we just have no will, you know, when it comes to that. And, again, I feel like I'm turning into Marc Andreessen the more I think about self driving over the last few days.

Flo Crivello: 10:54 No. I'm with you on that.

Nathan Labenz: 10:55 So where else are you accelerationist that may not be obvious as we kind of think about this, you know, this kind of AI safety and regulation moment that we're in?

Flo Crivello: 11:08 You know, honestly, pretty much everywhere, man. Like, I'm a libertarian. I used to work at Uber where I saw regulatory capture and I saw cartels, and I do believe, you know, at the deepest level that cartels and regulatory capture and generally, I think it's Mancur Olson who calls them extractive institutions, who are just in the business of they don't wanna grow the pie. They just wanna grab a little bit more of the pie for themselves. Even if it actually shrinks the pie, they don't care as long as they get a bigger chunk. And I think that's the world that's just rotten with thousands and thousands of these institutions where whether private or whether unions or governmental, it doesn't matter. We just have so many of these cartels floating around, and it's killing everything. Right? It's a tragedy. And I totally understand how folks like Marc Andreessen would be they have built such a deep and justified hatred and reaction for this nonsense that is destroying everything that they immediately the pattern recognition immediately triggers when they see what's happening with AI. They're like, it's happening again. They're doing it again. It's like, chill. I totally get it, but this time is really different. Like, this is really something special that's happening. Not just in the markets, not just in the economy, not just in the country, in the universe. Like, there is a new form of life that's being built, and we're in new territory, and we need to be careful right now. Right? And so that's where I'm coming from is I totally see that point of view, and I'm like, regulation, for sure, there's going to be cartels. For sure, we're gonna screw up 90% of it. Politics is gonna get messy and vested interests are gonna get into play. And it's all worth it because what may very well be on the line, it sounds alarmist, but I'm sorry that we need to say the words, maybe literally human extinction. Right? And this is not some tinfoil hat theory. There's more and more experts that are coming around and saying that. It's actually funny. Marc Andreessen, if you dig it up, I'm sure you could find it. I think it was an interview from him. I wanna say between 2017 and 2020. That doesn't help there. So many of those, but I think he said something like, at the time, he was actually appealing to an argument about authority. He was like, look, he was saying the same things he's saying today, progress is good, it's just a tool, and by the way, the experts say there's nothing to worry about. I don't know, you guys don't know anything about AI. I don't know anything about AI. They do, and they're telling us there's nothing to worry about. Well, that's not true anymore. The experts are telling us there is something to worry about. And now it's just like, oh, arbitrary regulatory capture. No. No. It's not regulatory capture. Like, OpenAI was founded on that premise from day 1. So if it was regulatory capture, there's one hell of a plan. It's like, my god. We're gonna create this industry and we're gonna start regulatory capturing right now. Right? It's like, that makes no sense. It was literally the plan from day 1. Yeah. That's where I'm coming from. I'm largely in the e/acc camp. I am in team technology, team enterprise, team anti regulation, but here something very special and potentially very dangerous is happening.

Flo Crivello: 11:08 You know, honestly, pretty much everywhere, man. I'm a libertarian. I used to work at Uber where I saw regulatory capture and I saw cartels, and I do believe, you know, at the deepest level that cartels and regulatory capture and generally, I think it's Mancur Olson who calls them extractive institutions, who are just in the business of they don't want to grow the pie. They just want to grab a little bit more of the pie for themselves. Even if it actually shrinks the pie, they don't care as much as they get a bigger chunk. And I think the world is just rotten with thousands and thousands of these institutions where whether private or whether unions or governmental, it doesn't matter. We just have so many of these cartels floating around, and it's killing everything. It's a tragedy. And I totally understand how folks like Marc Andreessen would be, they have built such a deep and justified hatred and reaction for this nonsense that is destroying everything that they immediately, the pattern recognition immediately triggers when they see what's happening with AI. They're like, "It's happening again. They're doing it again." It's like, chill. I totally get it, but this time is really different. This is really something special that's happening. Not just in the markets, not just in the economy, not just in the country, in the universe. There is a new form of life that's being built, and we're like in new territory, and we need to be careful right now. And so that's where I'm coming from. I totally see that point of view, and I'm like, regulation, for sure, there's going to be cartels. For sure, we're going to screw up 90% of it. Politics is going to get messy and vested interests are going to get into play. And it's all worth it because what may very well be on the line, it sounds alarmist, but I'm sorry that we need to say the words, maybe literally human extinction. And this is not some tinfoil hat theory. There's more and more experts that are coming around and saying that. It's actually funny. Marc Andreessen, if you dig it up, I'm sure you could find it. I think it was an interview from him. I want to say between 2017 and 2020. There's so many of those, but I think he said something like, at the time, he was actually appealing to an argument from authority. He was like, "Look, progress is good, it's just a tool, and by the way, the experts say there's nothing to worry about. You guys don't know anything about AI. I don't know anything about AI. They do, and they're telling us there's nothing to worry about." That isn't true anymore. The experts are telling us there is something to worry about. And now it's just like, "Oh, arbitrary regulatory capture." No. It's not regulatory capture. OpenAI was founded on that premise from day one. So if it was regulatory capture, there's like one hell of a plan. It's like, "My god. We're going to create this industry and we're going to start regulatory capturing right now." That makes no sense. It was literally the plan from day one. Yeah. That's where I'm coming from. I'm largely in the e/acc camp. I am in team technology, team enterprise, team anti-regulation, but here something very special and potentially very dangerous is happening.

Nathan Labenz: 14:29 So let's go back to your use of the phrase a new form of life. I, as you may recall, am very anti-analogy as a way to understand AI, because I think it's so often misleading. And I often kind of say AI, artificial intelligence, alien intelligence. It may be tempting for people to kind of hear, or not tempting, but it may be sort of natural for people to hear you say a new form of life and understand that as an analogy. But do you mean it as an analogy? Or, you know, I guess we might start to think about, like, is that actually just literally true, and what conditions would need to exist for it to be literally true? And you might think about things like, can AI systems, like, reproduce themselves? You know? Are they, like, subject to the laws of evolution? But, like, for starters, how literal do you mean it when you say that there's this, like, new form of life in AI?

Flo Crivello: 15:30 I mean it pretty literally. I think if you zoom all the way out literally from the birth of the universe, the evolution of the universe has been towards greater and greater degrees of self-organization of matter. And there's actually a case to be made that this is just a natural consequence of the second law of thermodynamics. There's this amazing book that e/acc people love to quote, which...

Nathan Labenz: 15:49 I was going to say, you're sounding very e/acc all of a sudden.

Flo Crivello: 15:51 It's a good point. It's called "Into the Cool: Energy Flow, Thermodynamics, and Life." And so if you look at the big bang, you know, like, a few fractions of a second after the big bang, it was just subatomic particles, and then they ganged up together and formed atoms. And then the stage after that was the atoms ganged up together and formed molecules. And then the stage after that, the molecules became bigger and bigger because they formed into stars and exploded and caused all sorts of reactions. And so a few generations of stars later, we have, like, pretty big molecules and pretty heavy ones. And then these molecules form into sort of, like, proteins and RNA and forms of proto-life. We don't totally understand, there's a chain here that we don't totally understand, but there's a form of proto-life that formed and then life. And so you can think of, like, at first it was just DNA or actually it was RNA, DNA, nucleus of a cell, cell, mitochondria came into that, and then, okay, cool, we have a cell, and then the cells have started ganging up together and now we have multicellular organisms. And then we have brains at some point, like, that's like a big leap, but we have brains, like, on the great march towards greater and greater degrees of self-organization. And at some point, we have us, which with a little bit of hubris perhaps, I am considering the apex of that thing for now. It just seems crazy to me that everybody is saying like, one, this is totally normal. Like, we're normal. It's like, dude, this is quintillions of atoms that are organized in this really super coherent fashion and that are pursuing a goal in the universe. Like, what's happening right now on Earth is wild to begin with. Right? So people are already thinking that this is normal and that's what it is and that this march is going to stop at them. Right? And they're like, "Well, maybe we're going to get slightly smarter or maybe we're going to get augmented." And I'm like, you are such a leap compared to an atom, right, or compared to a bacteria that, like, there is no reason to expect that there wouldn't be another thing above you that is as much more complex or bigger than you as you are to the bacteria. Like, there's nothing in the universe that forbids that from happening, from a being to exist that is about as big as a planet or a galaxy. Like, there's nothing forbidding that in the universe from happening. And for the first time now, if you squint, we can sort of see how that happens. And silicon-based intelligence certainly seems to have a lot of strengths up its sleeve versus carbon-based intelligence. And so, no, I actually sort of mean that pretty literally. It is sort of in line with the march of the universe, and this is the next step perhaps. It's significant. And so I am hopeful that we can manage this transition without us being destroyed. That's what I want to happen.

Nathan Labenz: 18:32 Hey. We'll continue our interview in a moment after a word from our sponsors. So when you say manage the transition, does that imply an inevitability to advanced AI? I guess a lot of people out there would say, "Hey. Let's pause it, slow the whole thing down," and then you get kind of the response from, like, an OpenAI where they're sort of saying, "Yeah. We do take these risks very seriously, and we want to do everything we can to avoid them. But we can't really pause or we don't think that would be wise because then the compute overhang is just going to grow, and then things might even be more sudden and disruptive in the future." Where are you on kind of the inevitability of this, you know, increasingly capable AI coming online?

Flo Crivello: 19:20 I don't think it's totally inevitable. I am generally a huge believer in human agency. I think we can do pretty much anything we set our minds to. I see a contradiction, by the way, in the arguments that, like, on the one hand, it's inevitable, don't try to stop it. On the other hand, "Oh my god. If you do this, I'm going to stop..." It's like, you got to decide here. So, unfortunately, it's not necessarily inevitable. I am actually worried. As much as the next guy, I agree, there is a risk that we overregulate and miss out on the upside, and the upside is significant. And, you know, if you look like during the Middle Ages, we successfully as a civilization stopped progress. And in a lot of countries, if you look at North Korea, they did it. They successfully stopped progress, so you can stop progress. Progress is not inevitable, and arguably, it is actually quite fragile. So no, I don't think it's inevitable, and I'm hopeful that we can, again, I want us to get to the upside without experiencing the downside.

Nathan Labenz: 20:16 You know, I mean, the North Korea example is an interesting one. You know, if I was going to kind of dig in there a little bit more, might say, okay. I can understand how, like, if things go totally off track, then we could maybe enter into, like, a low or no or even negative progress trajectory. You know? If there were a nuclear war, you know, then, like, we may not, you know, come back from that for a long time. Or if, whatever, asteroid hit the earth or, you know, pandemic wiped out 99 percent. Like, there's extreme scenarios where it's pretty intuitive for me to imagine how progress might stop or, you know, just be whatever greatly reversed or whatever. If I'm imagining kind of a continuation of where we are, then it's harder for me to imagine how we don't kind of keep on this track. You know? Because it just seems like everything is, we're in this, I would call it, you know, I don't know if it's going to be a long-term exponential, but if not, like, we seem to be entering a steep part of an S-curve where hardware, you know, is coming online by the order of magnitude. And at the same time, like, algorithmic improvements are, like, taking out a lot of the compute requirements. And we're just, like, seeing all these existence proofs of what's possible and all sorts of little clever things and scaffolding along the lines of some of the stuff that you're building is getting better and better. Is there a way that we can, do you think it is realistic to think we could kind of meaningfully pause or even, like, stop without, like, a total derailment of civilization?

Flo Crivello: 21:52 The derailment of civilization thing, like, you could imagine the most extreme scenario, which I am not proposing. But you could imagine the most extreme scenario, which is no more Moore's Law. You do not exponentially improve your semiconductor systems anymore. That's crazy. Right? But it wouldn't derail civilization. Right? Civilization is not predicated upon that. Like, we would do just fine with the chips we've got today. And if anything, I think we have a lot of overhang from the chips we have today. Huge, huge, huge overhang. Right? So I actually think it is possible to do that if we wanted to, and I don't think that even this, which I think is the most extreme scenario, would actually derail civilization. We are actually lucky in that there are a few choke points in the industry. Actually, more than a few. There is ASML, there's TSMC, there's NVIDIA, like all of those three are individually choke points. Like, a regulator could at any point grab one of them and be like, "No more. You just stop." Or, you add this chip into all of your GPUs moving forward, so we have a kill switch. At the very least, we have that. So if shit really hits the fan, we have a mechanism in place that, like, shuts down the compute on this. Right? Now that would be disruptive, but potentially less disruptive than a rogue ASI. So, no, I actually think it is very much possible. These things are all on the table, and I don't think they would be all that disruptive.

Nathan Labenz: 23:16 So maybe that's a good transition to kind of where we are right now. Right? We just had this executive order put out this week, and I think everybody's still kind of absorbing the 100 plus pages and trying to figure out exactly what it means. What's your high-level reaction to it? And then I'll get into some of the specifics.

Flo Crivello: 23:34 First of all, it's an executive order for now. It is not law. It's very early. Overall, I am pleasantly surprised, not by the specifics, but by the fact that we're reacting quickly, by the fact that the measures that are proposed are not insane. Like, I was afraid of, like, there's a really good case to be made. It's like, "Look. We have a gerontocracy in place. Now a bunch of 70, 80-year-olds who don't know anything. When they were born, there was no mobile phone." Can't really blame them for not really understanding anything. And so I was afraid that the regulation would go something like, "If you install Microsoft Office in your AI, then you have to make a report." And so what? Like, the regulation actually sort of makes sense. It's talking about FLOPs. It's talking about amount of compute for training. So I think it's a step in the right direction. I'm actually happy about what's happening with this executive order. Now the specifics, look, the problem is that it's almost impossible to regulate AI in a way that doesn't have any loophole. So they are regulating it according to a number of FLOPs and that's okay, but at the end of the day, then you get stuck into, "Okay, what happens when you have algorithmic improvements? What happens when you do RL instead of fine-tuning?" And, like, there's just a lot of different loopholes that researchers are going to find. And so I think overall, it's an encouraging first step.

Nathan Labenz: 24:55 It's funny. I've, you know, there have been proposals around even a FLOP threshold that would drop progressively over time in kind of anticipation of the algorithmic improvements. That's a, you know, even a more probably challenging one to put out into the world, especially given, you know, people are not, in general, great at extrapolating technology trends or, you know, don't want to, don't want to accept regulation in advance of stuff actually being invented. You know, so we've got this FLOP threshold thing where, basically, as I understand it so far, it's like, if you're going to do something this big, you have to, like, tell the government that you're going to do it, and you have to bring your test results to the government. I would agree with you. That seems like a pretty good start. And also, the threshold seems like pretty reasonably chosen at 10 to the 26. Any, you know, kind of refinements on that or quibbles that you would put forward that you think, like, you know, maybe the next evolution of this should take into account? Nathan Labenz: 24:55 It's funny. There have been proposals around even a flop threshold that would drop progressively over time in kind of anticipation of the algorithmic improvements. That's a probably even more challenging one to put out into the world, especially given, you know, people are not, in general, great at extrapolating technology trends or don't want to accept regulation in advance of stuff actually being invented. So we've got this flop threshold thing where, basically, as I understand it so far, it's like, if you're going to do something this big, you have to tell the government that you're going to do it, and you have to bring your test results to the government. I would agree with you. That seems like a pretty good start. And the threshold seems like pretty reasonably chosen at 10 to the 26. Any kind of refinements on that or quibbles that you would put forward that you think, you know, maybe the next evolution of this should take into account?

Flo Crivello: 25:57 I think ultimately, well, tiptoeing around is the issue, but ultimately, we need to come to an actual technical blanket solution. We will not solve ASI alignment by asking for reports from AI companies. That's not how it's going to happen. So again, I think it's a step in the right direction. I'm happy we're taking action. I'm happy the action is not totally nonsensical. But at the end of the day, we're going to have to talk about the kill switch. Right? The proposal I just made is one that I see more and more talked about, and that's the one I would feel best about. You've got to put this chip into your H100s and the government, there's a centralized entity that can shut down all GPUs all at once. And by the way, it wouldn't necessarily shut down every computer because your laptop doesn't have an H100. Your iPhone doesn't have an H100. That's fine. Over the long term, it makes it so that your laptop and your phone actually end up with an H100, but at least that buys us a few years to make progress on AI safety and alignment. Ideally, we would then automate, just like reportedly the Russians did during the Cold War, we would automate. We would set up some detection systems to, God knows how we would do that. But, hey, there's an ASI going rogue. The world is really changing rapidly. Assuming it's not too late, which it may be, because at that point, God knows, but basically, that would give us the best weapon against the ASI. We would have a gun against the ASI's head and, boom, kill all the GPUs. You cannot operate anymore. God knows how effective that would be because at that point, all bets are off. If you have an ASI, God knows what it does and how it connects itself, but that would be what I would feel best about.

Nathan Labenz: 27:41 Do you have any sense for how that would be implemented technically? It seems like you would almost want it to be something that you could kind of broadcast. You almost want a receiver on chip that would react to a particular broadcast signal and just kind of, because you would not want to have an elaborate chain of command or, you know, relying on the dude who happens to be on the night shift at the individual data centers to go through and pull some lever. Right? So do you know of anybody who's done kind of advanced thinking on that? That stuff is like, you hear a lot of these kill switch things, but in terms of how that actually happens so that it's not dependent on, you know, a lot of people coming through in a key moment, I haven't heard too much, to be honest.

Flo Crivello: 28:29 No. I haven't seen too much research done on that. But, you know, I think the technical challenge, there's nothing in principle that makes the technical challenge insolvable. We already have a chip that can be broadcasted to for a dollar from space. The GPS chip does, a lot of chips, and it costs, it's just like you have one on your phone. And so why not put the GPS-like chip? Maybe we could literally piggyback the GPS protocol. I don't know. But why not put a chip like that in every GPU? Again, if you have an ASI, God knows, maybe it hacks the chips before you get a chance. You know, it hacks the satellites that broadcast the thing. I have no idea. But, again, I think pointing in this direction is where I would like things to go. Into the limit, I think, basically, and that's the most extreme version of this proposal, the Yudkowsky airstrike proposal. That's like, you cannot accumulate billions and billions of dollars of H100s and deal with this thing, else we will go up to airstriking you. That's the most extreme version of this, but that actually, I think, is directionally correct. We, this is going to be the most powerful force in human history, maybe even in the universe. You cannot accumulate that stuff any more than you can accumulate enriched plutonium. Right? We've got a 4-bit stack that's the lowest level possible. And so that level cannot be the application layer, because the application layer is just too diffused. There's like 1,000 startups everywhere, and any kid in their garage can build one. It's got to be a choke point, and the choke point today is the silicon.

Nathan Labenz: 30:03 Yeah. Let's unpack that a little bit more because I think that has been an interesting debate recently. You'll hear this kind of call for, let's not regulate model development, let's regulate applications. And then we can kind of have medical regulation for the medical and everything can be more appropriate and fit for purpose. And maybe there's something to be said for that. But, yeah, I mean, if you're really worried about tail risk, it's probably not going to be sort of mega medical device style regulation of, you know, diagnostic models or whatever that is going to keep things under control. So, yeah, maybe you could even do a better job of steel manning the case for the application level regulation. But, I guess, you know, give your account of why that's not viable in a little bit more detail. Hey, we'll continue our interview in a moment after a word from our sponsors.

Flo Crivello: 31:05 The problem is that as the fork in this analogy becomes more and more powerful, the argument loses more and more of its defense because ultimately, it's just a risk-benefit analysis. Right? And so the risk becomes greater and greater as the artifact becomes more and more powerful. So more powerful than the fork, an AR-15. And so, you know, the opinions vary about that. But look, at this point, if you look at the data, you actually save lives by heavily regulating the sale of AR-15s. You can't just be like, oh, sell them to everyone and just prohibit people from shooting each other with them. It's like, it's an AR-15. What do you expect people to do? Right? Now in the most extreme scenario, enriched uranium. You can't be like, you can buy all the enriched uranium you want. You don't even need to fill out a form, which, by the way, that is all the executive order says right now. At least fill out a form. Can you please at least tell us what you're up to? So, hey, you can build all the enriched uranium you want. Just don't bomb us with it, please. We wrote it in this piece of paper. You can do it. Oh, no. That's not how it works. So that is why I think it's important to regulate the silicon layer.

Nathan Labenz: 32:12 Do you have an intuition for sort of how likely things are to get crazy at kind of either various timescales or potentially various compute thresholds? I was realizing I did an episode with Jaan Tallinn a couple months back, just in the wake of the GPT-4 deployment. And he said, we dodged a bullet with GPT-4 or something like that. In his mind, we didn't know if, even at the GPT-4 scale, that might have already been, you know, no real principled reason to believe with any, like, super high confidence that the GPT-4 scale was not going to cross some, you know, critical threshold or whatever. I guess I don't really have a great sense for this. I just kind of feel like, and this is purely, like, gut level intuition that, yeah, we could probably do GPT-5, and it'll probably be fine. And then kind of beyond that, I'm like, I have no idea. Do you have anything more specific that you are working with in terms of a framework of, when you hear, for example, Mustafa from Inflection say, Oh, yeah, we're definitely going to train orders of magnitude bigger than GPT-4 over the next couple of years. Are you like, well, as long as you stay to 2 to 3 orders of magnitude more, we'll be okay? Or is it like, I just have no, we're just flying so blind, but I wonder if maybe you're flying slightly less blind than I am.

Flo Crivello: 33:35 I am of the opinion that GPT-4 is the most critical component for AGI, and that the gap from GPT-4 to proper AGI is not research, it's engineering. It sits outside the model. So I think we have a capabilities overhang here that can turn GPT-4, as it is today, into AGI, into proper AGI. I think generally that's the case for any technology. If you look, for example, at Bitcoin, what changed from a technological standpoint that allowed Bitcoin to happen? It was the same technology we'd had for a while, and yet Bitcoin took a while to happen. So there was this overhang, and Bitcoin, whatever your opinion about crypto, changed a lot of things. Right? I think there's this huge overhang with GPT-4. I think we basically have the reasoning module of AGI. I don't know if you saw this paper that found literally just asking it, hey, take a deep breath and take a step back. Just take a step back apparently also makes a huge difference. So I think there's a lot of tricks like that that will make a difference, and also the sort of cognitive architectural layers around the GPT-4, I think, can bring it to AGI. That is also why, you asked me about what sort of regulation I wish was put into place, we need to stop open sourcing these models. We don't know what kind of overhang exists out there. I don't think Llama 2 is there, but like I said, I think GPT-4 is there. So Llama 3, if it's GPT-4 level, boom, it's too late. The weights are out there. Okay. Now you can, maybe you can bootstrap from there. So we need to stop open sourcing these things. I expect my timelines for proper AGI to emerge is 2 to 8 years. I think there's a more than even chance of AGI emerging in 2 to 8 years. I think the base scenario is things are going to go well. Just for the record, I don't think there's, like, a 99 percent chance of doom. But even if it's 10 percent, I think it's worth being very, very worried about.

Nathan Labenz: 35:29 That's enough for me. That's enough.

Flo Crivello: 35:31 For me, 10 percent of all of us dying, let's talk about it, please. So 2 to 8 years, 50 percent chance of AGI. I think it probably will go well except for, you know, civilizational disruption. There's going to be stuff, there's going to be crazy shit happening, but 2 to 8 years, and after that all bets are off. I have no idea what the bootstrapping to ASI looks like, but I don't expect ASI to take more than 30 years. So I expect that you and I, in our lifetimes, we're going to see ASI. So that's a pretty striking claim.

Nathan Labenz: 36:02 I think it probably puts you in a pretty small minority, and I don't think I'm really there with you when you say that you think GPT-4 kind of already contains the kind of necessary core element for an AGI. So I'd like to understand that a little bit better. I mean, you have a lot of people who will say, look, it can't play tic-tac-toe. I think on some level, those kind of, oh, look at these simple failure objections are kind of lame and sort of miss the point because of all things it obviously can do. But I do, you know, if I'm thinking like, does this system seem like it has this kind of sufficiently well developed world model? Or, you know, I'm not even sure exactly how you're conceiving of the core thing. But, you know, for a question like that, I would say those failures maybe are kind of illuminating. On the other hand, I'm sure you've seen this Eureka paper out of NVIDIA recently where they use GPT-4 as a superhuman reward model author to teach robot hands to do stuff. And I thought that one was pretty striking because as far as I know, and I actually use the term eureka moment, I've many times said, we don't see yet eureka moments coming from highly general systems. You know, we see eureka moments from an AlphaGo. We haven't really seen eureka moments from a GPT-4 until maybe this. This seems like maybe one of the first things where it's like, wow, GPT-4 at a task that requires a lot of expertise, that is designing reward functions for robot learning, robot reinforcement learning, GPT-4 is meaningfully outperforming human experts. And so I think it's very appropriate that they call it Eureka. What do you think is the core thing? You know, is it this ability to have eureka moments? Is it something else? Why do you feel like it's there? And does it not trouble you that it can't play tic-tac-toe? Nathan Labenz: 36:02 I think it probably puts you in a pretty small minority, and I don't think I'm really there with you when you say that you think GPT-4 kind of already contains the kind of necessary core element for an AGI. So I'd like to understand that a little bit better. I mean, you have a lot of people who will say, look, it can't play tic tac toe. I think on some level, those kind of, oh, look at these simple failure objections are kind of lame and sort of miss the point because of all the things it obviously can do. But I do, you know, if I'm thinking like, does this system seem like it has this kind of sufficiently well developed world model? Or, you know, I'm not even sure exactly how you're conceiving of the core thing. But, you know, for a question like that, I would say those failures maybe are kind of illuminating. On the other hand, I'm sure you've seen this Eureka paper out of NVIDIA recently where they use GPT-4 as a superhuman reward model author to teach robot hands to do stuff. And I thought that one was pretty striking because as far as I know, and I actually use the term eureka moment, I've many times said, we don't see yet eureka moments coming from highly general systems. You know, we see eureka moments from, like, an AlphaGo. We haven't really seen eureka moments from GPT-4 until maybe this. This seems like maybe one of the first things where it's like, wow. GPT-4 at a task that requires a lot of expertise, that is designing reward functions for robot learning, robot reinforcement learning, GPT-4 is meaningfully outperforming human experts. And so I think it's very appropriate that they call it eureka. What do you think is the core thing? You know, is it this ability to have eureka moments? Is it something else? Why do you feel like it's there? And does it not trouble you that it can't play tic tac toe?

Flo Crivello: 38:02 For the sake of this conversation, I'm going to define AGI as a seed AI, an AI that can recursively self improve. That's a much more narrow definition of AGI than most people use, but that's actually what I care about. So, like, can we enter this recursive loop of self improvement that leads us to ASI? In order to get there, you don't need to play tic tac toe. You need to be a good enough, and the word good enough here is important, a good enough either software engineer or chip designer or AI and ML researcher. One of these things. So something that can get you to bootstrap. And so good enough does not mean better than the best human. It doesn't even mean better than the average human. It just means good enough that you can make a difference, a positive difference in your own ability to get better. Right? So if you enter that loop of self improvement, then mathematically, it's all. And, yeah, when I see the NVIDIA paper, I see that. When I see our own experience with the model, so today, we are using Lindy to write her own integrations, and Lindy is writing more and more of her own code. I see that. Even as it pertains to AI researchers and ML researchers, my hypothesis is that OpenAI is using GPT-4 more and more internally to perform AI research. My, not hypothesis, the fact is that NVIDIA is releasing papers that's like, well, not only can we use it for AI research through this Eureka paper, but we can also use it for chip design. It works super well. We train an AI model that does chip design super well. Right? So we are starting to see the glimpses of that kind of recursive loop of self improvement. Basically, the world model question that I kind of want to sidestep because I feel like at this point, the debate becomes silly for people who argue that it's GPT but doesn't have a world model. What matters is, is it good enough? And so even if it just overfit its training set, even if it's just predicting the next token and not actually understanding anything, I actually really do believe it understands a lot. But even if it's not, you can imagine there's, like, this many dimensional space with a ton of data points in there, and it's good by interpolating between the data points, and it needs much more data points to understand anything than a human. And so there's that envelope in that space where the data points are dense enough that it can perform. Right? And so that's called the convex hull. Okay? And then there's data points outside that convex hull, and it does really poorly outside the convex hull, much more poorly than humans. Convex hull requires a lot more density than humans to exist. Right? There's multiple questions, which are, one, all these data points inside the convex hull is the sum of all human knowledge. It's GPT-4 today knows more than you. Right? I know that it can reason better than you. That's the expanding the convex hull thing, but it knows more than you inside that convex hull. Right? And so inside that convex hull, an AI researcher that's read every paper ever, not just in AI, but in math and biology, every paper ever and the entirety of the Internet, is it better than a human AI researcher? I think the answer is yes, right? Is yes. Even if it's not better, does the outside of that potential? And this is my point about the capabilities overhang. Can we get this AI model to, through prompting, through cognitive architecture, to do better outside this context? And we're seeing that all the time. We're seeing people come out about like, hey, we have found an automatic way to rewrite a prompt that makes it a lot better. We have found a way that people that came out a few days ago that's like, if you ask the model to take a step back and to rephrase the problem you're giving it in terms of a universal problem, it performs a lot better. And that makes total sense because the specifics of the problem, it's probably not seen that specific problem in its dataset. But if you ask it to reframe it, it's basically translating the problem into a form in which it's comfortable with. And so we're actually getting it to grow its convex hull like that. That's my take is, I think the convex hull is good enough to get to that good enough point, and I think we can grow that convex hull. And so I think that basically, if GPT-4 is not a seed AI, for sure GPT-5 is one.

Nathan Labenz: 42:09 Yeah. It's an interesting framing. I found your analysis there pretty compelling. The idea that, you know, given what we have seen from, like, a eureka, you know, with this robot training or, there was another interesting one recently, I think it was out of Microsoft. I covered this in one of the research rundown episodes on recursive or iterative improvement on a software improver. So they basically take a real simple software improver that can improve a piece of software, and then they feed that software improver to itself and just run that on itself over and over again. And it kind of tops out because it doesn't, you know, in this framework, it doesn't have access to, like, tinkering with, you know, possible methods for training itself, but it makes significant improvement and gets us some pretty advanced algorithms where it starts to do, like, genetic search and, you know, a variety of things where I'm like, I don't even really know what that is. You know, like, simulated annealing algorithm. Like, what? You know? But it comes up with that and, you know, uses that to improve the improver. And, you know, this is all measured by how effectively it can do the downstream task. It does seem like it's not a huge stretch to say that, you know, could you take the architecture of GPT-4 and start to do, you know, parameter sweeps and start to, you know, mutate the architecture itself. It seems like it probably can do that, and I would agree. You know, it probably does, yeah. Or certainly, just based on what I do, you know, with GPT-4 for coding, I would have to imagine that it is in heavy use as they're, you know, performing all that kind of exploratory work, you know, within OpenAI.

Flo Crivello: 43:55 And so yeah. And I think to your point, we are seeing enough of these signs of life across the board in a lot of different areas, a lot of institutions are like, a little bit of recursive improvement here, a little bit here, a little bit here. It's not very hard to imagine it getting to escape velocity, to imagine it going super critical and passing a threshold where it's like, now boom. It can really take off. So and I've actually heard multiple people from OpenAI say that they believe, and I agree with the conclusion. And they actually told me that before I agreed with them. They told me that at the very beginning of the year, so before GPT-4 was widely available. And they told me, you know, we, I think we'll, you know, we have AGI, and we're in a slow takeoff. And I told them, like, yes. Right.

Nathan Labenz: 44:37 So you will not, AGI is getting achieved internally.

Flo Crivello: 44:40 Well, they didn't say, sorry. That didn't, like, they basically were talking about GPT-4. Right? Or, like, we, I think, and I am not representing that this is the universal position of OpenAI, but I've heard multiple people from OpenAI and other labs tell me that. We have AGI, and it will be a slow takeoff.

Nathan Labenz: 44:55 So given that, okay, we've got this compute threshold. We maybe need a kill switch. Now I'm getting, you know, we started this conversation with me with my, you know, EAC side coming out and, you know, being like, why can't we get my self driving car on the road and tolerate, you know, some reasonable amount of risk to do that? Now my other side is coming out, and I'm like, okay. What else might we do? Right? We've got the AI safety summit going on right now in the UK. I thought it was cool to see today that there's some kind of joint statements between Chinese and Western academics and, you know, thought leaders in the space where they're kind of saying, yeah. We need to work together on this. Like, human extinction is something that we think could happen if we're not careful. Do you have a point of view on kind of collaborating with China or coordinating with China? I mean, that's a tough question, obviously. Nobody really knows China, I don't think, super well. What do you think about that? I mean, are we naive to hope? I guess, I kind of feel like what else are we going to do except give it a shot.

Flo Crivello: 46:02 Yeah. 100%. And there is ample precedent. You know, everybody is always talking about these coordination problems. It's like they've taken, like, the one on one course of game theory, and they're like, look. We can't coordinate. Well, like, if you take game theory one or two, it's like solutions to the coordination problem. Right? And so the solutions to the coordination problem is few players in a very iterated game. And that is the game right now. There's very few players, and they're all in a very iterated game. They're not the best buddies, but they are actually able to agree on a lot of things. And so we can coordinate with China. And again, to your point, like, what choice do we have anyway? Right? And even if we do not coordinate with them, again, there's enough choke holds, enough of which are American. Right? Nvidia is an American company last time I checked. And so there's enough choke holds that we could actually do very much, not give them a choice like, hey. Your GPUs now have the chip right here. You know? And so whether you like it or not, we have a satellite up here, and we can kill all the GPUs, you know, out there. And, you know, that wouldn't be, we could even just downright forbid GPUs, by the way, to be sold in China. Like, we've done stuff like that before. So, no, I think coordination is definitely possible, and I actually think it's going to happen. I'm actually really very much encouraged by, we're winning. Like, I think the safety side is making really good progress. There is rising public awareness. I think Jeff Hinton is doing an amazing work here. The regulation is coming. It's mostly sensical. There's this sort of progress that's happening across the board. AI labs are investing more and more in safety and alignment. Even from a technical standpoint, the work that Anthropic is doing, I think, absolutely brilliant. So we're making really good progress across the board here. I don't want to represent that it's over and we will do.

Nathan Labenz: 47:47 Yeah. I totally agree. I would say my kind of high level narrative on this recently has been it feels like we're at the beginning of chapter 2 of the overall AI story, and chapter 1 was largely, you know, characterized by a lot of speculation about what might happen. And amazingly, kind of at the end of chapter 1, beginning of chapter 2, not all, but, like, a large share of the key players seem to be really serious minded and, you know, well aware of the risks. And it's easy to imagine for me a very different scenario where everybody, you know, all the leading developers are, like, highly dismissive of the potential problems. But it's hard for me to imagine a scenario that would be, like, all that much better than, you know, the current dynamic. So I do feel, you know, like, overall, you know, pretty lucky or pretty grateful that things are shaping up, at least to give us a good chance to try to get a handle on all this sort of stuff. One last question. This is super philosophical. I know you've got to go. But how much depends in your mind on whether or not, let's say, silicon based intelligence or AI systems or whatever might become or maybe already are or, you know, I'm not sure how we would ever tell the kinds of things that have subjective experience. You know, does it matter to you if it feels like something to be GPT-4?

Flo Crivello: 49:13 Have you heard of the word mu? I think it's in Zen philosophy in Buddhism, there's this story that's like someone asked someone else like, hey, does, can a dog have the essence of a Buddha? If the Buddha is everywhere and in every being, can a dog have the essence of a Buddha? And the answer to that is mu. And mu means neither yes or no. It's a way to unask the question. It's a way to reject the premise of the question. And basically, in this sense, it means that there is no such thing as the essence of the Buddha. Right? It's like the same question as like, hey. What happened before the universe existed? There was no before because the birth of the universe was also the birth of time. So the will to be full only makes sense in the context of the universe. And so, anyway, that's all of my instinct whenever I ask a question, whenever someone asks me a question about subjective experience and consciousness, I'm like, mu, doesn't exist, it doesn't matter, immeasurable, it's not a scientific thing, and so mu.

Nathan Labenz: 50:14 Alrighty, well, some questions bound to remain unanswered, and I appreciate your time today. This is always super lively. Next time, I want to get the Lindy update, and at some point, I want to get access. But for now, I'll just say, Flo Crivello, thank you for being part of the Cognitive Revolution.

Flo Crivello: 50:32 Thanks, Nathan.

Nathan Labenz: 50:33 It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.