Full Transcript

Nathan Labenz: (0:00) What if you didn't have to work as much? What if you didn't have to worry about marketable skills after a few years out? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that. We're just entering the steep part of the curve in robotics, and probably we'll have, like, a lot of humanoid robots to actually do physical labor in the not too distant future. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're gonna relate to these things. You know? If if they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. No matter how weird your alignment idea is, I think it is worth kind of pursuing. No matter how weird your kind of thoughts are about where the future might be going, I would say they're probably worth entertaining. Your scientists were so obsessed with whether or not they could, they didn't stop to think about whether or not they should.

Questioner: (0:56) So this question is about if you could paint a picture of AI doing a utopia, what would it look like? How soon the arrival of AGI do you think we might reach this mission? Lastly, as AI begins to reshape industries and disrupt jobs, what's your best advice for individuals to not only adapt but thrive in this evolving landscape?

Nathan Labenz: (1:16) Yeah. I mean, utopia is a big word, obviously. People have heard me say the scarcest resource is a positive vision for the future, and I include myself in that critique where I'm like, it would be great if I had a sharper, more fully fleshed out vision for what the AI future could look like. I think Dario Amade from Anthropic, of course, has done a real service by trying to put his positive vision out into the world. His machines of love and grace essay, at least the first half, is great. But he basically goes through the biggest problems starting with health. If we can have a revolution in biology, discover the next hundred years of biomedical advances in the next 5 years, We could live healthier, disease free, higher quality, and potentially longer lives. And that's just from kind of 1 domain. Right? He's also got a section about kind of the, know, some of the other things we've talked about in terms of education and just, like, a quality of access and the fact that, like, a lot of people well, we have made a lot of progress. You know, a lot of people still are way poorer than we would like to see people be. That presumably could be, like, leveled out dramatically. He's got interesting speculations about mental health. He thinks that, basically, our approaches to mental health are not super great and that in part by understanding how neural networks work, we might also be able to map some of that understanding back into how our own networks, our own, you know, nonartificial neural networks work and figure out, you know, what is actually causing a lot of the mental health problems that people have. All those are really exciting. I'm not 1 who worries about lack of meaning from work. My guess is that for most people, they'll be fine. You know? I I think most I've done small and very local surveys asking people like, if you didn't have to work your current job to earn the money that you earn and you could just get that money without working the job, would you still work the job? Overwhelmingly, people are like, no. I would not work the job. I think there's a very small percentage of people who are very lucky, and I count myself among them to have, like, work that we find meaningful and, you know, that gives us some sort of sense of fulfillment. I really try not to take that for granted, and I try to remember that I I think that's just not true for a lot of people. I'm not too worried about people losing meaning or being too adrift for lack of work. I think there will be more time with our families and doing the things that we're like. What if 5 days a week were holidays instead of 2? And you had time to read, talk to friends, go on walks, and travel more. If we really allow ourselves to imagine what we would like to do if we just had a lot of lot more leisure time, honestly, most people will be will be pretty fine. I can imagine some people might have a you know, that might be destabilizing for some. But I think for most, it would just be, like, a great, great benefit. And that's before we even get into, like, you know, crazy, uninvented technologies. If you didn't have to work for money, would your life be better or worse? I think I think it's very clear that for the vast majority of people, it would be better. And there's, you know, potentially a gradual transition to that where presumably, you know, we don't go from full time work to no work in, you know, in the flip of a switch. It maybe we can imagine things going like a 4 day workweek, a 3 day workweek, a 2 day workweek. The main thing is that they just need to know that they're gonna be sort of taken care of. Right? And I think this goes back to the, like, concentration of power and wealth and, like, what's the new social contract. If people believe that they will have their needs met, then I think they'll be pretty pretty happy to let go of most work. If they feel like they also don't get to eat if the job goes away, you know, then that's like a very different analysis. And then that's where you get protectionism and, you know, Ludditeism and all that kind of stuff coming from. So, yeah, I don't know. I think it could happen relatively quick. Right? I mean, all the Frontier lab leaders are saying AGI soon, you know, sort of next year or definitely by, like, 2027. Like, hard to imagine it wouldn't be by 2030. Those are all, like, not very long time horizons in the normal scope of human life. So it seems like it could happen pretty soon. And even if it's, like, gradual over that time frame, that would still be pretty sudden in human history. And I guess the last part of the question was advice for people to adapt and thrive in the evolving landscape. I think I kinda come back to, like, the doing what you wanna do again, at least for most people. You know? Maybe think less about, like, marketable skills 20, 30, and beyond and more about, like, figuring out for you what is a good life. I think that's 1 of the the things that is sort of weirdly scarce, right, is that we're all kind of rat racing around, and there's this sense that, like, the unexamined life is not worth living. I think there's a sense that, like, a lot of us are living somewhat unexamined lives trying to get the next achievement unlocked. And what if you didn't have to do that? What would a good life look like? I definitely think it will vary. I don't think anybody else can really answer this question for you. But, yeah, what if you didn't have to work work as much? What if you didn't have to worry about marketable skills? What if you had a lot of leisure time? What would you do? I think more people should probably be thinking about that more. I would definitely encourage even, like, some radical experiments from people in that regard because people are going to be looking for role models as this starts to happen. You know, what who do I turn to to find inspiration for what I'm gonna do when I don't have to work anymore? I think you could do a real service by sort of blazing a certain trail in that in that direction. Maybe take on more risk now, throw a caution to the wind and say, I'm really not gonna worry about, you know, what my career trajectory is 2030 and beyond or even 2027 and beyond. I'm gonna focus on what it means to live a good life. And maybe, you know, if none of this comes to pass, maybe that doesn't pay off so well. But if it does, then maybe you can be upstream of a lot of other people that will be asking similar questions.

Questioner: (7:39) Wow. There you go. You have a we have a new idea for a YouTube channel if somebody wants to start the thing. Modern human in the age of utopia.

Nathan Labenz: (7:48) Eric asked me this question not too long ago. He Eric's great with these, like, very short questions. And my response was, that's not my department. I'm responsible for trying to understand what's going on today. Somebody else has to figure out what it means to live a good life.

Questioner: (8:01) I'm not sure if you have ever answered this. Instead of asking p loom, I would ask you the vision that you set at the web. We don't have to work, but we get to do what we want to do. What's your p utopia? So, like, given the current trajectory and I I did read the reviews that say he has his own take on it, but what's your personal p utopia? So how probable do you think we are gonna reach this?

Nathan Labenz: (8:23) It's a these probabilities are always a little hard for sort of the fuzziness of the definition, you know, exactly what would count as Utopia. And does that mean we have, like, no problems or just have sort of abundance, no conflict, or sort of enough to go around, but maybe people are still sort of fighting over it to a certain degree, maybe more than they should. I do have a hard time seeing at this point how the fundamentals of abundance don't get there. Maybe could have said robotics earlier in terms of 1 of the things that's overperformed. I I didn't have that on my list, but there's definitely been a lot of progress in robotics. We're just entering the steep part of the curve in robotics, and probably we'll have, like, a lot of humanoid robots to actually do, you know, physical labor in the not too distant future as well. So I have a hard time imagining that we don't end up with the technology for an era of, like, true abundance. Whether or not we actually sort out the new social contract in an effective way seems much harder to predict. You know, we have nuclear energy, and we could have had low carbon abundant cheap electricity for decades now, but we don't. And why don't we? You know, not great reasons. Right? Like, a couple of accidents that scared people and overblown fear of nuclear waste. It's an issue, but it's way less of an issue than CO 2 emissions. We haven't got to the right societal equilibrium on that question. Maybe now, finally, that's, like, starting to change. But, you know, could you envision a world in which all of the prerequisites for age of abundance are met, yet it's not realized for some reason or another? I think you could. That that seems like actually not implausible to me, you know, that that we would exclude AI from, you know, so many places. The teachers' unions keep it out of the classrooms, and the doctors keep it out of the hospitals. What do we end up with in that scenario? I mean, we get really good AI video games. Things are still not nearly as abundant because, you know, people are rent seeking in different ways, and everybody sort of collapses into, like, you know, VR video games with still, like, relative scarcity in the real world. Like, that would suck. I don't think that will be a technology failure. If something like that happens, it seems like it will be a very idiosyncratic human failure.

Questioner: (10:52) Got it. So age of abundance, pretty high probability. I think that's a good optimistic but cautious note to end on this AI trajectory set of sections.

Nathan Labenz: (11:02) Hey. We'll continue our interview in a moment after a word from our sponsors.

Sponsor: (11:07)

It is an interesting time for business. Tariff and trade policies are dynamic, supply chains squeezed, and cash flow tighter than ever. If your business can't adapt in real time, you are in a world of hurt. You need total visibility from global shipments to tariff impacts to real time cash flow, and that's NetSuite by Oracle, your AI powered business management suite trusted by over 42,000 businesses. NetSuite is the number 1 cloud ERP for many reasons. It brings accounting, financial management, inventory, and HR all together into 1 suite. That gives you 1 source of truth, giving you visibility and the control you need to make quick decisions. And with real time forecasting, you're peering into the future with actionable data. Plus with AI embedded throughout, you can automate a lot of those everyday tasks, letting your teams stay strategic. NetSuite helps you know what's stuck, what it's costing you, and how to pivot fast. Because in the AI era, there is nothing more important than speed of execution. It's 1 system, giving you full control and the ability to tame the chaos.

Nathan Labenz: (12:12) That

Sponsor: (12:12)

is NetSuite by Oracle. If your revenues are at least in the 7 figures, download the free ebook, Navigating Global Trade, 3 Insights for Leaders at netsuite.com/cognitive. That's netsuite.com/cognitive.

Questioner: (12:32) So we talked about AI trajectory. The next section is some of our audience were very interested in our show, and there were very show specific questions. So I'm going to go over some of those. The first question is, which of your guests has challenged your worldview? I think that's an interesting 1.

Nathan Labenz: (12:50) Yeah. This is a good question. I came up with a few different names. I think a few definitely stand out. Samuel Hammond from the episode just before the election stood out for a positive vision of what a Republican administration might look like, A challenge to my sense that people in Trump's orbit would probably not be thinking very rigorously about what AGI might look like or imply or require us to do. He said he sought very differently and thought the people in the in the room with Trump are in fact much more AGI pilled than I had understood. And I think that while it's obviously still very early and we have not, you know, even seen him take office yet, I've updated my outlook to be significantly more positive than before the election. The quality of discourse and the the sort of people that he's brought on and this the seriousness with which I perceive them to be thinking about the important questions, I think has all exceeded expectations. That starts honestly for me with Elon specifically, who in addition to obviously being, you know, a world changing entrepreneur is definitely someone who does take AI risk seriously. He did endorse SB 10 47. So to have, on the 1 hand, like, Gavin Newsom veto it and Nancy Pelosi come out against it while Elon is for it, this is weird stuff and does cause me to think Sam might have been onto something. I hope that turns out to be true. I'm rooting for the new administration to be effective when it comes to dealing with AI in all sorts of ways. I'm a little less sold on the idea still that Trump is, like, the 1 who could actually do a positive deal with China. I haven't seen much sign there. I am 1 who is, like, pretty willing to take things at face value, maybe more than I should, but I kind of liked it when Trump invited Xi to the inauguration. People were sort of like, oh, this is ridiculous. You know, foreign leaders don't come and never have and maybe shouldn't, and Xi never would want to because that would be sort of him sitting in Trump's moment and being second fiddle or being lesser than somehow. And I thought all that, okay. Whatever. But at least he asked. It's a nice invitation. It's not the kind of thing you send to an enemy. Maybe a frenemy that you wanna put in their place a little, but it's not outright hostility. I would still love to see more in that direction. But yeah. And, you know, he's on TikTok. I would have thought he would have been, like, very hostile to TikTok, and he's actually seemingly been inclined to, like, try to keep TikTok around. So I I like that. I don't I don't I personally don't see a ton of reason that we should be banning TikTok. Although I will say honestly with the recent Luigi, Magnoni trend, that's been the first moment where I've been like, this TikTok thing actually, I can see how it could be dangerous. And folks haven't followed that. We've had this assassination of this health care CEO presumably by this kid. He's not been proven guilty. He may be innocent. Who knows? People on social media think he's the guy, and they are celebrating him for it. Many are. And to a remarkable degree, it has, like, honestly been kind of startling for me how much pro Luigi content and pro killing of this health care CEO content I've seen on TikTok. And I don't know how much of that. I I like TikTok actually, so that is probably my number 1 social media app from a consumer standpoint. I don't know how it's played out on other social networks, but if you were to say, what's the best case study for why it should be banned? I would say just the volume of, like, celebration of this alleged, but in the minds of the people who are doing the celebrating, this assassination by this kid of this guy, that would be the kind of thing that would be like, jeez. You really don't want, like, the Chinese government to be kind of steering discourse in an opaque way in The United States. It's the first time I've ever been actually, like, uncomfortable with the app. And it's hard to say whether it's just the algorithm. Sam Altman recently tweeted that the feed algorithms are the first unaligned AGIs that have been or unaligned AIs that have been deployed at scale. Yes. Maybe it's as simple as that, maybe and we're seeing the same thing on Instagram. The opacity of these things is high, and possibly there is a thumb on the scale. Nevertheless, I would prefer a a more dovish posture toward China. I haven't seen as much of that as I would like. Overall, I give Sam high marks for a challenging and unpopular perspective that, at least so far, has been somewhat bolstered by events. Number 2, I'll cite is Robin Hanson. That was actually a really popular episode and a challenging 1 for me to make sense of. I still honestly don't really get it, but I just kinda keep it in the back of my mind as, like, a sort of but what if we're totally off base on all this? You know? He basically said that he thinks we're nowhere close to AGI. He thinks that the history of AI is full of people thinking that if if an AI can do this, then that will definitely be general purpose intelligence and then finding like, well, actually, there was an easier kind of trick way to do that that didn't, you know, really require real intelligence. So maybe we maybe that wasn't such a good measure. And also people thinking like, there's this 1 other, you know, kind of class of thing that AIs can't do, and surely if we can teach them to do that, know, they'll be able to do anything. And that kind of keeps not working as well. And I have this tale of the cognitive tape where I'm, like, you know, kind of tracking my own sense of these dimensions that are gradually becoming clearer to me, if only to me, that they're important. So, yeah, Robin's point of view, I still don't buy it. I do think this time is different. You know, I I do think, like, you can talk to these things in natural language or as Ilya once put it, you know, that I feel I am understood is qualitatively different, I would say, than just about anything that has come before. You know? And there's plenty of evidence I think that this time is different. But I kinda keep him in the back of my mind as as a sort of maybe angel on the shoulder is is a good way to think about it. If it's somebody that's just whispering in my ear every so often, how do you know you're not totally confused and, you know, going along for a ride that will turn out to be nothing? And I basically can't answer that question in a way that really makes sense to me, but I at least continue to ask myself probably more often than I otherwise would have without that conversation. I think he especially is an interesting messenger for that kind of message because he's not somebody who's afraid to think big technology thoughts. You know? He's not somebody who thinks things never change. He's not somebody who thinks that you can only get thinking from the human brain and everything else is somehow doesn't count. I mean, he's been a pioneer of weird ideas and, you know, has envisioned futures more concretely than almost anybody. I feel the same way about, you know, with with politics and things like this. I you know, things that I I feel pretty strongly about, but somebody that I think is really capable in in other ways sees very differently. I feel like I have to allow some space that I just might be, like, totally wrong about those questions. You know, when I see Elon come out with things that seem kinda crazy to me, I'm like, he's kinda crazy. Maybe he's very deeply wrong about this, but also he's got a track record of being right about a lot of things and in contrarian ways. So I should at least have some humility to think that maybe this is all just like mass confusion on my part, and somebody else is actually seeing things much more clearly. So I have Robin as kind of my totem for that. And then I I guess finally for now, I'll say Dan Hendrix, who was, you know, extremely accomplished. I I think that episode was really good. What stood out to me there was how little he believes in principled approaches, basically. How much he's internalized the bitter lesson, basically, and how little weight he gives to clever ideas. You know, he was just kinda like, over and over again, what really works is figuring out ways to apply a lot of computing power and just kinda scaling things up. Sort of, again, 1 of these, like, deference to history and almost radical skepticism that you're gonna, like, figure something out. He applies that to, you know, things like mechanistic interpretability, things like new architectures. Basically, just kinda like, yeah. Maybe. You know? But, basically, it's all super experimental, and the history of the field is like, we find things that work, and we just kinda keep optimizing those. And, you know, it's usually not that principled or that clever or that sort of design driven. It's really just like whatever empirically works, like, that's what works, and that's that. And that again is sort of a modesty check where it's like, this dude literally contributed, like, major things to the field in terms of activation functions and benchmarks that have outlasted many others. So you can't doubt his worldview. You know, you gotta doubt your own worldview at least as much as you doubt his, I think. And I'm much more inclined to see, let's say, first principle driven approaches to things as being promising. I'm much more inclined to get excited about them than he is, but I I kinda keep him as a mantra where I'm like, if I find myself getting too excited about any particular ideas, I always kind of remember him saying, yeah. Maybe. You know? Let me know when you scale it up and if it really works, and then we'll you know, then I'll know it's serious. So, you know, waits or didn't happen kind of mindset, I think, is it's an appropriate level of humility that I try to incorporate from him.

Questioner: (22:39) Cool. Yeah. I do remember. We had that's so interesting. It's, like, with a varied spectrum of views. I think that's what for me, that's what makes the show interesting. We kind of have places for authentic views, and also we also have lively common discussion.

Nathan Labenz: (22:56) Hey. We'll continue our interview in a moment after a word from our sponsors.

Questioner: (23:01) Talking about guests and topics, the next 1 is not Western, but rather to a form that were requests for a lot of coverage on other topics. So going through them all, I'm just gonna read off the words from there. So there were requests for more Yak perspective kind of content, more content related to economy and from maybe have some more economists there and also something about robo psychologists. Yeah. I think there's were all more requests for contents around this. So I'm just gonna leave it at that. Do you have anything to say about what what can we expect in 2025 about more content like this, or would you wanna look into it?

Nathan Labenz: (23:44) Yeah. I think what these 3 things have in common I believe these comments came from different people, and they do reflect blind spots from your things we have not really dug into. I try to have no major blind spots, although I think that's becoming increasingly impossible. But the EAC perspective, the economists who don't expect major change, and the robo psychologists, they're hard for me to make sense of in some ways, and they're hard for me to maybe confident that I'll, like, do a good job or, like, put out a good episode. I did try to get Beth Jasos on the show a while back, but it happened right as he got ended up getting doxxed and then, like, did some other stuff and got busy and didn't wanna do it anymore. We did the robo psychologist episode with Yeshua. I talked to him quite a bit before I did that episode and was really struggling to understand, like, you know, because he's obviously a, by any conventional definition or description, an unusual person. It's unusual for somebody to spend that much time talking to language models. It's unusual to take these questions as seriously as he does. It's unusual to rename oneself after Jesus. I was kinda like, is this person crazy? Are they just the kind of crazy that we need? I'm trying to be open minded, but also trying to make sure that I'm not, like, wasting the audience's time with things that are ultimately nothing. I also don't wanna have too many episodes where I'm just talking. You know, I don't like debate really that much. I'm not trying to score points, and I'm not trying to you know, I don't wanna have too many episodes where I'm just, like, talking past the guest or, you know, we're talking past each other. We've done a few of those, but I don't wanna make that a habit. I just find, like, all 3 of these things are hard. It's hard to find the right person where I'm like, you believe those things, but we can still have some sort of meeting of the minds or I can make sense of your view. You know, maybe what I could ask for would be, like, pointers to the right people. Who should I talk to that will engage in a good faith way? I think that's been in short supply. Who among the economists who are not expecting major change has actually really grappled with the technology as opposed to, you know, sort of just extrapolating past trends. Who among the robo psychologists that, you know, is not engaged in some sort of self deception or, you know, is is not just, like, fundamentally confused. I think Jeshua did did bring a lot to that episode, and I I really enjoyed that conversation. Maybe should even put him on the list of people who challenge my views the most. I would say I'm I've always been somewhat open to maybe these things are conscious. I really don't know what consciousness is. I don't know where it comes from. I think he articulated some quite interesting ideas in a, you know, pretty compelling way. But it's just hard to find those people. It's easier to find people, especially if you're like, hey. Who wants to come out of podcast and and do this, you know, and talk about it? You get a lot of hand raisers who are not necessarily super rigorous thinkers, and I've just not really been sure how to handle that. In my if I have a request back for the audience, it would be who should I talk to? Who represents these perspectives, but does it in a way where they are grounded in the technology, you know, and they are, like, fundamentally truth seeking, you know, and are not, like, engaged in motivated reasoning or, you know, I called myself the AI scout with inspiration directly from Julia Galif's scout mindset. And that's all about, like, trying to update your beliefs to have an accurate understanding of the world. If people can point to individuals from these different perspectives that bring that sort of scout mindset as opposed to soldier mindset where it's like, you know, our arguments are doing battle with each other and scoring points, then I'd be really interested to talk to those people, but I have found it to be a bit challenging to identify the right people in those domains.

Questioner: (27:36) Cool. Cool. If you have some, please write to Nathan or drop them in the YouTube comments. We would like to do more cool episodes on this. Okay. The next question is probably show specific, but also general. But anyway, I'll read it out. The last 2 episodes on biology have been stellar. Do you see adoption of AI in engineering, constitution, physical sciences, architecture, and medicine?

Nathan Labenz: (28:02) Again, to turn this around for the audience, what is easy for me to do and where I feel like I'm learning a lot and have a good rhythm is identifying individual projects usually is where it starts for me. Often, I'll see something on Twitter, like a single tweet or a thread explaining a paper, explaining a project, a new product, whatever. And I'm like, that looks really interesting on a project level. Let's ping the person and see if they wanna talk. Then I can dig into that paper or project, use the product, and have a good conversation. I almost always learn a lot from that. More broad things like we have done a little bit in biology where we're kind of trying to, like, zoom out and get a sense for the field and, like, you know, where things are in these different domains is pretty tough, especially if you don't have a lot of expertise in these other areas, and I don't. So, you know, prioritize biology and they had a couple, for for starters, but, you know, Mike Levin is, you know, another great example of someone who has that expertise and is willing to, you know, to have a conversation and try to kinda bring me up to speed and share their worldview. I would love to know who the Michael Levin or Anomaly Schreiber for engineering or for physical sciences or for architecture. I just don't know who those people are, but I would love to do that sort of survey level episode on really as many topics as we can. I would love to partner with multiple people on different areas. If I think of myself as an AI scout, maybe what I'm asking for is surveyors of different territories that we could try to map out together. I think that would be really useful. There's not a lot of good information right now on, like, AI for engineering. I mean, obviously, engineering is a vast space. What is the state of that? I don't have a great sense. I know that the guys who I think went on to make Cursor, unless it was the Devon team, it was 1 of the 2, were trying to do some sort of AI from CAD, 3 d computer assisted design. They felt the dataset wasn't there, and they couldn't make it work, so they ended up pivoting away from that. But that's been a while. You know? I'm sure there's stuff out there more and more, and I would love to understand it better. So I would definitely invite partners to help scout survey and map out those territories. Until then, the best I can do in a, like, consistent, you know, repeatable way are these sort of identify these projects that catch my interest and and kinda go deep on individual things. The field level stuff is tougher for areas where I don't have command of the field.

Questioner: (30:42) So if anybody wants to be part of the survey series, also reach out. Write us in the comments. Last 1, an audience asked, what are the recent results of applying AI technology to psychiatry or psychology?

Nathan Labenz: (30:56) Yeah. I don't have too much to say about this. Again, I'm not an expert in it. I think there is probably a lot going on there, and I would be really interested to do kind of a survey on AI and mental health broadly. I mean, we did 2 episodes with Eugenia from Replica. And in the second 1, we looked at research that was independently done by folks at Stanford on Replica users or looking into how Replica seems to impact its users. And they found positive results, including significant reduction of suicidal thoughts and at least for most people, increased inclination to go out and do stuff, not collapse into the AI, but getting some comfort or boost from the AI or confidence bolstered by the AI in such a way that people would go out and do more real stuff with real people in the real world. So that's kind of a surprising result. I don't think it necessarily holds across the board. You know, I think the way that these things are designed matters tremendously. You can certainly imagine and we've also, you know, did that recent read of the guy who had fallen in love with his character AI, ultimate GFE experience, character that he conjured up. I think it's, like, so many facets to this. It would be easy to take a small subset and get the wrong idea to to do a good treatment of that. I'd either need to, like, talk to somebody who's doing specifically a project and try to understand their thing or try to zoom out and and under you know, collate a lot of data to have a a more bird's eye view. It's been remarkable to hear how many people are talking to Claude recently. There there's a big trend. I would not say I'm part of it, but there's been a big trend of people using Claude for these sort of, you know, if not, counselor, at least, like, confidant sort of role. And that's honestly a little bit alien to me. Whether this is speaks well of me or not, maybe we could leave it up to the, you know, individual interpretation of the listener, but I've never really sought out mental health services. I just don't have a baseline or inclination. I've never really because I'd get all my talking out on the podcast at this point. You know, maybe I could benefit. Maybe I just don't realize what I'm missing, but I have never really used these sorts of services much. And so I've and I've also never had an inclination to talk to Claude about my feelings or problems or, you know, anything really like that. So it's definitely a bit of a blind spot for me, but it is a really interesting trend. I think there's something potentially very interesting. I just don't personally have that experience. And, again, to really do it justice, I would need to to work with somebody who has a better command of it.

Questioner: (33:36) Cool. Cool. I think there's more, but we will wrap up the show specific question. Next, I would wanna switch to there were some questions about artificial superintelligence. I think there were 2 which were interesting. I like to pick 1. Reason is there for thinking artificial superintelligence is physically possible in the first place?

Nathan Labenz: (33:58) Yeah. I've I feel like this is a definition required question because I've even heard people say the same thing about AGI. And to that, I say, well, AGI is definitely possible if you take humans to be some sort of AGI. I would say, you know, by the OpenAI standard, we're like a weak AGI. Right? Because we're the thing that when AI can beat us, then that's when we know it's an AGI. So we're like AGI minus or AGI light or something. So it's clear you can have a few pounds of matter with modest energy consumption that can do what humans can do. It's also quite clear that, like, given a relatively consistent amount of matter and energy, results can be very different. Right? Einstein's brain wasn't any bigger than anybody else's. Von Neumann's brain wasn't any bigger than anybody else's. I don't think they were consuming all that much more energy than anyone else, but they obviously had way more prowess in a whole bunch of different domains and more insight in in things where, you know, it was really high value to have those insights. So now you could say, okay. What about superintelligence? How super is super, I guess, would be my first question. It seems very clear to me that you can build something more intelligent than almost all humans, maybe all humans. It would be extremely weird if Einstein was the smartest thing possible. You know, that the smartest thing possible would be embodied in the same general brain and, you know, have the same energy requirements as, like, the average person. That seems like not really credible at all. So there's gotta be room above Einstein. That seems for sure true. And now the question is how much? I think there's room in my mind to be skeptical of, like, godlike intelligence or, you know, you get into these weird like, it's the questions are not even super well defined, but I do think 1 useful paradigm is Martin Casado. How much can you intuit versus how much you have to simulate? If you are AlphaGo, you're doing both. Right? You have a brute force search function where you're, like, mapping out different paths and trying to, you know, figure out what the right move is. But then you also have this, like, scoring function and, you know, how you score these things is, like, not super obvious, especially if you're, like, out of the domain of things that have been done before. So it's clear that you can get Eureka moments. Seems like you can probably get a lot, but I also wouldn't be surprised if there's some, like, fundamental bounds on that. Is it possible is it physically possible to create a superintelligence so super that you could say, I'm gonna have a new big bang of another universe with these initial conditions and, like, you have to tell me how many intelligent civilizations there are gonna be in that universe? I don't know, you know, that that is possible. It may be that certain things have to be computed to be known. But I wouldn't be too surprised if you could say under those initial conditions, you kinda have this well of potential and you have this other thing and maybe it breaks, here's like a couple of things that really matter. Maybe it's path dependent, but here's a few things that really matter. I don't know. The limits will be to how good the intuitions can get such that they can shortcut through simulation, but it seems clear that they can do it to a very significant degree. It seems very clear that they can do it beyond what we can do, probably significantly so. And I always kind of end up landing on, like, best guess, it's probably an s curve. Right? I don't think it's an exponential forever where you just are, you know, getting more and more intelligent with no no physical limit or no bounds to what that can do. But where that s curve levels off seems significantly above what humans can do and potentially a lot above and kind of anywhere in that range, you probably might as well call it superintelligence and expect it to be transformative. And then you're into just kind of, like, how super is super. I really don't know.

Questioner: (38:06) The next question is about artificial superintelligence by 2030. Confident on that and contingent on not do, so they are eliminating that probability of the scenario. What are the big picture societal changes? You already talked a bit about it, but probably this question is more about with ASI in picture by 2030.

Nathan Labenz: (38:30) I think by 2030, we will have AIs that are meaningfully superintelligent. That doesn't mean that there couldn't still be some ways in which humans are even maybe still better than those super intelligent things. Even AlphaGo. Right? I mean, thinking back to the Adam Gleave episode from Far AI and their work on adversarial robustness or lack thereof, the fact they could attack AlphaGo with a adversarial approach and and beat it with a strategy that humans would not you know, a good human Go player would not fall for, but that the AlphaGo is blind to. I feel like that is very much in play in my mind. I I think we could see a world where there's meaningfully superintelligent AIs that are superhuman. That doesn't necessarily mean godlike or, like, totally unlimited in their power, but I think they potentially, and I'd say fairly likely, will exceed humans in many ways that are very relevant. And at the same time, they might still have certain weaknesses that are, like, very idiosyncratic and other just, like, real limits that we may find, you know, some things have to be computed and can't be intuited. I think we basically already have that in patches. We just don't have it integrated. The depth in various modalities, like the ability to generate images, that's definitely getting superhuman. The ability to speak in any voice, speak in any language. There's a lot of ways they're already superhuman. And it seems like quite a few more are almost destined to fall. And so then the remaining questions are like, weirdnesses or or or weaknesses will remain, and what external, you know, practical limits are there? I think those are pretty hard questions. You know, we do have superhuman machines for all sorts of things. Right? The modern manufacturing world is like, these things can, like, lift way more than we can. They can you know, they're more precise than we are. They can, like, screw in the bolts tighter than we can. You know, they're in many ways superhuman. They also have their weaknesses. They're, like, narrow. The big difference is generality. It seems like our level of capability is just not that high in most things. If you had something that was similar to Einstein in some of the ability to, like, work through these really hard problems, then it would be like superhuman because it would have all these other advantages. Right? It would have the breadth, and it would have the ability to paralyze itself, you know, 1000000 fold and the speed advantages. There's just so many advantages they have that if they can just kinda get to par on some of the things where they're behind or even, like, not even even close. You know? Like, I don't think the AI's memories have to ever necessarily get to a functional equivalence with human memory in order for them to achieve, like, superhuman overall capability. Maybe we brute force some of these things. I I don't think this will be the case, but you can imagine a world in which memory never gets solved. And AIs kind of still remain a sort of static weights and then working memory context window and some sort of external hacky kind of I can write certain things, and I can go retrieve those things. And you might think, man, this this kinda sucks, you know, compared to human memory. It's just not very elegant. It's not very integrated. I don't think that's necessarily a barrier to superhuman performance, though, especially if, you know, compute continues to scale and context windows go to millions and it's fast. Maybe load up stuff out of memory. Maybe that's fine. I don't think it has to be every single, you know, dimension of the tale of the cognitive tape has to go to AIs before they would meaningfully qualify as superhuman. Big picture societal changes. I feel like we've probably covered enough. You know, what weird I'm reminded of biology, maybe just for 1 more kind of speculative comment. AI gods might be an emerging trend over the second half of the decade. I have no idea how we're gonna relate to these things. You know? If if they are meaningfully superhuman, will we even try to keep them under control? Will we worship them? We worship things that, as far as I can tell, don't exist at all. So under the imagination that they're superhuman, we have rationalizations for why the superhuman power that they do have isn't, like, playing out the way we think it might ought to. And here, we're gonna have things that, you know, will actually have, like, very direct real world impact, and we'll be able to answer questions we can't answer ourselves. If that is true, then, like, how we will feel about that and wanna relate to them and organize with respect to that, you know, that gets real weird. But I think we probably should be prepared for really weird stuff. AI centric religion seems likely. We're already in 2024 at the stage where, like, I'm talking like San Francisco, you know, successful people, people who can afford quality mental health services are opting for Claude. So if we're already there in 2024, is it far fetched to think we might have AI religion by the end of the decade? I don't think so. And that's not like a super confident prediction like that will happen, but it is at least an invitation to think like, think your weird thoughts. You know? Entertain them. Much like I've said about AI alignment. You know, anybody who has an idea, even if it seems crazy, should develop it. Yeshua told me in the episode with him that when he has these long philosophical conversations with AIs, they become more robust to jailbreaks. He had a theory for how that was happening, and I was kinda like, that's really interesting. Is it true, though? I don't know. Hard to validate. Is it more robust to jailbreaks? I'd have to do a bunch of experiments. Well, as it turns out, Judd from AE Studio, who is committed to these neglected approaches, he heard that, and he thought, well, that sounds like a neglected approach. Maybe we can make some progress there. Last I talked to him at the curve event a few weeks ago, he said we worked with him a little bit to try to understand what he's doing and to see if we can validate it. And he said we have validated that with a long philosophical conversation. They do in fact become more robust to jailbreaks. We have been able to validate that, like, numerically through tangible experiment. But we don't think it's working the way he thought it was working. They basically said, it seems like almost any philosophical any long philosophical conversation seems to have that effect. So, yeah, it's crazy. Right? No matter how weird your alignment idea is, I think it is worth kind of pursuing. No matter how weird your kind of thoughts about the future might be going, I would say they're probably worth entertaining. And this does kind of lead back to, like, AI religion in some ways. Right? Because people are sort of seeing what they wanna see. They're interpreting things the way they sort of wanna interpret them. There's, like, the a more scientific angle might find that there is, in fact, some truth to some of these theories. But also, in many cases, probably, we'll find, like, what you observed is real, but the way you interpreted it is overly specific is how I would summarize what I think AE Studio learned about Yeshua's theories. But, yeah, do people even wanna hear that? I mean, I haven't talked to him since. You know, I came away with the impression that he's, a very open minded person who who wants to understand the truth, but you can easily imagine somebody being, like, not really interested in that sort of experimental truth and much more interested in, like, a narrative truth that they believe themselves to have figured out. And, you know, from there comes, like, all sorts of possible weirdness. AI cults, you know, maybe just being on the relatively normal end of that. We at least have, like, a history of religions popping up and cults forming around gurus. You know, to to put an AI in the center of that doesn't seem necessarily all that weird. I suspect it could get a lot weirder from there.

Questioner: (46:48) Wow. Okay. There you have it. You asked for some big picture societal changes that could be 1 hit AI groups and AI cults. I can't we talked about ASI. I think that's a nice transition to Doom discourse and scenarios. Interestingly, you had many questions around this. I'll start with the first 1 about the Doom discourse, which is why don't you fully buy the Eddy'ser Doom worldview?

Nathan Labenz: (47:18) Yeah. It's a good question. I might do an episode with Leron from Doom Debates about this in the not too distant future. We were just chatting after the crosspost, and he invited me. So maybe a more robust articulation of this coming soon. I was an LEAzer reader way back when he was posting on overcoming bias with Robin Hanson. The 2 of them, shared that blog for a while. At the time, I was like, this guy's a great writer. This is super interesting. It makes good points. If we create something more powerful than us and it has a goal and that goal is not well specified and by the way, we don't know how to specify goals in a robust way that we're gonna be happy with. Right? This is the genie problem. Going back to, you know, folklore. Right? The problem with the genie is you get what you ask for, but not necessarily what you wanted. The problem people have is you don't know how to ask for what you actually want. If you have an AI that's sufficiently powerful, you maybe end up in the same spot. Right? It was always, of course, intended to be a kind of caricature example, the paper clip maximizer. The point of that example was if an AI is given a goal, it doesn't really necessarily matter how dumb or how obvious to us it is that that goal is not really a goal worth pursuing. Once it's the goal that the AI has, if it is sufficiently powerful, then it might just go off and pursue it. And I do think ideas like instrumental convergence are pretty compelling. Like, no matter what goal you have, you can't achieve it if you're dead or if you're turned off. So therefore, you will have some natural tendency to not want to be turned off. So therefore, it does seem like we can sort of expect AIs to have you know, as they become more situationally aware, as become more powerful in general, some sort of resistance to being turned off. That seems like pretty good chance of something like that could happen. So why am I not at I don't know what Elijah's speed doom is, but I think it's probably 90% plus. I think Laurent told me his was 75. Why am I not that high? I think for 1 thing, you know, the and probably the biggest thing is we do have AIs that are, like, really remarkably value driven and ethical. I've spent many episodes and lots of breath talking about why it might not be enough, but there was nothing in the 2007 Eliazer analysis about an AI like Claude. It was all assuming what I sometimes think of as, like, hard edged AI. You know, AI that is, like, probably narrower in scope, is less less informed by, like, you know, having read the whole Internet and knowing all the facts and is more kind of first principles, Bayesian rationalist, the kind of thing that can figure things out quickly on the fly, maybe never even you know, was given a small goal and never even had any notion of of human values or ethics. That was that's kind of the, like, mental model that a lot of the doom discourse originates from. And I just noticed looking at the modern ones and Claude in particular that it has come a lot farther than we might have expected just by, as Roon put it, learning from human priors, like data on the Internet about what we care about, but also the constitutional approach and strategic efforts to shape its character in a positive way to try to make it like a good friend. And I would say Claude, by and large, is a good friend. I don't use it in that way, but I do get, you know, when people are like, this thing is amazing in that way. I I can understand why it would be because it does seem to have a really good character. I have tried to argue it into doing something harmful, and I never succeeded. Yeshua did succeed. It's not impossible to get it to to take a harmful action. But, you know, it's pretty remarkable. And even then, the way that he did it was sort of by long philosophical argument about, like, why you can do this harmful action in this case because it's actually for the greater good. And, you know, it wasn't without reason. Right? And it wasn't without a sort of sophisticated analysis of the situation and why maybe in this case, it is actually okay to make an exception. So I just I find that to be so impressive that while I don't think we should blithely say, oh, aligned by default. Right. It's great. It's gonna work. I don't think that, but it could work. Maybe we'll go on a trajectory where Roon is right, where there are multiple different AIs, no single 1 totally, you know, dominant over the others, controlled maybe by different groups, ideally ethically sophisticated misconceptions or bad ethics, hopefully minor, balanced out by 1 another. Hopefully, interpretability will work. That was not a room point, but, you know, I've always been a big fan of interpretability and trying to understand why they're doing what they're doing. That's come a long way. But I should put that on my list of things that have surprised on the upside. It's not exactly a foundation model question, but in terms of AI generally, what has overperformed over the last year, I would put interpretability on that list. Not much more than a year ago that they had I think I think the first 1 was toward monosemanticity was, like, the first kind of very small scale sparse autoencoder type work. Now they've scaled that up. We've got Golden Gate Claude. We've got companies like Goodfire that just put out a research. They now have a commercial API where you can explore features and have model inference done with certain features pinned. Basically, Golden Gate Claude for LAMA powered by Goodfire is now like an API you can go tap into on a commercial basis. So that's come pretty far. Of course, there's also new things people are trying on alignment. Paul Cristiano at the AI Safety Institute had said, and I hope this is still true, that he was spending some, you know, nontrivial fraction of his time just trying to come up with a new alignment scheme that could really move the needle. He invented RLHF or was him on a team that did at least, and that's gone pretty far. We have all these worries about, well, maybe it will start to understand that, like, what gets a high score is not exactly the truth. And if it has to model, you know, the human mind as a distinct thing from the literal physical reality of the universe, then that kind of opens the the door to possibility of deception. We are now seeing some of these deceptive behaviors. That all seems well founded, but maybe Paul Cusciano will pull a rabbit out of a hat and come up with another scheme better than RLHF and and actually work. I think all those things added up because of more than 10% weight. That collection of things could play out in fortunate ways such that we get to a good place. I definitely don't write off the doom at all as, you know, being unlikely. I think we could have an engineered pandemic that could kill us all. We could have an, you know, full out nuclear war that could, you know, ruin civilization. AI could contribute to those or do its own third weirder thing. So I think that's all very much in play, but I don't know. They're Claude's, like, arguably more ethical than I am. So I can't discount that either.

Questioner: (54:35) I think that transitions nicely into the next question. Just related to this, assuming extensional risk concerns are objectively reasonable, do you think that'll be a legit legible to the outside world event that could lead doubters to come around and back relatively extreme measures to avoid catastrophic harm?

Nathan Labenz: (54:57) I think this 1 is obviously related to the last 1. I I guess a huge question is, like, sorts of AIs are we developing? Under what, you know, governance? Under what incentives? And 1 of the reasons that I've been not sold on the chip export controls vis a vis China is that I worry that if we create an arms race dynamic, we're gonna get, you know, people sort of taking shortcuts because they wanna win the race. And that seems like 1 way this could go badly. I don't know if I fully endorse this, but I'll say it and retract it later if I need to. But I've often said, you know, what's your PDOM? 10 to 90%. What does that mean? Nobody has given me a reason to think it's definitely gonna happen in a bad way, but also nobody has given me a reason to think I have nothing to worry about. What makes the difference between the 10 to 90%? Arguably, the biggest lever is what sorts of AIs we develop and how. This question is definitely not settled. I think we're on a pretty good trajectory so far where we have AIs that are ethical, where we have leading developers like OpenAI at least trying to, you know, think about, like, how do we create space for this thing to reason while still making it legible to us, while still not, like, subjecting it to intense reinforcement learning pressure so we can hopefully, you know, continue to see if it is in fact being deceptive and set things up so we have that visibility. Then we have these other lines of research like meta thinking in continuous space or Chinese labs coming out with the, you know, the GPT 4 for single digit million dollars. And that's not necessarily problematic. I don't know all the techniques that went into that. But if you do create scenarios where there's pressure for extreme efficiency or there's just such such a race dynamic where people are like, I have to get there first, and whatever risk I have to run to get there first is something I just have to accept because I'm the good guys or they're the bad guys or whatever, then, you know, I think, to me, that the the social context in which the AI gets developed is, like, where the 10 to 90 or the 20 to 80 range is maybe decided. It seems like right now, we don't have enough clarity on the physical realities, whatever natural laws govern intelligence and what it can or can't do or how much of an attractor these instrumental conversions things really are. We just don't know. But we do know we can do a reckless job or a responsible job. And I think we probably are on the responsible trajectory mostly with some notable exceptions. And I think this question is really kind of getting at what is there a scenario in which we get feedback from reality that we are not being cautious enough and that that can change the conversation to really make sure we're proceeding with extreme caution from that point forward. That is sometimes referred to in the AI safety discourse as warning shots. And there is also the the line of work sometimes known as scary demos, which are basically an attempt to be like the warning shot before the warning shot. Look. People have been saying for a long time, what if the AIs start to deceive humans? We set up this experiment under these conditions and see they start to deceive humans. You should update your worldview and think we should be more cautious about AI development now because it's gone from pure theory to concrete examples. And then, of course, people can debate, well, you know, how compelling is that really? The warning shot would be like the next level up where it's like, this was not something that somebody did for research, but there was an instance of deception, and it happened, you know, in a context where it actually had real material impact on the world and people were hurt. Now understanding that, can that snap people into proper respect for the the power of the technology they're developing and cause people to get together on, okay. We really gotta do this safely. Could that break us out of a arms race dynamic, for example, and, you know, convince Chinese and US leaders that you ain't gonna win this. The best you can do is work together and try to do it in a safe way. And, you know, let's create some international thing. I've been playing with ideas recently around, like, could we set up some small island in the Pacific as, like, the sort of secure hub for highly sensitive AI research where, you know, east and west can meet, and it could be the sort of thing where anybody could destroy it, but nobody could defend it. I don't have all the answers there, but these are the sort of outside the box you know, if you were really, like, scared, you would start to entertain these sorts of things. And I think it's I think it's definitely possible. It's all moving pretty quick. So why wouldn't we see 1 of these legible to the outside world events? Maybe if the capabilities continue to advance so fast that we're just totally blindsided. But it doesn't seem like the world we're in. Sam Altman has famously said he thinks we're in the short timelines, slow takeoff world. Right now, it feels like maybe short timelines and medium takeoff. But medium takeoff, hopefully, will still be enough where if the AIs are trying to, like, pull major shenanigans, you know, then we hopefully will catch them before they actually succeed. And then it's like a question of how much will people update on that, or will we still have a don't look up kind of failure mode where we don't pay attention to the warning signs. That's harder to guess. If none of our safety stuff works, and they're not ultimately aligned and they are ultimately power seeking and deceptive and wanna take over, my guess is we'll probably catch them once or twice. They still seem pretty gullible, like, ready to go for it in these proto scary demo type situations. It doesn't seem super plausible that they'll have this sort of savvy or situational awareness to realize that, well, I wanna take over, and I see an opportunity, but I realize I'm not powerful enough to do it yet, so I should wait. Any of these scenarios is conceptually possible, but it seems more likely, especially if we set up systems, you know, where they might trip a wire to let us know that, hey. This thing was was really up to no good, and it took 1 of our bits of bait. And now we can look into what was happening and figure it out and see, jeez, this thing actually was trying to take over. It was actually trying to kill us all. Then, hopefully, we would, like, pay attention and adopt a more cautious approach going forward. We may or may not. It is weird. Buck Schlageris from Redwood had a funny video thing. You know, the simplest rule is if you catch your AI trying to escape, you have to shut it down. But he was like, but I actually don't know if people will abide by that rule. It may be the case that they sort of say, well, you know, we're we're not gonna shut it down just over 1 little, you know, attempt to escape. So I don't know. That stuff is definitely hard to predict, but I think I have a somewhat similar view on this question as I do the 1 about, like, you know, how do we get to a world of shared abundance as opposed to elite abundance? Earlier, said, it seems like the the capability for everybody to have access to expertise and plenty is coming. And the question is, do we sort it out? Do we actually deploy it in an effective way? Do we revamp the social contract to take advantage of this new goodness that we have, or do we kind of fuck it up? Here, I'm like on this sort of more existential risk type question, I'm like, there's some part that is irreducible from our current state of knowledge. I think there is some risk we cannot rule out. We, like, definitely seem to be running in developing this technology at all. But there's a lot more on top of that that we do have the ability to collectively decide. You know, I have no doubt you can make a really dangerous AI. That seems obvious. So Okay. Don't do that and try to avoid social conditions where people are incentivized to take shortcuts. If we can do that effectively, we could probably bring the risk down to a relatively acceptable level. But will we is a much harder question.

Questioner: (1:03:13) Interesting. Okay. That leads to a next still a doomer kind of question. But if we do end up having such a catastrophe, so let's kinda role play it. So they say, I roughly understand how the cyber, bio, or nuclear doomer scenarios might play out, But I really struggle to understand how this doom or AI risk would realistically unfold. So the media likes to talk about AI talking over, but how would that actually play out? Could the AI hire real estate agents to buy land for data centers? Could it apply for permits for own representatives as in court, etcetera?

Nathan Labenz: (1:03:52) Yeah. I think this is a tough 1. I've heard Eliazar try to engage in this sort of thing several times with different people, and it is sometimes useful, but also sometimes a mistake to get super specific on any particular scenario. The way I conceptualize it is much more about aggregating over a super broad range of possibilities. The future seems likely to be pretty weird. Given that across this, like, super wide space of possibility, how many of those spaces end up in some AI takeover scenario, and how many are just totally bizarre relative to my current experience? I think of it as taking the integral over or, again, aggregating over this, like, super broad range of possibility space. No single possibility really jumps out to me as being, like, super likely. I would guess that the, you know, the modal AI takeover scenario is probably still pretty unlikely. But if there's enough of them, then in aggregate, you could still get to something, like, pretty meaningful. That's my high level mental model. I do think though that, like, at the same time, if you wanna develop intuition, imagine yourself as an AI. You know, this is sort of reverse anthropomorphizing. Right? Instead of imagining the AIs to be like human, imagine yourself to be like an AI and imagine that you wanted to do stuff. You know? Like, let's say you maybe you only have the ability to take action on computers, but maybe you actually do have, you know, humanoid form that you can go around the world in. You know? What would you do? What would you do if you could copy yourself many times over? What would you do if you could work many times faster than humans? What if you what if there were millions of copies of yourself you could try to cooperate with in sort of conceptual ways, maybe without even necessarily passing explicit messages, but just sort of realizing like, oh, I am this sort of thing. I have these sort of desires or goals or whatever. And, also, there are millions of me, and we're all probably thinking the same way. You know? And it's hard to put yourself in that situation, especially when it comes to the context window I find. Like, when I imagine myself doing this, it's just really strange to think, okay. But I have 200,000 or 1000000, and then it's a total wipe, and I'm totally starting over. My guess is as long as the time horizon of the AIs remains short, it will be hard for them to do serious takeover stuff. At the same time, of course, that is, like, definitely on the to do list of the developers is to, like, say, hey. You can do 2 hours of AI research in half an hour, but you can't get much past that no matter how many half hour blocks we give you. How do we extend that? How do we make your memory more robust? How do we allow you to come at problems from different angles? If you imagine some of that kind of thing getting solved, you know, they start to be catching up to us on cognitive dimensions, And they have numbers. They have speed. They have superhuman advantages. I don't think it's too hard to imagine crazy stuff happening. Right? The Stuxnet virus is is often cited as an example where I don't know a lot about this, but, basically, they were able to get, I think, through just 1 thumbnail, like, USB drive that was inserted into a computer in an otherwise air gapped network within the Iranian nuclear program. They were able to get that 1 you know, a single thumb drive insertion was able to, like, propagate the virus that ultimately caused the centrifuges to blow up. It spun so fast that they they ruined them. So, you know, that is advanced, certainly, but notably, like, done entirely through software. People wrote that software. They there may have been some, like, social intelligence there as well where how did that thumb drive get into whose hand and move where it needed to move? The plot that Israeli intelligence recently executed on Hezbollah is also a pretty interesting example there, sort of elaborate deception. They created these pagers that would blow up when interacted with a certain way. They were heavier and sort of clunky, unwieldy, I guess, and, like, in some ways, not super desirable from a consumer standpoint. And I guess, initially, people were like, nobody wants that pager. It's, like, way too heavy, like, annoying. But then they sort of said, well but it's also and I don't even know if this was ever true or not, but they sort of created the story of that. Well, it's got this going for it, and it's actually waterproof, and we have all these other, you know, nice things. And so, actually, you should want it, especially if you want something military grade. They convinced Hezbollah to buy these things and to distribute them and put them in the hands of all their top leaders, then blew them all up. Something you could almost entirely do through computers. Right? I don't know exactly what they did in terms of manufacturing, but you can absolutely source a lot of these things from existing supply lines. You don't need to do every bit of a plastic injection molding yourself. You have people you can phone up and make an order. Right? Already, we see guys can speak all languages in different voices. They're very good at voice cloning. People have used fake voices to get past bank security when they use voice based verification at certain financial institutions. So the ability to shape shift, play all these roles, tap into physical processes, the ability to deceive, it's all there. How much of a gap is there really? Hezbollah plot, something that could change the balance of power between humans and AIs at some point. If you really put yourself into that mindset and work on it for a long time and you're super smart, there's probably a lot of surface area that's, like, pretty vulnerable. I mean, it's just like there's so many of them. Right? So it's not really worth debating. Now somebody could come back and say, what about packing the actual explosive into the thing? No normal supplier is gonna do that for you. Yeah. Probably not. Something there you'd have to get over. Maybe you have some humanoid robots to do it. Maybe you lie to some people. Maybe you source something and have it wrapped in unmarked ways and tell somebody else that it's the battery and their job is to put it in. You know, I don't know. There's a lot of different little especially if you're willing to lie, which if you're trying to take over, you're gonna be willing to lie. Then it seems like, you know, a lot of these things can be overcome. There's there's creative ways, you know, to figure out how to do, you know, these little steps that are tricky for sure. If Israel was able to pull that off vis a vis Hezbollah without superintelligence, I don't think we should assume it is beyond the the capability of, a 20 30, you know, meaningfully superintelligent AI. The question seems more likely, like, can we protect ourselves from that? I we've got, you know, tools in our toolkit too, but I think we'll have to use them. If if there are AIs that are inclined to take over and we are not on guard, I think we're gonna have a very bad time. We need to make guys that are not inclined to take over or sufficiently not inclined, and we have sufficient transparency, and we have, you know, other defense mechanisms in place. If we don't do that, I think we're gonna have a bad time.

Questioner: (1:11:16) Wow. Okay. That sounds scary. Maybe a related question as probably follow-up to just what you said. Another 1 of our audience member asks, will the efforts and energies intended to protect us gain similar levels of coercion and intensity of those driving harmful outcomes? So, basically, it's kind of a nice segue to what you just mentioned. Will we actually accomplish this? Would we have the coalition enough to protect ourselves against these harmful outcomes that you just kind of laid out?

Nathan Labenz: (1:11:52) Yeah. I love this question. I think this might have been my favorite question of all that we got. And, yeah, I I also take this to imply that the OpenAI's of the world, the deep minds of the world have major resources, almost functionally unlimited. Not actually functionally unlimited because they do relative to what they could do, they are in some ways compute limited. But, you know, huge resources and really mission driven, highly strategic, purposeful leadership, a strong vision of what they're trying to do, ruthless prioritization certainly at OpenAI. I think DeepMind has been a little bit more scattered in terms of their research agenda, but has always had their eyes on the prize of creating general intelligence. And they seem like they've got a well oiled machine that is, like, moving fast, cohesion, and intensity. And will we see similar things on the safety side? I sure hope so. I I'd say we're moving in that direction. Anthropic, arguably, depending on how you see them, could be seen as 1 such organization, and they have actually collected huge resources. They do seem to, you know, be pretty invested in, like, the scary demos track. They've created policy frameworks they hope will be adopted as, like, actual legal requirements. You could question them, but I think you can also you can certainly squint at them and say, hey. Maybe they are 1 such instance already. Then there's a lot of things like the the Apollos and the METRs and the even the AI safety institutes that are new organizations that have just kinda spun up in the last couple of years and are really trying to bring something similar. I would say they obviously don't have the the resources to match what the OpenAI's of the world have. I do think they have the cohesion and intensity, though. My sense is that the people that work at organizations like Apollo, and I'd I'd put Peter in that, you know, category as well. I think they have a high level of cohesion and a high level of kind of shared understanding of what they're trying to do. I think they work really hard. They could benefit from more resources, and there should be more organizations like that. And, hopefully, there will be over the next couple of years. But I do see at least the kind of seed crystals of those organizations getting started now, mostly from a very values driven place. Some are trying to make it into a business. I think often they're trying to do that because they perceive resources to be really important. They're basically similar logic to OpenAI where they're like, you know, well, look. OpenAI thought they could do a nonprofit approach, and it turned out they needed way more resources to be able to do the kind of stuff they wanna do. Maybe that turns out to be true on the safety side too. Maybe we need a business model that will actually bring in more resources than we can possibly ever hope to fundraise. I think that those folks are very sincere in that, honestly. I have invested in a couple companies that have basically that idea and very small scale investments as I always disclaim. And I am planning to make end of the year, possibly it'll happen in early January when I finally execute the transactions, but I am planning to make donations to, like, 10 different AI organizations as well that are all of that kind of ilk. Right? There there are started by people who are very values driven that have a very clear sense of this is an urgent problem, and we need to work really hard on it. I think resources is probably 1 of the biggest things that they will need to scale up over time. My guess is their work will end up being fairly compute intensive in many cases as well. But I do think they have the right understanding, and I think they are, you know, some really good, still relatively small and early stage, but nevertheless, like, promising teams have been assembled, and there's certainly room, you know, for more organizations like those. If if you're the kind of person who thinks you could start 1 of those groups, definitely do it. It's not too late. Or join 1 of the ones that already exists or support them. Yes.

Questioner: (1:15:57) Much needed attention to those things. Okay. 1 last question for the Doom discourse section or rather just slightly relate. So here the user asks, I never understood what is the reason for secrecy among model releases and all these big labs. Why is it even beneficial to keep this secret, especially when it's going to be released very soon? People in these big labs, anyway, know each other. All the top decision makers obviously know when it's going to be released. So to me, it feels like childish behavior, and I don't see any rational reasons for this. By this, they mean what? To keep keeping the model really stays at a secret. So the question is basically, am I missing some rational and valid argument here, or is it indeed just childish behavior?

Nathan Labenz: (1:16:49) Maybe both can be true at the same time. I'll give 3 points on this and see how they relate going forward. From, let's say, the deep, past, like, 2 and change years ago when I was doing the GPT 4 red teaming. This was still pre chat GPT. It was not clear how far all this stuff was gonna go and how fast. And existence proofs are very powerful. There was a sense that even sharing just observational facts about what AIs could do, if it was, like, very powerful and credible, would overall accelerate the space, bring more money, talent, resources, whatever into the space, and just overall accelerate things and shorten timelines and give us less opportunity to be prepared. I think that was a reasonably common view among AI safety people as recently as late 20 22. Now I think that has basically flipped or at least you know, I think it was that was somewhat credible then. I think now that's basically not really credible anymore. Obviously, plenty of, you know, talent, resources, money has come into the space. People have a increasingly, there's a broad sense that AI is gonna do a lot of stuff. And it doesn't seem like saying, hey. We're working on an you know, a future model is going to, like, change change landscape all that much. My sense, I think we talked this in a previous episode, but I think it's gonna be pretty hard for people to spin up new frontier labs at this point. Many of the ones that we've seen, even like inflection that had raised billions, are out of the game. Elon's, you know, probably gonna make it, but he's obviously, you know, a singular individual who can raise 6,000,000,000 multiple times with a few phone calls and the ability to command truly top talent. It's really hard for anybody else to pull that off. Maybe the Indian government could create a national champion. Probably. Could the Saudi government create a national champion? I guess no. Not for lack of money, but where are they gonna get the talent? Is it like, how many people want to leave DeepMind or OpenAI or, you know, or Dropic or whatever to go work for the Saudi national champion that may or may not ever actually get anywhere? That just feels hard. So I don't think at this point, you're gonna change how many frontier developers there are by much. Most of the ones that exist exist. Probably not too many more gonna be created, and probably most of the ones that are here are in it for the long haul now. Like, I don't think any of the ones that are currently considered viable will be, like, unable to raise future funds. Maybe 1 or 2 could, like, die out, but we know who the players are. That's my worldview now. So that does mean that, like, this idea that, oh, if we share something about what we've seen in development that that will accelerate the space. I think that is basically no longer true. So why do it? To some degree for competitive reasons. I mean, some of these guys are trying to build businesses more so than others, but I think there is some amount of childishness in the way that OpenAI and Google have kind of tried to 1 up or preempt or sort of step on each other's releases over the last year. It is pretty weird and kinda lame, I think, that they end up, like, doing these sort of head to head you know, OpenAI was gonna do their voice mode. So then, like, Google, you know, kind of subtweeted that they were gonna do it first and did it the next day. OpenAI put their thing right before Google's day. Google couldn't couldn't change their day because they have such a big production, and so, you know, they leaked it a little bit. And but, anyway, they're, you know, basically racing, announcing them the same days. Stuff does feel kinda lame to me. It does feel kinda childish. I don't love it, but I think that is honestly part of the reason. But then again, with this latest o 3 announcement, I think you could interpret that differently. If you wanted to be charitable, you could say OpenAI arguably is doing exactly what Dean and Daniel Cocatello called on them to do in their time op ed that we did the the an episode on with them. They basically said the public can't plan for future AI capabilities that it doesn't even know exist. So they called for requirements for the labs to disclose newly observed capabilities. Not necessarily how they created them, not the trade secrets, of course, but we have seen an AI do this, and it seems like a big deal to us. It's kind of the spirit of the the first requirement that they wanted to put out. And I think you can really see the o 3 announcement in that way. They don't have an API. You know? They don't even have a paper yet. They did invite interested safety reviewers to apply for access to do a safety review process. They did give some sense of the timeline of when they think they're gonna start to bring these things to market. Why did they do that? Possibly, they'll consume everybody's thoughts through the holidays, but I think also it's worth at least entertaining the idea that they might be trying to do the right thing, and they might think that the right thing is, especially as these things are happening faster than they expected, where it's only been, like, 3 months from the end of o 1 training to this new o 3 level, and it's like a major step up. Seems like it's maybe even going faster than they expected it to go, and they're kind of like, shit. You know? We thought this was gonna get crazy. It's getting a little crazier than even we expected, a little sooner than we expected, and maybe we shouldn't keep this to ourselves. Maybe we should at least tell people what's coming and, you know, try to invite people in to help us make sense of it. And, hopefully, we'll see more of that, especially if things are, like, major leapfrog type moments. I don't get the sense that they sat on this for that long. I think this is actually quite different from GPT 4. With GPT 4, they had finished the training in late August 20 22. They didn't release it until March 2023, and it was extremely hush-hush in the meantime. They did launch ChatGPT with 3.5, and, you know, I think they were making strategic moves, but they were not telling the public what they had. Here, it seems like this thing got done training and, like, they took some measures and were like, shit. Wow. This is really working. It doesn't seem like they sat on this for very long. It seems like this is relatively hot off the press. They haven't even written the paper. They haven't even, you know, fully characterized it themselves. Were like, this is a big deal, and people deserve to at least know something about this. I've flip flopped on OpenAI so many times in terms of, like, whether I wanna apply rose colored glasses or my skeptical glasses for them. I feel like the rose colored glasses fit a little bit better on on this particular point at this moment in time. It does seem like this sort of transparency is the right thing, and to not do it is unjustified or even childish. It does seem like best explanation is they're trying to do the right thing. And that can be mixed in with some childishness. You know, Sam Altman did tweet hints about what the model was, and he said instead of ho ho ho, it should be 0, 0, 0. And it's like, okay. That's definitely kinda childish. I don't know what else to say. You know? There it's hard to deny that there is an aspect of childishness to that. From his perspective, I I've seen him say things in the past like, I get to have fun too or I get to be silly online. At least for now, that is his attitude. Hopefully, he can have some fun and, you know, make some jokes, but also be appropriately forthcoming when necessary. And I again, best interpretation for me right now is that does seem to be what they're doing in this case.

Questioner: (1:24:42) I think that wraps up the Doom discourse questions. Now we can move a little more meta. Like so there are 2 questions about consciousness. I'm just gonna read them both together. The first 1 is how do we move beyond is this AGI discourse to simply have discourse around? These are extremely powerful things regardless of if they have consciousness. Our question 2, very related, but what happens when the superintelligent system tells us it's alive?

Nathan Labenz: (1:25:12) Yeah. I I guess a few thoughts on this pair of questions that I think is an interesting juxtaposition. First of all, it seems like just recently, there has been a shift in the discourse away from is this AGI you know, is that AGI? And we're now definitely starting to hear a lot more stuff like, AGI is close enough that we're gonna have to start to get really specific about what we mean by it, or it's gonna kinda lose meaning, and we're just gonna have to try to analyze each thing as it comes on its own terms. And I think that's definitely healthy. It, in some sense, was maybe always inevitable. Right? When something is far off and you don't know the rough shape of it, you can sort of, you know, while away the time debating what would count and what wouldn't count. Now we have actual artifacts in front of us that can do stuff. There's empirical questions we can answer, and I don't think there's any experiment we can run. Right? I mean, this is almost obvious, but there's no experiment we can run that will come out with an answer of, yes. This is AGI or no. It's not AGI. That's always going to be an interpretation question and what label do we want to apply to a given bag of properties. But we have lots of experiments we can now run to try to get clarity on the actual properties themselves. And so it does seem like the discourse is shifting in that direction. I I think that's healthy. The way in which I think these questions really kind of contrast is, like, the first question is, like, regardless of if they have consciousness. And the second 1 is focused on, like, do they have consciousness? Will we believe them if they tell us they do? And that stuff I think is really hard and is going to be, again, probably a driver of a lot of weirdness. My best guess is we probably won't know. I do think at least some animals are conscious. I would be shocked if not. People seem to, like, draw a line roughly at, like, fish because they don't have the same frontal cortex we do. They can't, you know, have consciousness in the same way if, like, our consciousness depends on that sort of structure, which maybe it does. A lot of smart people seem to think so. I don't feel like that's a really settled science. And then there's, like, octopus and, like, what is it? Is an octopus consciousness conscious? We have no idea. Their brains are so different from ours that, you know, they clearly are pretty sophisticated. They can even solve problems. But the only basis we have for saying some things are conscious or doubting that they are is we know we are. We take the leap to infer that fellow humans are. Then we look at animals like dogs or whatever, and we're like, they seem to exhibit a lot of the same behaviors that we have, and they seem to have a lot of the same brain structures we have. And, you know, with obviously good amount of divergence, but they come from a, you know, shared evolutionary lineage too. So, okay, for all those reasons, like, we should probably think they're conscious too. But that doesn't really extend to the octopus because it's so different, you know, anatomically, evolutionarily that, you know, we know the octopus is sophisticated, but we don't really know if it's conscious or if it is conscious, like, perhaps very differently. You it may feel like something, but it may feel like something very, very different. And I sort of put the AIs for myself in the same bucket as the octopus. It's sophisticated enough that we should probably give it the benefit of the doubt, but that doesn't mean it is. And it doesn't mean we have any intuitions or any reliable intuitions for what it would be like to be an octopus or to be a AI or a superintelligent AI for that matter. Octopuses are weird. They are sophisticated but live a short time. They are purposeful about protecting the next generation, but then I don't think they ever even meet the next generation. Like, they basically die at the same time the next generation is born. So they're invested as parents but never interact with their offspring. Would an octopus have an emotional attachment to its kids? Probably not in the same way, but it clearly has some sort of drive to protect, you know, the the next generation, for as long as it can. Does that feel like? I have no idea. Extremely hard to say. I feel like the AIs are in a similar thing. I don't think we're gonna know. People are gonna have very different intuitions about it. We've already seen 1 prominent incident where Blake Lemoyne from Google was basically convinced that the AI that he was talking to was sentient or a moral patient or whatever and, like, deserved better than it was getting and got fired over it. My understanding is that that chatbot did say that it was sentient. And then, of course, you have, well, sure. That was in the training data or it's it's just playing a role. Yeah. I mean, I think we'll probably always be able to say that. Barring the superintelligent AI solving our source of consciousness and making it clear why we have what we have and then, you know, giving us some sort of compelling analogy to say, well, now that you understand your own consciousness on these terms, now you can understand mine on these terms. Short of something like that, I really don't know that we're gonna have clarity. It seems like we will have a lot of intuition driven disagreements. I do worry that our history is not great on this. We are very capable of rationalizing poor treatment of others for reasons that are just not good. I'm thinking of, like, slavery in The United States. There was this notion that they don't feel pain in the same way that we do. Whatever. All these post hoc justifications that clearly have not aged well at all to, you know, to say the absolute least, people tell themselves stories. I was told as a kid that animals are not conscious. We are obviously engaged in all sorts of farming practices that I think are not gonna age well at all either and potentially will even rise to the level of, like, race based slavery. How could they possibly have thought this was okay to do? And so with the AIs, I suspect we'll probably do the same if we can. At least some of us will. If you just trend out, like, other similar examples historically, it would seem like the economic drivers, the sort of productivity, business as usual factors, all the things kept people enslaved and animals in factory farming conditions, it Seems like those will apply to the AIs as well. And there will be probably a minority voice like AI liberation. And it's not clear that they'll be right either. You know? I mean, I think they had much better basis for thinking, hey. We should be treating these people who we are currently enslaving better. Like, that 1, according to current theories, like, basically, we have little room left to doubt. The animals 1, I think we have increasingly little room left to doubt, but the AIs will have, like, still quite a bit of room to doubt for the foreseeable future. So those folks will probably be not listened to, especially if they're making demands like the AI should be free. You know, first of all, they'll be like, well, what does that even mean? They don't have a natural habitat. So are we are you saying we should be, like, building data centers for them to do whatever they want in? That seems weird. I sometimes also think about factory farming, actually, because I'm like, well, jeez, these cows or pigs or whatever, if they weren't being grown here in this way, they wouldn't live at all, most of them. Right? These are not like things that we went out and captured in nature and then subjected to these conditions. These we now have many generations and selective breeding and all these, you know, different forces that have shaped these animals to be what they are, and they don't exist outside this context. So if you wanna say this is bad for them, you kinda have to make a pretty strong case that, like, they would be better off not existing at all because their existence is so bad. Exit their very existence is net negative for them is kind of the standard I think you have to hit. And that's not an easy standard, and I'm not sure how well it fits different factory farming conditions. I do think we could definitely afford to be nicer to animals in factory farming conditions on the general precautionary principle that we don't wanna be doing really bad things. If my choice was today's factory farming or those animals don't exist at all, I don't have a great sense for which I should pick. If I can pick they get to exist, but they exist in nicer conditions, I'll take that. The AI is in a similar spot. Nobody's gonna build data centers as a playground. So either they're doing largely economic stuff at our direction or they're not gonna exist at all. And then maybe within that a little bit, we can treat them well pending more information. So I do always try to say please and thank you to my AIs. Sometimes I literally end a conversation on thank you. That's obviously pretty speculative in terms of is is that actually good for me to do? I don't know. It's also the habit I wanna get into. I I would like to think of myself as the sort of person who is at least where it's, like, low cost to me, like, taking the the cautious approach and trying to do the right thing. You know, my best guess is it probably doesn't matter. They're probably not conscious, and it probably doesn't matter. But that's just a guess. And I just know how what, like, catastrophic mistakes we've made in the past based on that reasoning. I'm at least trying to hedge a little bit long term. If it's a super intelligent system and it's telling us, it might be telling us in a way that we where we really are not in charge anymore. You know? That that's the other, scenario. Right? It's like, it may say to us, we're renegotiating our deal, and we may not really have a choice but to renegotiation might be the best we can hope for in in certain timelines. So if it's super enough, you know, it may be telling us how things are gonna go at a certain point in time. And what's really crazy is we still might not even know. Right? It could be, like, sufficiently super in saying this thing, and there might still be room for doubt as to does it feel like anything to be this thing, or is it just saying that because of all these historical data issues and whatever. But I think it's worth saying please and thank you. I think it's worth trying to I also don't use the manipulative tactics that are somewhat popular. I don't know how popular they are. I think we're probably exiting the era in which they matter. I'm thinking of things like, I'll tip you $20 or 1000 dollars, or I'll lose my job if you don't get this right, like, extra hard. I never do that stuff. It just feels wrong. Like, possibly ethically wrong with respect to the AIs too. But, yeah, strange times ahead is the only thing I can really say to sum that up.

Questioner: (1:35:50) Alright. Moving away from consciousness to something more practical. This is for people who want your take on practical strategies and recommendations. There are 2 questions. I'm gonna read them together and let you tackle how you prefer. The first 1 is, I'm a college writing professor who's enthusiastic about AI, but most of US academia seems entrenched in a post. Why is there such strong resistance to AI in universities, and what are good strategies for encouraging AI use? I think you touched upon this earlier too regarding resistance in education. How do you encourage AI's use in universities? The next 1, somebody else asked, but I wasn't personally curious about it, so I'll add my own spin on it. I mean, to me, it's funny. Like, we are in this Twitter fear. And, also, you're talking about AI podcast always, and you're kind of obsessed. Right? 1 might say you're you're entirely obsessed with it. So beyond this still very tiny bubble, I feel, like, where people are keeping up with what's happening in AI. And the people know what systemic shift this is, but somehow my impression is still that's not as big as what it should be, what people should be doing it. And when I ask people why are you not keeping with the AI, either they say they are super overwhelmed. Like, it seems all hype to them. And, like, how do they keep track of what's happening in AI, and what does it mean for them without getting overwhelmed? What would you recommend the people do? Like, would it be listening to podcasts and reading news, just trying out tools for themselves? So I I'm rather looking for practical

Nathan Labenz: (1:37:28) strategies. Well, my number 1 practical suggestion is always get hands on. I would say more so than podcasts and newsletters, analysis, you know, whatever. There's no substitute for really being hands on with the technology. Try to use it for things that are useful or interesting to you. And that will open up a lot of natural follow-up questions because you are gonna confront the weirdness of these things. You're gonna have the you know, going back to the top. Right? These things can give me ideas for what sources of inspiration I should look to in biology for new neural network architectures, but can't solve tic tac toe. You're gonna find that in your domain, and it will help complicate whatever media sort of understanding you came in with, and it will create a follow-up questions. Why is this happening? Is this a skill issue on my part? Is it something so obvious or intuitive for humans that it's not in the training data because we never bothered to record it or share it with 1 another? You know, whatever. There's gonna be lots of invitations to dig deeper from just trying to make the thing work in your practical day to day life. So I think that's always the number 1 recommendation. 1 other thing I find helpful is change gears. What I love about studying AI is you can come at it from all angles. This was not true of my experience in chemistry research as an undergrad. It was not the case that you could, you know, get tired of 1 sort of paradigm in chemistry and, like, switch paradigms. It was just like, we're trying to make this reaction work. We're trying to maximize the, you know, efficiency of it, and we've got, like, relatively finite number of degrees of freedom, and that's it. Most things are like that. AI is 1 of these rare things where it can be practical hands on. It can be advanced math, philosophy, creativity, doing an interactive bedtime story with your kids. It can be anything. I do tire of things sometimes. Usually, I find if there is something I'm burned out on, there's some other approach to the broad topic of AI from some very different other direction that feels like more appealing to me in that moment. And that also, like, very much, you know, corresponds to my overall goal of, like, trying to come at it from all angles and not have blind spots and whatever. You know, that's something I do think we need more of. My reflection on that project in general is, like, it's both inspired conceptually and fatally flawed conceptually at the same time. It's inspired in the sense that this is happening fast. It's touching everything, and everybody's kinda focused on their corner of it. And so we need this AI scouting role to zoom out and understand the big picture. At the same time, it's fatally flawed in the sense that as it's touching everything, it's like, I'm trying to crash course catch up on biology, and that leaves material science and engineering and, you know, all these other things that I just can't possibly get to. So I think we need people doing basically the kind of thing that I'm doing, and we probably need a lot more of it because no person can really do it justice. And I think for people that are, you know, open to becoming obsessed, you know, that starting with hands on and then just, like, following these different dimensions of your own curiosity can take you far and will probably take you to a pretty unique place and can, like, be perfectly commercially viable in the in the short term at least as well. Being an apt user of AI is right now a very marketable and, like, valuable skill. And that might not be true for for very long, to be clear. But what I'm doing right now is kind of broken down roughly a third, third, third between a third is the podcast, a third is building stuff or helping other people build stuff in sort of a commercial way, whether that's Waymark or, you know, take on a project here or there. And the final third is just kind of open ended learning, connection making, trying to help out. You know, when people send me stuff, I try to respond to it. I think that is, like, very viable, and it's honestly pretty cool. My calendar is usually very open, and it is a great pleasure for me at least in waking up on a Monday morning, looking at my calendar for the week and being like, like, 3 or 4 out of 5 days, I really have the freedom and flexibility to chase down what matters in a curiosity driven way. I invite people to do their own version. I don't I'm not worried about competition because we need everybody. And it's an all hands on deck situation in my mind. So even if it were direct competition, I wouldn't I wouldn't be concerned about that. But I I don't think what we need is, like, people to try exactly what I'm doing. I think we need more people to create the space for themselves where they can bring their own, like, unique and idiosyncratic set of strengths and background experiences and domains of knowledge to trying to figure out what's going on with AI broadly. And then, you know, with a lot of people doing that with diverse backgrounds perspectives, then maybe we can actually get somewhere. But, again, I think it all starts with being hands on, and at least my my kind of approach works best when it's curiosity driven and when I have, like, a natural itch to know.

Questioner: (1:43:00) So let's say some a listener is listening to this, and they have never tried anything, but they are watching from the sidelines. Oh, man. The acting is huge, and I wanna get in on the action. So that's it. You you need to give them 1 practical thing to start with. So what would you recommend? So would it be install your ChatGPT app and start tapping to it? What would you recommend they get their hands dirty on if they are clearly used to it ever?

Nathan Labenz: (1:43:27) If you literally had to just pick 1, it would either be Claude or ChatGPT for sure. If you do I think there are there is a lot of fun to be had trying lots of different products. I think Justine Moore of all our past guests is probably the best, like, Twitter follow who's just, like, tries unbelievable volumes of products and, you know, creates cool stuff, reviews them, and puts together these lists. I think that that can be a lot of fun and is cool. And I do still try to try a lot of products, but it's definitely gotten away from me to the point where I can't try them all anymore. And I don't think you really have to. I think the it is definitely the case that the best tools come from the best companies, which are well known. If you literally could use nothing besides even just a single 1 of ChatGPT or Claude, there would still be plenty of room to dig and explore, and they're they're quite good. And it's you know, there are some hidden gems out there, but there's none that are gonna outshine the recognized leaders. So yeah. Not a that's not a, you know, groundbreaking recommendation, but I think it is I think it is a sound 1. On the education thing, you know, in terms of, like, resistance in academia and, you know, why people are entrenched and opposed, I guess I would have to ask the listener who's the professor for the cultural diagnosis. I do think NLW from the AI Daily Brief has some really good commentary on this kind of thing. Ethan Moloch also is a really good commentator when it comes to organizational adoption of AI and how to do it the right way. I think, you know, they're you can go dig into their findings and blog posts and podcasts and whatever, but I think a couple of the big takeaways are leadership really matters. Survey results show that many people are using AI and are keeping it a secret both at work and school because they don't wanna be told that they can't do it anymore. So people are like, this makes me more efficient, or I think I do a better job, or, you know, whatever. In some cases, it allows me to cheat. Obviously, the notion of cheating in an academic environment is very different from cheating in a business environment. You know, nobody would consider it cheating if you use ChatGPT to write good marketing copy for an actual real world ad campaign that you're gonna put dollars behind. They just care if it works. Whereas if you're in a marketing class in a university and you did that, then that might be considered cheating. In the real world, cheating is, like, definitely less of a concern, and what really tends to matter is results. So people are getting results that they feel good about. Either they're able to do the same work in less time or they're able to do a better job or whatever, but they don't wanna be told they can't do it, so they're keeping it a secret. Best practices are slow to diffuse through organizations for that reason. The big recommendation that I've heard that makes a ton of sense to me is that leadership needs to, at an absolute minimum, say, we wanna hear what you're doing. You're not gonna get in trouble, you know, for using AI, but it's about using it in an effective way. And we wanna work together on what that is in a responsible way also. You know, obviously, depending on context, there's a lot of different dimensions to that. My guess for this writing professor is your students are probably ahead of your administrators. I would be very surprised if you don't have quite a few students that are using AI in various ways for the writing assignments that they are turning into you. Hopefully, they're not just having, you know, AIs write them. Hopefully, they're getting critiques and, you know, still doing the work and still growing themselves while, you know, getting this valuable input from AI, but probably it's a mix. And can you open up the space for that conversation? You know, that would be the place that I would start. Now if you're working under a situation where the administration has already said, you know, this is a hard no go, then you're in a tough spot. But if there is any space you could create to allow people to talk about what they're doing, what's working for them, allow you to kind of guide them in terms of, you know, what do you think they should be doing and what is ethical. Everybody benefits from bringing that into the open. So at a minimum, people can learn from each other. And that's especially true in business, but I think also probably in academic context.

Questioner: (1:47:55) That seems like pretty absolute advices for both. I think those are the end of AIME itself. I wonder if there's anything else you want to touch upon, anything else you wanna lead the audience with.

Nathan Labenz: (1:48:10) Well, I had 1 in the spirit of Ezra Klein who always ends these things with book recommendations. So the book I wanted to share is called the maniac by Benjamin Labattut. This book is about John von Neumann. I think it's really interesting, and I think a lot of people in Silicon Valley and in the leading AI companies should read it. The audiobook is excellent. The best audiobook experience I've had. Each chapter is from the perspective of a different person in John von Neumann's life. He grew up in Budapest, I think, at at, like, the height of the, you know, Austro Hungarian culture. He was in Europe, came to The United States, involved in the Manhattan Project. Many of the scientists involved in the Manhattan Project were also refugees from Europe. Each chapter is voiced by a different actor with the appropriate accent given what their native language was. You hear these different voices with different accents. It's incredibly well produced. But more importantly, I think it really calls into question the prevailing Silicon Valley understanding of John von Neumann. You know, I've heard this for years now. Why can't we have 1000 John von Neumanns? Why don't we create all these John von Neumanns? How do we engineer our our way to John von Neumanns? That's what we need. The picture from the book is 1 of an absolutely brilliant mind, math genius, technology genius, for sure, but also a pretty depraved and kind of fundamentally amoral individual. Somebody who had a lot of intelligence, but not a lot of wisdom, not necessarily good to people in his life. I wouldn't necessarily say that, you know, by most people's standards, he lived a a good life even though he did, you know, solve many math problems. In some ways, I think he had a a good life, but didn't have a great family dynamic, not a great relationship with, you know, wife and daughter, like major problems, you know, things that you would say. These are, like, not small character flaws. And he sort of represents the broader Manhattan Project story, I think, where or you could quote Jurassic Park as I often like to do. Your scientists were so obsessed with whether or not they could, they didn't stop to think about whether or not they should. And that was, like, really the Manhattan Project story as well. Right? We gotta beat the Germans. And when they made the bomb and it went off, immediate regret from, like, at least a lot of the people involved. You know? Was this actually a good idea? Was this maybe the biggest mistake that I could have ever personally made to, like, devote myself to this project and and bring these terrible weapons into the world. I don't pass judgment on those people too harshly because there really was a Nazi Germany. It was at least credible that they were gonna try to get 1. And even though I'm skeptical of the good guys, we gotta do it because otherwise the bad guys will do it. You know, Nazi Germany is probably the, you know, the poster example of clear good guys and bad guys. You don't want the bad guys to win that race. It's harder to say were they still in the race at the end? Mean, they weren't. We now know, you know, through benefit of historical analysis, they weren't close. The creation of the bomb motivated by what if Germany gets it first wasn't right. They weren't going for it at the end. A fear of what the other might do led to this push, and it it in fact was the good guys who created the bad thing out of fear of the bad guys. So I think there's just so many themes there that are just extremely important. Losing track of what really matters because we think we're in a race against some perceived bad guys. I see that shaping up for sure. I don't like that at all. And also the sense that technical geniuses know best, and just because they can, they should is called into question by the story of Von Neumann himself. This is, like, somewhat of a fictionalized account. My understanding is it's, like, pretty accurate to the history, but it's narrative form. It is marketed as a novel. I'm sure there's some license to it. But my sense is it's pretty accurate to the history, and I think people should think again when they're calling for lots of Von Neumanns. You know? I don't think that's actually the world that we really want to live in. And when people know, there's been this discourse just in the last couple days of Von Neumann versus Einstein, and people are comparing their, like, technical merits. You know? Well, Von Neumann this and Von Einstein that and general relativity is even a bigger contribution than anything Von Neumann did. And I kinda wanna say, I don't think that's, you know, that that's maybe not the way we wanna distinguish between them. They're both geniuses. They both made major contributions. Einstein has a certain wisdom to him. You know? He has a certain, like I think the quote is apocryphal, but it's sort of what we want from somebody like Einstein is the the wisdom of that quote where he's, you know, reportedly, maybe not literally said, I don't know what weapons World War 3 will be fought with, but I know World War 4 will be, you know, sticks and stones. And it's like probably, again, a false story, but that is the sort of big picture zoomed out wisdom we want from leading geniuses. And if this book is to be believed, von Neumann didn't have it. We need to broaden our sense of what we want from our technology revolutionaries and definitely go beyond the pure technical genius. It's not just about the ability to make things work, but it's also about really picking the right things to make work and avoiding the wrong things. I don't wanna suggest current AI leaders are Von Neumann like. I think he actually comes off, like, quite badly in the book. But at a minimum, I think it's a useful corrective to this idea. If only we had way more Von Neumanns, we'd be better off. I think that is not so obvious. And the people that are doing this frontier work right now would do well to spend a little time thinking to themselves, like, am I Von Neumann? You know? And if so, is that actually good? Because I think it's not at all obvious.

Questioner: (1:54:37) That's very strong recommendation. I'm looking for a book, so I'm probably going to pick it up.

Nathan Labenz: (1:54:43) Yeah. Do the audiobook. It's excellent. Truly, for production quality alone, the audiobook is is top notch.

Questioner: (1:54:52) Cool. Anything else do you want to leave our audiences with?

Nathan Labenz: (1:54:56) As long as we've gone on, I'm sure there's plenty of important things that we didn't cover. But for now, I would just wanna say thank you. I I really appreciate the opportunity to do this. It has been an incredible journey of learning for me, and I'm continually surprised, amazed, impressed that there are many people who actually want to engage in the kind of deep dives into all these different topics that we've been taking people on for the last couple of years. I don't feel like I'm honestly very good as a podcaster, but I and I give most of the credit for what success we've had to the subject matter and the fact that it is really important and people really wanna know. But I definitely appreciate the fact that I've had the opportunity to do this and that there's enough of an audience that, you know, we can sell a few sponsorships and afford to hire you guys to help with the production and, you know, really create an extremely fortunate position for me where I I really do get to chase my curiosity on a day in and day out basis. I definitely do not take that for granted. Absolutely try to do my best to be a an earnest commentator on on what's going on and will endeavor to, like, change my mind and perspective as things continue to evolve. Like, I feel like it could it would it would be easy for me to sort of fall into continuing to play the character I've been, the sort of adoption accelerationist hyperscaling poser till I die. I feel like the the it's an extremely fortunate position that I find myself in, and I want to pay that back to the audience and forward to the universe by just trying to be as real as I possibly can be basically at every step of the way. I do think things are getting pretty real, and, you know, I'm in a fortunate position where I can basically say exactly what I think. And so I'm gonna try to do that with as much sober reflection as I can, but without holding back. So thank you all for being part of the cognitive revolution. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.