Full Transcript

Transcript

Transcript

Nathan Labenz: (0:00) And then we've also seen the Microsoft Research paper in which they basically said that GPT-4 is appropriately considered an early AGI. Can a language model do the entirety of this job? You will almost always conclude, no. But that's going to be 5% of cases, I would guess, in the early going. And then before too long, it'll be 1% of cases. And then before long, 0.1% of cases. Everybody who's ever wanted to take piano lessons and didn't feel like they have time might have time. The AI might give you the piano lesson, by the way, as well. But there's not necessarily money waiting for you to perform at the end of that. I know this thing can answer my medical questions, but it's not allowed to. And it's not allowed to because of who? The top comments that I saw were all women saying, maybe at least ChatGPT will listen to me when I go in and talk about my problems. So, hello and welcome to the Cognitive Revolution where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my cohost, Erik Torenberg.

Erik Torenberg: (1:27) Before we dive into the cognitive revolution, I want to tell you about my new interview show, Upstream. Upstream is where I go deeper with some of the world's most interesting thinkers to map the constellation of ideas that matter. On the first season of Upstream, you'll hear from Marc Andreessen, David Sacks, Balaji, Ezra Klein, Joe Lonsdale, and more. Make sure to subscribe and check out the first episode with a16z's Marc Andreessen. The link is in the description. Let's get into it. So in our last GPT-4 solo episode, we talked about how you told the OpenAI team that there will be economic transformation, and you are confident in that. Why don't you describe or unpack what you mean by that?

Nathan Labenz: (2:13) Yeah, sure. I think there's a lot to be still figured out in terms of the details. Maybe just to recap real quick what I did and sort of what my takeaways were from it. I just set out to get the AI, which is now the model we now know is GPT-4, to play the role of a lot of different professional advisors that we interact with in daily life and that we pay real money for and see real value in and that people work hard to get to these positions where they're qualified to play these roles. So the obvious kind of first one would be a doctor. You have a concern. You want to talk to a doctor about it. In today's world, you have to set up an appointment, you have to drive in there, you have to wait in the waiting room. It ends up often taking half a day to do that. A lot of people feel like even when they go to all that trouble, the doctor doesn't necessarily listen to them as well as they might like. But that's kind of the state of the art. Setting up a dialogue with an AI doctor was trivially easy for me to do. I used just simple kind of role casting as it's called in the prompt engineering world. Tell the doctor or tell the AI that it is a doctor, that it's going to interact with a patient, set up a dialogue, and then just chat back and forth. And I really found it to be extremely easy to essentially replicate the interaction that I had had with a human doctor. Now I'm not a doctor. Evaluation on this stuff is really hard. And so I kept my experience pretty few in number and focused on experiences that I had had, fortunately minor for me, medical issues or things that we've been concerned about with our three-year-old that we might have otherwise called a pediatrician about. Just within that two months, we did get to a point where there was one incident where we didn't call the pediatrician where we otherwise would have because we got some advice from the AI and we deemed it good enough, reliable enough to kind of assuage our concerns. And we went on with our day without even actually calling the pediatrician. So just in that two-month window, knowing that it's an alpha, knowing all the difficulties around evaluation, we personally got to a point where there was a clear moment of substitution, where a real question that we had, the AI was able to answer satisfactorily enough, we didn't have to call the doctor. We didn't have to go in. Now it wasn't a big problem. Turned out it wasn't really a problem at all. So that's maybe an easy case. But that was a call that otherwise would have been made, and it wasn't as a result of our access to GPT-4. So you kind of go down the line of a lot of different sorts of professional roles. And I think I told the story last time of the dentist, where I had the weird thing on my tooth, and it gave me this feedback that's like, sounds like your dentist did something nonstandard that is not in line with best practices or standard of care. And sure enough, that was definitely true. Tried it in an immigration law situation, did some kind of far out things like playing with a three-year-old, giving tech support to my grandmother, asked the AI at one point to be a mediator between two neighbors that had a dispute over a fence. I played the role of both neighbors that were kind of at odds with each other, and the AI's job was to bring us together and try to find some common ground. Did a pretty admirable job on that. Interestingly, I sent it to a friend of mine who is a lawyer now and asked what he thought about it. And he said, well, from a legal perspective, this is a pretty open and shut case. The one neighbor's fence is on the other neighbor's property, and the neighbor whose property it is can say what happens, and that's that. But I thought it was actually kind of revealing in a way too, that maybe we didn't actually want a lawyer for that situation. What we needed was the mediator, not the legal perspective, because we're going to continue, this is all fictional, but we're going to continue living next to each other as neighbors, and it really helps if we can resolve this in a non-legalistic way. So what role you ask the AI to play can make a world of difference, I think, in terms of how effective it's going to be for you. Other experiments that I did were setting up a personal virtual personal trainer, set up a group chat type of dynamic and then brought in the AI to be the coach. So we've got me who was in okay shape. We've got my wife who's pregnant. We got my brother in there who is probably in the best shape of all of us but frequently injured. And it's doling out specific advice to each of us. Like, Nathan, you should see if you can do five more next time. And Amy, well, you're pregnant. You just got to take it easy. Anything you can do is a bonus. You just got to focus on staying well, getting to the end of this thing. Very kind of friendly. Honestly, people talk about the Turing test for a long time. If you double blinded something like this and ran a study of which of these is the real virtual trainer by text versus the AI, you're not going to distinguish those. So giving gifts, coming up with gift ideas, generating grocery lists, saying, hey, I want to cook this. Can you pull me up a recipe? Translate that to a shopping list. We can put that in the cart. It just goes on and on. I called a virtual car repair place and told them what's going on with my old car that had a problem with it. I would say better service from the AI in the role of answering the phone at the garage than I would expect to get calling a real garage. I mean, patience and just kind of the willingness to listen to you and really sit there and get to the point where you're happy with the conversation is a huge strength for an AI system compared to the guy who's at the garage who's not getting paid to talk to you on the phone. He's got cars there that he needs to fix. He's probably got grease on his hands, whatever, he's trying to talk to you. We've all had that experience where the guy's just like, look. Bring it in if you want to bring it in. I can't really diagnose this over the phone. And you're kind of frustrated, and he doesn't really want to deal with that. If the AI doesn't care, it will sit there and answer your questions all day long. It did in that moment remind me that, hey, really, you should bring it in, and we're here for you, whatever. This is all without any fine tuning. All I'm doing here is setting up your job is to answer the phone at a car repair shop and help people diagnose what's going on, give them a sense of what to expect, and then obviously, ultimately, they're going to come in and we'll get it fixed. Hardware store associate was another one that I tried where I have an old house, and we've got these old light fixtures in the basement probably put up in the eighties. The lights are old halogen lights. They're energy hogs, whatever. So looking at potentially upgrading those, it's a very idiosyncratic thing. These particular lights with these particular kind of wirings, what can we do about that? Do I have to rip the whole thing out, or can I just replace it? It suggested a specific light bulb, which I was then able to search for, find that that light bulb did exist, that it is the right light bulb to kind of convert these old fixtures into a modern low energy lighting. Order them, put them in, and they fit. So it's like, wow. That's amazing. Just for me setting up my situation exactly as I would do if I walk into a hardware store. Start with a story. Got an old house. I think the guy put it in. Same exact dynamic. I was looking for a solar panel just to have a little bit of energy generation backup in this scenario that my power goes out because we had a big power outage during this time of red teaming. And I wanted to be able to charge a cell phone. So I had to figure out, well, how much solar panel do I need to charge a cell phone? Well, how much energy does a cell phone take? So go through that whole conversation. Get a consult on what kind of power generation is enough to charge a couple cell phones. Like, what would it

Erik Torenberg: (10:39) take if I wanted to

Nathan Labenz: (10:40) Run a refrigerator off of it? What would it take if I wanted to run an air conditioner off it? Spoiler alert: you don't want to run your air conditioner off a retail solar panel. But cell phones take very little power. I was actually really amazed to learn how little energy it takes to charge and run a cell phone. Menu planning, catering. My wife puts on these huge events, a thousand people. It's typically an all-vegan menu in the community that she serves, and she's had a lot of problems getting—and they do them all over, right? So it's different cities, and often there's a new catering company that she's working with. A lot of challenges in getting a good vegan menu out of a typical catering company that doesn't often do full vegan menus. I sent her an example: here's a menu setup for your three- or four-day event. She said GPT catering is amazing. It's better than all of our caterers at setting up the menu. So it goes on and on. There were a few that I tried that were definite fails at the time, and I would have to go back and verify to what degree this has been patched. But the math ability was still pretty limited. It could do your SAT problems, your high school level story problems, pretty consistently and pretty well. But if you went up a level beyond that, it would start to be iffy, like college math. Yes, I can do some calculus and stuff, but you would hit the limits. And then, especially as a teacher, there were challenges. If you got confused as a student, then a lot of times it might also get confused and think that what you said was right. We've seen these sorts of things with Bing, where it gets very out of sync in terms of what day it is. And I definitely experienced some of that kind of stuff with math, with chemistry as well, like balancing chemical equations. It could do basic equation balancing. But if I posed as a student and said, here's what I think it is, then I found that it was pretty easy to confuse it still. So in the end, I probably did a dozen of these different roles in some depth, and they were a pretty broad cross-section of the jobs in society that range from literal MD or lawyer—advanced degree, high salaries, high prestige professions—to things where there is a lot of idiosyncratic knowledge, but it's not necessarily so high status, like a hardware store associate or somebody taking a call at an auto mechanic shop. And just across the board, it seemed it did very, very well, with just a couple exceptions that I mentioned. So this has all been really borne out now by publications that have come out as well. We've seen the GPTs are GPTs paper, which is—the second GPT there is general purpose technology. And then we've also seen the Microsoft Research paper in which they basically said that GPT-4 is appropriately considered an early AGI. And I think they've characterized it at quite a bit of length there too. So I would definitely recommend both of those papers for just further characterization of what the model can and can't do. All of that is just kind of raw material, right? This is what we saw. This is what has been observed. Now, how does that lead to economic transformation? On one hand, it's pretty obvious. At least for me, if you just ask your gut: okay, I now have an AI that can handle a first conversation with a doctor, a first conversation with a lawyer, and go 10 rounds and have some real depth and substance to it—does that feel like it would be transformative? Yes. To me, it definitely seems pretty obvious. But then you still have the question of how. One of the lines in the GPTs are GPTs paper that is really true is that if you go around and look at jobs as they exist today and you ask, can a language model do the entirety of this job, then you will almost always conclude no, that it can't. And that could be for multiple reasons, right? There could be too much context required. It could be that there's a physical nature to some of the work, or all of the work, that the language model, obviously without a robot to control, is not going to be able to do. So it's almost all jobs you'd look at as they are bundled today and say, yeah, language model can't do that. From that, though, I infer that what we're about to see is a huge unbundling of jobs into tasks. And I think that that is basically the same lens that the GPTs are GPTs paper takes, looking at what are the tasks that constitute a given job, how prevalent are they, what is the task mix for any given role, and then which of those tasks can a language model either do outright or accelerate. I think they said, you know, cut the time by at least half that it would take to do that task. And there they find that a lot of the tasks are doable. So they categorize jobs by how exposed they are to language model impact, where they define exposed as what percentage of your time is spent on tasks that can either be, again, done or greatly accelerated with language models. And there they find that about half of all jobs have about half of all tasks exposed to language models.

Erik Torenberg: (17:04) Omneky uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omneky so much that I invested in it, and I recommend you use it too. Use CogRev to get a 10% discount. So there's still a

Nathan Labenz: (17:24) lot of wiggle room in there to figure out exactly what happens and exactly how things play out. But I think we're going to see—and I think it's likely going to start with major corporations because they are the ones that are going to be buying the most advanced systems. They're the ones that are going to be seeing huge dollar sign savings opportunities in fine-tuning the most advanced systems for reliability to do things exactly the way that they want them to be done or need them to be done in a corporate context. And so I expect that you're going to see just tons of, first, large businesses, and then it'll probably trickle down to smaller businesses looking at: what is it that we do around here? And you see a new kind of Taylorism for knowledge work. So instead of—we've seen this once before, with the transformation of artisanal manufacturing to assembly line style manufacturing. If you go back before Henry Ford and before the assembly line and look at how people made cars, they were making cars before there was an assembly line. But the tolerance, machining was not so precise. The tolerances were all much looser. People had to make things fit in a very artisanal, hands-on, runtime, problem-solving sort of way to actually get a car out of a process that would work. And so every one that they made tended to be a little bit different. You'd have less reliability, less power. You could not get the same kind of precision. Well, now you go into a factory and you see a high level of automation. You see extremely tight tolerances. In the most advanced, go to a semiconductor fab, and a speck of dust can throw the whole thing off. And so it's been really studied, broken down: what are the tasks, how precisely can we do them? And that's totally transformed manufacturing. I think a very similar phenomenon is coming to knowledge work, where people are going to say, okay, let's look at this job. Let's figure out which parts of it can be done by a language model, and then let's train a language model to do it. Now, sometimes they might even just be able to do it off the shelf, but a lot of times there is going to be some special training required to really make sure that you're getting the—especially the reliability of the output that you want. So people will really spend the time and put some elbow grease in to ensure that the language models are reliably doing the tasks. And hopefully people will take care that they're robust to weird scenarios and all that kind of stuff. But honestly, you only need to get to a certain threshold of reliability before it makes sense to say, all right, look, we pay people X dollars an hour to take phone calls. And if we have a language model do that instead, then we probably see at least a tenfold reduction in cost, 90% reduction in cost. And often, maybe a 95%, 98%, 99% reduction in cost relative to what the human would have cost. In the doctor scenario, if you figure whatever average appointment is $100—that's probably conservative in the US—you can have that 45-minute conversation for basically a dollar at current prices. If you're talking about a call center or customer service sort of thing, maybe that difference is not 100 to 1. Maybe it is more like 10 to 1. But it seems like many things are going to be 10 to 100 to 1 cost ratio. And that's going to be very enticing, especially when you consider the fact that it's also going to enable 24/7 access. And especially as tooling and systems, databases, memory types of things get built out, then you also won't have the problem that you have so often today where you call and the person doesn't know who you are. You didn't talk to them last time, so they have no memory of this, and the notes in the system often aren't that great. All that stuff is going to get smoothed over as well. And you end up pulling these things apart. So what parts of your job can be done by a language model ultimately likely do get sent that way. The parts that can't, then they maybe get rebundled into another job. For a while, if you take the call center type of situation, escalation may be the thing that is the most common thing that the language model can't handle. If you've ever said, is there anybody else I can talk to? Can I talk to a manager? I'm not happy about this—then that type of thing maybe is, in the first generation, where the language model responsibility ends. And it kicks up to a human who maybe has more discretion, more context, whatever. But that's going to be 5% of cases, I would guess, in the early going. And then before too long, it'll be 1% of cases. And then before long, it might even be 0.1% of cases. So I think people get tripped up a lot of times when they think about this future, and they're like, well, there's no way that AI can do everything. You know, we'll always need humans. And I think that that is true, but doesn't quite mean what—or at least it's true with the current technology. Right? I'm bracketing no further major upgrades. I'm not trying to analyze GPT-5, 6, 7 here by any means. But even with just the current technology, we'll always need humans to—yes. But what the humans are doing is probably going to shift a lot. There's going to be a flurry of activity analyzing jobs, pulling things apart, figuring out what tasks can be delegated to AI, figuring out the training set and the reinforcement that can get good performance out of them, and then figuring out how to wire that into a system, how to handle escalations when they're needed, how to handle whatever other corner cases. And then the things that are not doable, those maybe get rebundled into other jobs. So you think about going to a doctor's office. Today, you might have the same person answer the phone, schedule your appointment, and then also take your weight and your blood pressure when you come in. The scheduling, I think it will be months at most before there is a competitive market in automated scheduling systems for doctors that you can just call and talk to. And now everybody wins, right? The patient gets better service. You can call anytime. The doctor's office is going to save money on that. In theory, that gets passed through to the customer. All great. Well, what about that person who was sitting there taking the calls and checking people in when they get there? Well, it probably depends. It depends on a lot of details, but they don't have to take the calls anymore. Is there still enough of a job left to merit somebody to do the weight and the blood pressure? Maybe the doctor picks that up in certain offices. Maybe if it's big enough, you had three receptionists before that were also doing that, and now that goes down to one. The details, the context, and all that kind of stuff are really obviously going to matter a lot. But people are very good at figuring out—certainly, it's what entrepreneurs do, right? Identify opportunity to use new technology to solve problems in dramatically more efficient ways. And I think that's going to happen extremely quickly and will be everywhere. You could say, does this mean there will be no jobs or that people will have nothing to do? And I wouldn't jump to that conclusion. That's another thing where I think there's a fallacy that's like, it's not all or nothing. It's not that AI is either going to complement people copilot-style, or it's going to replace them, assembly line, replace artisanal manufacturing. It's going to be both at the same time. And similarly, what are people going to do, I think, is a much more nuanced and contextual question than saying nothing's going to change or everything's going to change and nobody's going to have anything to do. The outcome is going to, in all likelihood, involve a ton of change. But the end state is not going to be on one of these poles. It never is.

Erik Torenberg: (26:39) Yeah. It reminds me, you and our friend Antonio Garcia Martinez had a back and forth on Twitter the other week, where he posted some of the stats around how every technological innovation where people were concerned that it was going to lead to the end of certain jobs. It did in some instances. You copy and pasted the farmer stats where the country used to be 90% farmers, and now it's 3%. The technological innovation ended up creating more jobs in general because human desires and needs are infinite. Do you think that is an unhelpful paradigm for thinking about what we're seeing here with AI? Is this time actually different relative to all the other previous technological advancements? And what types of people will be the farmers that we'll be looking back on? How would you expand on some of those ideas?

Nathan Labenz: (27:36) Yeah, it's a really tough question. So I should say I think I have a read on where it's going, but certainly I also expect to be surprised by plenty of things. I do think this time is different in that it's not really clear what we could shift everybody into. And I also think there's a generational question, which is a big one. So I would cite Yuval Harari in Sapiens, I believe, or his next book after Sapiens. And it's funny, people hate on that book, but I think there's a lot of value in just trying to zoom out and tell that story from as far removed as possible. Obviously, you open yourself up to being criticized for missing lots of important aspects of the story when you try to do that, but I find it to be a pretty useful attempt to cover 100,000, 10,000 years of history in just a few hundred pages. And I think it was from him that I heard the general notion that we can do a couple different general kinds of tasks: physical labor that relies mostly on our muscles and then cognitive labor that relies mostly on the brain. And obviously, there's some overlap there. But there's not an obvious third place for us to go. So when physical work got semi-automated through machines and the harnessing of fossil fuel and electrical power, then it was like, okay, cool. We don't have to do as much of that stuff anymore. We can go do more cognitive work, and that's great. Largely, people prefer it, and there's certainly a lot less injuries from it and whatnot. So it's a win. But if you zoom out to that level, where are we going to go beyond cognitive work? And you could maybe come up with some candidates. Well, what about emotional work? Is that distinct enough from cognitive work? My experiments suggest that the AIs are getting pretty good at that sort of emotional work as well. I don't have a lot of experience with cognitive behavioral therapy or that kind of modality, but it seems like that will be pretty readily provided by language models. Is there some sort of connection, the sort of realness of which can't be recreated? Possibly, in some scenarios for some people, I could see emotional work becoming a category that's distinct from cognitive work and kind of remains a place where humans are maybe not even dominant, but preferred for reasons. I think you could also imagine an interesting local bespoke service economy, highly individualized entertainment. People do these dinner party murder mystery games, for example. Could an AI put together a good murder mystery for a group to solve? Yes. I don't think that's out of the range of what it could do. But I could imagine that there's, in a world of abundance and all the bullshit jobs and frontline customer service and all that kind of stuff being delegated to AI, then maybe we end up in a more local network sort of economy where we're kind of doing cool, fun things with and for each other. But honestly, it's hard to... if you think of work as stuff that people don't necessarily want to do, and therefore they need to get paid to do, and it won't get done otherwise, it seems like that's a shrinking category. It seems like there's gonna be a lot less work that only humans can do that they won't do unless they get paid for that won't get done otherwise. That seems like a shrinking category. So I do think people will have plenty to do. I don't worry about myself at all. If everything I do got automated, I don't think I'd be bored. I'd still be interested in studying AI, for example. Or I'm interested in history. I'd love to read more history books than I currently feel like I have time for. So I think there is tremendous potential for people to be self-actualized, to develop their talents, to create stuff that they find worthwhile and that others find worthwhile. But I don't think that that's quite the same as work as we think of it today. I need to maybe refine this definition a little bit, but I'm kind of liking the people wouldn't, it wouldn't get done otherwise. People don't like to do. They either have to pay them to do it, that only people can do, that there's not really a way to mechanize. Something like that does feel like a pretty good definition of kind of work as it exists today. And I do think that is going to be a significantly shrinking category.

Erik Torenberg: (33:06) Yeah, it is interesting. It does feel like the concept of Muggles from Harry Potter almost might emerge in our society and how we think about just the different classes of people and their capabilities. And to some extent, it kind of exists today. Some people can code and build tremendous things, but it's not as... once those people are supercharged, and the other people are, or many people are basically replaceable, it does feel like that gap in understanding of people is going to widen. Feel free to challenge any of those premises.

Nathan Labenz: (33:47) Talked to Omnejad a little bit about the 1000x developer, and that definitely seems like part of the future. It seems like some people are going to be so good with these tools, these programming assistants or whatever, that they likely will be able to accomplish what a full team can do today. And I do think it's true that there's a lot more demand for software, especially at extremely low prices relative to the software that's being created today. But again, I don't know that that's infinite. Right? I mean, we can only... What sort of software is there infinite demand for? I guess, video games. Just software that entertains on some level could be kind of the end state. If there's nothing else to do, then you'll create things to explore and mess around with. But even then, we only have so much time. Right? So you can kind of imagine that at some point, if you're getting to... and by at some point, I don't mean this is a hundred years away. We're talking a handful of years away maybe, where you can sort of have a narrative language model type system kind of generate lots of games or customizations to you from archetypes that have been previously created. And on top of that, you can kind of spin up 3D environments. You can have the language model speak to another specialist model to generate a 3D landscape for you to explore. And you can do all this in kind of a virtual reality or an augmented reality generated for you on the fly. I mean, how much more software do we need beyond that? If you can see a pretty clear path to enough software to entertain people on an individualized level nonstop, I don't know really what more is needed than that. Language models right now cannot do science. So that's kind of, I think maybe a really important threshold that we have not hit yet. We've hit a ton of thresholds over the last 2 years. I can't really write coherent copy. It's not really even useful. It's like a marketing copywriting assistant was 2 years ago, and now we're at closing in on expert level doctor performance. There's a lot of little thresholds that have been hit along the way. We've not yet hit the one where it's like AI can do original science. So for now, that remains a thing. And similarly with hardcore engineering, it's not yet to a point where it's going to set up its own semiconductor supply chain anytime soon. Well, maybe sometime soon, but again, not with this generation. This generation will not do it. Next generation, as soon as that comes, we'll have to reevaluate all claims. So again, there are some things that are not in the province of what a language model can do even with fine tuning. It's just not there yet. But yeah, it seems like a shrinking category, and it seems like something that fewer and fewer people will be kind of the rare specialists that have skills that AI just doesn't touch yet. So I don't have a percentage on that, but it feels like today... how many people are really in semiconductor fabrication? Right? Not that many. How many people are really optimizing at a low level how iOS interfaces with the hardware? Not that many. It's an important job. It's extremely high skilled, and it's not immediately under threat from a language model. But I just don't think there's that many people. The other thing I wanted to mention too from earlier was the generational question. And I think this is really important because this is happening so fast. There's a lot at this point. Right? We grew up in the end of history era, not to encroach on Moe's territory and discussion topics here, but it was assumed for a while that the economy will adjust. And it's fine if... I'm from Detroit. I'm in Detroit. It's fine if the car companies send all the jobs to China or Mexico or Vietnam or what have you because we'll adjust. We're dynamic. Everything's gonna be great. And we ran that experiment for 20 years, and it was a pretty gradual process. A lot of people did not adjust in the way that the textbook economics predicted that they would. Right? There's towns all over the Midwest that are not what they once were because the main employer, the main factory, whatever, is gone. There wasn't a full recovery. There wasn't... there was some adjustment, certainly, but a lot of the adjustment was people left the town or some of the other adjustment was people adjusted to lower standards of living individually. They got jobs that pay less and are lower status than what they're used to, and people are not happy about that. So and that's obviously been, I think a significant force in American politics by any telling. That was a slow process compared to what I think we're gonna see over the next 2 to 5 years. And if you do those kinds of things on a generational timescale, as a society, we would give ourselves the opportunity for, okay, well, yeah, it might suck for this person who's in their forties and they've got 20 years of this experience and that job is going away. To some degree, there's creative destruction, nothing we can do about it. But their kids can come up and kind of adapt to a different reality and they'll be educated in a different way and they'll prepare for other jobs, and they'll get those jobs. So I think society... I guess what I'm trying to say there is I think society historically has been more adaptable than individuals. And a big part of that, as I understand it, is the generational change that you can educate the next generation for the opportunities that will exist. The previous generation maybe doesn't catch up, and that sucks for them, but that is kind of something that we've tolerated. If that happens to a huge section of employed people over just a couple of years across many sectors all at the same time, then again, I don't really know how we adjust to that. And that's not to say that people would be unfulfilled or wouldn't be incapable of finding things to do that are self actualizing. It's just not clear to me that there's going to be a lot of demand to pay people to do that sort of stuff. Everybody who's ever wanted to take piano lessons and didn't feel like they have time might have time. And the AI might give you the piano lesson, by the way, as well. But there's not necessarily money waiting for you to perform at the end of that. So you can kind of play that out across a lot of different things. People have a lot of hobbies. They have a lot of interests. They have a lot of ways that they would like to entertain each other and socialize and explore. And it could be a very rich, fun, rewarding life. But it seems like the paradigm of money flowing to those activities does not necessarily cross over as far as I can tell.

Erik Torenberg: (41:42) Is there anything I haven't asked you that I should ask you or any big questions you yourself are having that we have not yet discussed on this topic?

Nathan Labenz: (41:52) The impact of regulation is an interesting one that could shape a lot of how this plays out. I would not expect that we're going to avoid the kinds of changes that I'm describing, but I do think there are a lot of details to be worked out. The timeline and exactly who can do what, under what circumstances, with what licensing and whose say-so is a really big one. Honestly, I've been a little surprised by how slow the interest groups have been in responding to the developments that we've seen. It seems like there's been more denial than actual attempts to do something about it for the most part. But take a couple leading American civil society organizations, the AMA and the ABA, the Medical Association and the Bar Association. It seems to me the leadership of those organizations is going to look at data like this and say, we've got to protect our members' interests here. How do we do that? The obvious thing would be to say, we'll try to make it illegal. It's already illegal to practice medicine without a medical degree, it's already illegal to practice law without a law degree or a license. Maybe we can say it's illegal to use a medical language model without a proper license. Or at a minimum, it has to be under supervision of a licensed doctor or lawyer or whatever. It seems like that should already be happening. Maybe it is starting to happen. We've heard a little bit of noise from the FDA around regulating language models as devices, which means you do have to show safety and it might just be a safety standard. I would have to double check to see if there's also an improvement on previous clinical best practice. There's a little bit of a difference between device regulation and drug regulation, which I don't have full mastery of. But FDA is starting to get into the game a little bit and saying that this is going to fall into a device type of paradigm. Haven't heard a lot from lawyers. I haven't heard as much as I would expect from doctors. But I think that has to be coming. I would be shocked if we get out of this year without the first fights ramping up on who is going to be allowed to do what with these models, who's going to be responsible for supervising, who's going to be liable. If OpenAI is providing something directly to a user, is it their responsibility to make sure that it doesn't dispense medical advice? Is there some sort of reasonable standard where they try to filter, but then if you jailbreak it as a user, that's on you? These protectionist questions are going to bump up against safety questions in weird ways. And everybody, of course, will want to frame their interest. This is how it's always done, right? The interest group will frame it as being in the public safety interest, even though a lot of times it's much more about protecting the interest of the incumbents in a market. So I don't have a lot of predictions around politics. I guess I would be very surprised if there aren't fights about that soon. I would be very surprised if the doctors and the lawyers don't get their way. I would guess that the higher status a profession is in society currently, the more effective it will be in creating restrictions around the use of language models to do their core stuff. You don't see the same level of protectionism or organization in customer service. So the drive-thru at McDonald's will probably be language model powered, and nobody's going to stop that, right? Because there's just not that powerful of a group there and Congress is probably just not going to take it up. But there's a lot of uncertainty there. If it does play out where there is a lot of restriction, then I think what you get is a leapfrog scenario where

Nathan Labenz: (46:42) strategically, maybe because it's the only way it can go, maybe just because the demand is so great and it seems almost irresponsible or immoral not to, maybe these things just get deployed in places where there just aren't that many doctors. So you think, how many doctors per capita are there in the Democratic Republic of the Congo? Not that many. But people do have cell phones, and they do have important questions. And maybe they can get those answered. And maybe that's just where this stuff goes first, right, and works its way up the country ladder by income, more or less as it becomes clear that it works and it's really not in the public interest to restrict this stuff at some point. So eventually that could tip. Is it conceivable that we could get stuck, say in the United States or the West or whatever, in a similar spot where we can't build any nuclear power plants or we can't build a Second Avenue subway? It's conceivable, but I really don't think so. Those are fundamentally fixed location. There's some interesting political science theory around what allows a state to pop up. There's recently a great EconTalk about this, looking at state formation in locations where grain naturally grows versus where tubers naturally grow. And the big difference between grain and tubers from a taxation standpoint is you can store the grain in a central silo or whatever. But a tuber, like a potato, once you take it out of the ground, it rots quickly. You can't store it long-term. So looking back in history at the time where states were formed, they see that there are a lot more state formation in the grain areas because there was something that you could tax. Whereas in the tuber locations, you can't really tax the tubers because they rot, so it's pointless. Those locations maintained more decentralization and less state formation. I think there is something similar here where the AIs are not in one location. It's really hard to choke them off entirely. You know where the Second Avenue subway is, so you can just stop people from doing it there if that's what you want to do as a state. It's going to be a lot harder, just like we have the Great Firewall in China, but there's still VPNs in China. There are ways to get around that. I don't see how, absent very draconian, super heavy-handed measures, I don't see how people are going to prevent individuals from accessing their AI doctor, especially as it might get deployed in other countries. If it's made illegal here and OpenAI is like, well, okay, fine, we're going to respect the law here. But we are going to make a subsidiary in Kenya. We'll have our Nairobi office, and that will be the place where that gets run out of and served out of. People are still going to be able to access that from their devices and their networks in the United States, unless there, again, unless there's a super heavy-handed regulatory regime on that. So it's just hard for me to see how this stuff gets denied because there are so many little cracks that it can get through. People are going to be motivated too, right? I mean, middle of the night, your kid's sick, you're thinking, it's winter, you're thinking, God, do I need to go or do I not need to go? I really just want to talk to somebody. And I can do it at 1% the cost. If going into that thing, people will be pretty motivated to find ways to access these services. So I expect a lot of drama and a lot of battles in the regulatory space, but it seems like in the end, that will be more a question of speed than whether or not people ultimately can consult an AI doctor. It just seems very hard for that ultimately to be prevented.

Erik Torenberg: (51:15) I'm a bit more negative or cynical on what's going to happen than you are perhaps. I think people are going to go full-on panic mode. I think you already see some of this. With crypto, it transcended the tech media sphere into mainstream. But the real concern there was, hey, does this thing really do anything? Are there any use cases? Is this just speculation? Is this just a waste of time, waste of money? AI is similarly broken out of the tech media sphere, but some people will say, hey, this doesn't really do anything special. Even the people building it will try to downplay it. But then I see on my podcast feed every day, a whole number of mainstream podcasts saying, hey, what does this mean? This has already captured the concerns of a media class. And as the thing just gets better and better, I think those concerns are going to get louder and louder. And we already have, say, in education, my belief is that education prioritizes the teachers over the students. And sometimes they use the language of student benefits in order to justify not allowing school choice, for example, when really it serves teachers. That's going to be harder and harder to do as it's clear that AI presents all these opportunities. So they're going to try to use all these benefits. They're going to try to use the language of equity, but AI will be available to all, hopefully. So I think the contrast between supporting the beneficiaries of the services versus the providers of the services is going to become harder and harder to conflate. But still, ultimately, I suspect that those stakeholders, the doctors, the teachers, unions, the lawyers, just have so much power that they're going to cause greater negative influence on this than we would want. Can they materially slow down OpenAI? I don't think so. I think what we'll have is just a continuous bifurcated economy. Mark Andreessen likes to point to that chart where it says, here are the industries that are affected by government, regulated heavily by government, housing, healthcare, education, cost just rises and rises, and here are the industries that are not affected by, or are not regulated by government, the cost decreases, decreases. I just think that that will continue to bifurcate in a way that's unavoidable, and you can't not see it. Anyway, those are my concerns.

Nathan Labenz: (54:04) I think they're going to try. A lot of interest groups will try. That seems unavoidable. But it kind of feels like the rise has been so fast. You can imagine a different world where things were much more gradual in terms of just the rate of capability improvement, and we had more time to sort of say, yeah, this thing could sort of be your doctor, but it's a shitty doctor and it shouldn't be out there. But we kind of just crashed through that world in the last year before anybody even knew what was happening. And so we're waking up collectively to a world where that concern is kind of already over, or it's very soon to be over. There was just a paper that somebody sent me this morning where GPT-4 beat MedPalm, and then MedPalm 2, which also just came out in the last week or two, is neck and neck still seemingly with GPT-4. But it's basically expert level on answering these standardized medical questions. It just happens so fast. It's like the first people are hearing of it, it's kind of already at expert level. So I think you're right to say it's going to be increasingly difficult to do that conflation, and it's going to be more and more obvious that some of these things are just kind of naked power grabs. And then it does remain a question of ultimately political economy. I do think if you had some heavy-handed laws, you could definitely slow things down with enough FUD. You could probably scare off a lot of the consumer population for a bit, at least. I don't even want to say a while, though, because people are already kind of bent out of shape about AI censorship. How are people going to feel when it's like, I know this thing can answer my medical questions in a pretty good, strong, reliable way, but it's not allowed to. And it's not allowed to because of who? Because of the doctors who convinced Congress that I shouldn't be able to get these questions answered on my own terms. I mean, we live with a lot of crazy stuff, so that could happen. But I kind of think those curves are going to bend. I would guess that I would expect convergence. I think costs will ultimately drop even in those government sectors. And for me, it's a question less of can that line be maintained and more like, how fast or slow might the retreat ultimately be? And by the retreat, I just mean ultimately accepting that more and more can be done by AI and at 1% the cost, there's a pretty good rationale for it. It seems like we're headed for just a consistent advance on that front. I'd be surprised if they can hold the line. By they, I mean any of these interest groups that might try to say there should be no direct consumer access to an AI lawyer or an AI doctor. It must be done this certain way. I don't think that's going to hold for super long. But maybe a few years, five years, wouldn't be crazy to me if we were sitting here in 2028 and we're like, because we've done dumb stuff, right? We've put a lot of people in jail for nonviolent crimes. There are certainly moral outrages where it's like, how is it that the little people in our society are getting treated so badly based on some high-level conceptual argument about safety? Now that could happen again, but we've largely backed off of the war on drugs, and people largely seem to agree now that way too many people are in jail. And I think this is going to be even way more obvious in the case of the doctors. Who's really being harmed here? Who's really being protected?

Erik Torenberg: (58:21) It's hard for me to imagine that it goes five years before people have pretty good direct access. That's an optimistic note to wrap on. Maybe I'll wrap here, but I'll try one more line of questioning. Earlier, we were talking about how we're going to need to figure out what people are going to do. It's interesting because over the last decade or two, we've really worried about blue-collar automation, what the truckers are going to do, how they're going to find meaning once we no longer need them. It turned out we need them longer than we thought we would. But we thought of all sorts of things like UBI, and then we said, is UBI really going to fill their sense of meaning? I think it's interesting because to the degree that it's coming for white-collar work, people who work in white-collar jobs are probably more sensitive to feeling useful, to status concerns about their importance, given how hard they've worked their whole lives to rise up these respective ladders. Of course, blue-collar people work just as hard in their own way, but white-collar people are used to climbing up the ladders of success and being told their whole lives how successful they are, how special they are, how important they are. It's going to be a huge shift in morale. Even myself, I've started to feel a little deflated when I see my abilities to synthesize information or summarize or create conversations or have certain analyses on certain things just be dwarfed by ChatGPT. So it's not a question of UBI or the resources as much as how are people going to feel important? How are people going to feel needed in ways that fulfill what they want to be needed for, or what they prepared to be needed for their entire lives, going through all these hoops of school, grad school, jobs?

Nathan Labenz: (1:00:25) Yeah. In some sense, I think it's another way in which this AI technology in general is a great leveling force. There's a definite strain in politics that is resentment of one class of people who feel and say that they got where they are based on merit and their unique abilities and the wisdom of the market that values it highly. I think we're all going to be in much more of a similar boat before we know it. Maybe one optimistic take on that would be that maybe we can all be a little bit nicer to each other knowing that we're all under some similar pressures in that respect. That could be good in some sense. I do think people are going to react all sorts of different ways to it. We've been seeing things like, I just saw something from a 3D artist who was like—in fact, I think you sent it to me—that was like, "I just lost everything I loved about my job. Now I sit here prompting Midjourney all day and it kind of sucks. I don't feel creative in the way that I loved being creative before." There's probably going to be a lot of that. I don't really know any way around it, honestly, at this point. The flip side of that is some people are tremendously empowered by it. For every one person like that, you can also go find somebody else. It's like, "I've always had great stories in my mind, but I was never able to visualize them, and now I can." So I think you do see both sides of that trade-off. Another idea that I've thought is really interesting is this flip from humans seeing themselves as distinct from the animals and it being our justified privilege to rule the world based on our intellect, to now there's the reverse of that, where it's like, well, what makes us special relative to the AIs is that we feel that we have these animal traits that we at least are pretty confident don't exist in the AI as they exist today. So our feeling, our emotion, the sort of realness of that is what some people are now trying to set up as the specialness of humans. But it is interesting that that stuff is largely shared with the animals. So what separates us from the animals puts us in the same bucket as the AIs, and what separates us from the AIs puts us in the same bucket as the animals. We're in the center of that Venn diagram for the moment. But yeah, I think it's going to be tough. I do think there's certainly going to be a loss of sense of specialness to the intellect, the things that we can do. I saw a TikTok of this doctor sitting there at ChatGPT talking to GPT-4, getting an amazing diagnosis, and feeling like, "Bro, I went to med school for four years. This thing just spits it out." And he's feeling defeated, exactly like you said. But what was really interesting to me about that video—I'll see if we can find it, maybe we can put it in the show notes—the top comments, this video went viral, right? So tons of views on it. The comments that had thousands of hearts, the top comments that I saw were all women saying, "Well, maybe at least ChatGPT will listen to me when I go in and talk about my problems." So I don't know. That's tough. You've got the doctor that is feeling like you described, defeated: "I worked so hard. I was on this path to success. Now my status is in jeopardy." But again, you see in that same one little TikTok interaction a real pent-up sense that, yeah, but you're not—not you, this one doctor, but doctors aren't necessarily serving all of us super well. And we do see a lot of upside in an AI that might be a little more patient, that might actually listen to us when we feel like we weren't listened to by doctors before. So we're going to have to find something else. Relationships, obviously, in just about every wellness, life satisfaction type of study, quality, number, and quality of relationships are some of the biggest factors determining how people feel about their lives. It seems like an emphasis on relationship and relationship building is probably a big part of where we might go. That's again connected to my earlier idea about highly bespoke, highly idiosyncratic, highly local custom services. That's light commercialization of just relationship and community. Some of it is maybe paid and some of it isn't. But it seems like there's a lot of potential there for people to enrich their lives, as long as Replika doesn't get so good that that also gets crowded out by language models. And certainly, I think that will be an element of the future too. But I don't feel like I have answers for what happens with society. It just seems like everybody's incentive—we're headed to a new equilibrium where everybody has the same incentive and you kind of can't not do it if all of the other doctors' offices in an economy are now 24/7 scheduling just to take the beginning of the transformation. How are you not going to do that? How are you not going to sign up? How are you going to be the one that's like, "Well, actually, we only take calls 9 to 4, but it's also more expensive. The good thing is it's less convenient and more expensive. So that's why you should continue to choose us." Like, it's just tough. The incentives are all toward rapid adoption. And I think we're going to kind of have to sort out our feelings separately from how the actual structure of how stuff gets done evolves.

Erik Torenberg: (1:07:00) It is interesting. One of the prominent narratives that even people as left-wing as Ezra Klein are promoting is this idea that even liberals, although well-intentioned, are making the government regulations so complex that we have—someone else's word, can't remember who—the vetocracy, where it's just so easy to veto things, to shoot them down. I think we'll see in the next few years, or even much sooner, how much power these organizations, these three-letter agencies actually have. In the face of an obviously, evidently better solution for the customer or constituent, will they prioritize that, or will they prioritize the provider or the group that's providing the service, even when it's so clearly at the expense of the customer? Maybe we'll wrap there. Does that sound like a good complete stop to the conversation?

Nathan Labenz: (1:08:00) Yeah. We'll certainly have time to continue it. And keeping with the general philosophy of this, my note to listeners would be that I think this is an honest take that we really aren't ready for all of the change that is coming at us. Tyler Cowen said, "A lot of copes coming from a lot of directions." This is an attempt to be the sort of cope-free zone and really try to understand the technology on its own terms, what it can and can't do on its own terms, how it is likely to be applied based on the strengths that it clearly has and the weaknesses that it still has as well. That, unfortunately, at this point in time, does not lend itself toward tidy answers or comfortable, confident outlooks. And so we just have to invite you to think about that a lot on your own as well and continue to participate in this conversation. I think that's really all we have right now, just figuring out as fast as we can where things are going in an environment where there is a lot of uncertainty.

Erik Torenberg: (1:09:21) Great to close on. Nathan, as always, thank you for having a wonderful conversation, and until next time.

Nathan Labenz: My pleasure. Thank you, Erik.

Erik Torenberg: Omneky uses generative AI to enable you to launch hundreds of thousands of ad iterations that actually work, customized across all platforms with a click of a button. I believe in Omneky so much that I invested in it, and I recommend you use it too. Use CogRev to get a 10% discount.