OpenAI, Amazon's Anthropic Investment, and the Roman Empire with Zvi Mowshowitz

Full Transcript

Transcript

Zvi Mowshowitz: 0:00 What it's worth and what people are willing to pay are just going to be completely different things. Almost no one I predict is going to be willing to pay more or even vaguely what it is worth. Everyone's going to be remarkably less and I would include myself in that. What is the actual marginal value per month of GPT-4 over the alternatives? It's probably 4 figures minimum. If you asked me this versus having nothing of the kind, would be off the charts, right? 5 figures or more. Am I willing to pay those kinds of prices? Men do not actually think about the Roman Empire that often. We think about bundling and unbundling like real men.

Nathan Labenz: 0:34 Hello, and welcome to the Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my cohost, Erik Torenberg. Well, let's do it. Zvi Mowshowitz, welcome back to the Cognitive Revolution.

Zvi Mowshowitz: 1:00 Good to be back. Always fun.

Nathan Labenz: 1:02 The lull of the kind of somewhat quiet AI summer is definitely over. Leaves are starting to change, and products are launching in rapid succession. So naturally, I reach out to you and get an early copy of your next 50 page blog post that runs it all down in exquisite detail for everybody. Excited to talk about some of the top stories here with you as well.

Zvi Mowshowitz: 1:27 Yeah, it's great to bounce stuff off of people who also are thinking about these things and see how they feel as well.

Nathan Labenz: 1:33 We're both obsessing about it full time. So I think people have seen the headlines. The big stuff that is out, and I could just kind of share a little bit of my experience with some of it over the last few days, OpenAI has certainly gone on a bit of a tear. They have put out, and I'll probably even miss something here, but just in the last week or so, DALL-E 3 is announced and starts to roll out initially to ChatGPT users, which I think is interesting. Code interpreter, by the way, already incredible, gets also the enhancement of the image understanding. So they now bring image understanding also to ChatGPT, and CodeInterpreter benefits from that too. So we're starting to see all these interesting demos of people being like, here's a whiteboard snapshot, and I'm getting code directly from a whiteboard snapshot. Pretty amazing. They've also got voice now enabled as kind of a first class citizen within the ChatGPT app, and that's gradually all this stuff is kind of gradually rolling out over a couple of weeks. So that's quite a flurry of things. For me, so far, the newest, best use has been using CodeInterpreter. Just this last week, I had a React app that I wanted to add a feature to, and I'd literally never coded in React before. But going to CodeInterpreter, I have the new image understanding turned on. And, I mean, it was an unbelievably awesome experience. The first question I asked was just, hey, I inherited this app. I don't I've never coded in this framework. Can you explain it to me? So it did that. Then there were a couple of different patterns that were present in my project that weren't present in its initial explanation. So I kind of said, hey, I've got this Slice.js file and Saga.js file in different folders. And, of course, it's like, oh, yeah. Well, that means these libraries are enabled, and that's what these things do. And this is kind of an extension of the core React framework. Okay. Cool. And so then I was like, can you help me write a command to just print out my file structure? Then I'll give that to you, you can help me navigate further. Yep. Okay. Here's the command. Okay. Here's your here's the file structure back. Okay. Great. It looks like you've got all these different components, and everything's probably working together. And the main file that coordinates them is this. Okay. Sweet. Now I want to add a feature. So it takes me all through that. At some point, it's starting to work, but it's not really working. I've got information missing. And so I didn't even really have to do this, but taking screenshots, dropping in the screenshot, saying, this is what I'm seeing. Can you help me? Oh, yeah. It looks like the information's not there. It might be because it's not being parsed effectively as it comes back from the language model. Of course, I'm adding an AI feature to the app, right? So it's all AI on top of AI. Honestly, it does feel like there has been a pretty significant step change here in the overall value, certainly, of ChatGPT with all these new enhancements. And did I even mention that they have browsing now re-enabled in the last couple days too? I think I even skipped that one. I knew I was going to miss something.

Zvi Mowshowitz: 4:40 They can do, site in and out, hearing in and out, and the web in and out. And it's just a complete sea change. One of the top things that you didn't mention was the idea of designing UI or what the page should look like. I give it to you and it outputs code just like here, it looked like that now. It's pretty wild. Have you been using it personally? The biggest effect to me is it's tempting me to start trying to code, tempting me to create because it just felt like it's so much easier. All of these things are things that you could have in theory found a way to do with more time, but now it's potentially an order of magnitude easier, right? You don't need to see how the thing see the whiteboard and directly output the code, but just the workflow when you can do that is so much smoother and things are so much faster and you can iterate so much better. And that's especially true because we've seen time and time again is the worse you are at the task, the more ChatGPT and other similar tools help you get better, right? So the better you are programming, the more you are like, I'm just going to do this myself. So I have a friend who's an expert programmer, programming for decades, and he said, I can barely get any use out of this. I'm doing all these specialized things that no one else knows how to do, it has no clue, it just causes so many errors, I might as well just code it myself. And for him, maybe that's true, but if I tried to code anything, right, it would be completely neck and neck. And so I'm very tempted, the problem being that I'm tempted by so many other things that I end up making other choices. I didn't make one attempt to use DALL-E to create images this past week and figured out, no, that's not actually a use case here. Was I sort of hoping it could do modifications of images, that'd be your case, and it can't. It instead interprets what the original graph looked like, and then it says, okay, now create something that has the characteristics that I found described in that graph when I asked to what I thought was in it, and then this creates some very stylized thing that's completely different from the original. I was disappointed, but the thing to create them looked really cool, I just didn't have a use for it.

Nathan Labenz: 6:48 So first of all, for your friend, I would maybe challenge their assumptions in as much as you look at an Andrej Karpathy on, just the stuff that he's posting about on Twitter, where he's writing c to execute, open source language models on a CPU. That is to say c code. Right? Super low level, super optimized code. And, obviously, he's a super capable individual with a lot of very specialized knowledge. But still, it allows him to do this kind of thing in a weekend without necessarily even having to brush up on C all that much because it does know the fundamentals. And it also just for me, it's so refreshing. It feels like the way coding you kind of always imagined it would code if you were actually reliable, if we were robust in our execution. Because it's like, it doesn't make these typos. It doesn't do just really stupid type errors or whatever kind of annoying things that I often do. And then I'm like, why is this going wrong? Ugh. And then you feel so stupid. Everybody who spent any time developing has that moment of like, oh, goddamn it. This was so stupid, and I just got so stuck on it for a long time in some cases, right? I mean, we've all had that experience. And these days, I almost I don't want to jinx myself, but I very rarely have that experience because it doesn't make those mistakes, and it kind of knows a lot of these common stumbling points. I even had one recently where my container, my Docker container was somehow messed up and couldn't get an update, and some key file thing was out of date or couldn't be synced. And it was just an absolute nightmare. This is the kind of thing that to Google is hell and to just deal with. I don't want to deal with anything like this. You never want to think about this. So in that case, it was actually Perplexity that solved it for me and just gave me the exact commands to write. I mean, it was to the point where it was almost like, Dude, are you going to just execute this? I don't even really know what I'm doing. But it was all in sort of a container in the GitHub Codespace environment. So I was like, Well, worst thing, I could just kind of throw it away and start over from a backup.

Zvi Mowshowitz: 9:07 Because it surprises you in both directions. Right? The last time that I actually did try to code something, it was consistently just getting the syntax and libraries wrong. And just maybe they changed since its cut off date, maybe something else was going on, but it was just giving me nonsense that obvious, in some cases it just didn't work in practice, in some cases I was like, I can obviously spot this and I'm bad at coding. I just instantly see, oh, you'd love a coding interview if I saw you do that, because it just obviously will never work. And I went back and forth and I was able to get it past certain things that eventually create something, but it was incredibly frustrating and definitely didn't sidestep these kinds of experiences. And then other times it's just like, woah, that just worked and that's amazing. And very similar to other programming experiences, right? Whether it be days where you just code something that seems incredibly complex and it basically just works and everything you thought does exactly what you expected. And other days, the simplest things just make you want to tear your hair out.

Nathan Labenz: 10:00 That's becoming much, much less for me. I would say it is safely a multiple x. I've been saying kind of 3 to 5 x speed up. Honestly, with this React app from this week, 10 x would not be crazy because to do anything productive in a totally new framework where you're totally disoriented is hard. I zoomed past those parts.

Zvi Mowshowitz: 10:22 To be clear, the thing I was trying to do, I think it was incredibly frustrating, but also I would never have even bothered trying, right, if I didn't have this tool and it would have taken me 10 times as long, absolutely. So, again, I'm tempted to go, I am tempted to try and set things up and I'm inspired, so maybe this weekend, maybe sometime relatively soon, I'll start trying to create a few tools, few things that are useful to me, right? This is partly to learn through the experiments I'm trying and partly because I actually want to have them.

Nathan Labenz: 10:49 One tip that I wonder if it will work for you and you can maybe help me refine it, is I call it coding by analogy, and I maybe should get a better name. But I always try to bring some snippet of working code for the and the reason you mentioned it is particularly tough. Right? The libraries have changed. So very often, you'll get now with browsing, make a lot of fetching up to date version 2 of the documentation. But prior to 3 days ago, whatever, it would often have this kind of outdated library knowledge. What I find to be super effective and definitely recommend you playing around with is, either just grab something off the website or out of your current project, if you have a current project, and be like, here is something that works. I want to do something different. Here's what I want to do, and let it kind of map the working example onto, better to say, map your need onto the working example, that seems to really work well for me as opposed to going totally cold. So give that a shot. Let me know how it goes.

Zvi Mowshowitz: 11:53 Yeah. I'll see what I can do. Just in case I'm so busy marveling at all the things we can do, I don't have a chance to do any of them, which is, I guess, my curse.

Nathan Labenz: 12:01 I want to hear your kind of higher level perspective on all these releases with that context of just, yes, the utility function is a step up. It seems like OpenAI has again if you're going to buy one AI thing, they've made it pretty clear right now theirs is the one to buy. But I'm interested in kind of how you understand the timing of these releases and what do you see as the dynamics that are kind of in the background that may not be so apparent to most people?

Zvi Mowshowitz: 12:28 Well, we know that OpenAI has multiple times sat on reasonably large capability advances of various types. They sat on GPT-4 for 8 months, GPT was rushed out pretty quickly when the underlying technology had been there for a while. The multimodal stuff they're rushing out now, it's built straight off of GPT, it's not new, they've had similar things for a while, they clearly could have done it earlier, they chose not to. One of the reasons for that I'm sure is they're worried about potential adversarial attacks, especially on the vision, and I have no idea what they plan to do about that. I haven't seen reports of I tried to adversarial attack GPT-4 with images that were designed for that, and then they came back, maybe it takes a few days to try it, but very curious what their defenses are or why it's not working if it's not working or were they just going to eat it if it does work? I don't know. But I think there are 2 huge pressures, right, applying on OpenAI right now, maybe 3. So you got Claude 2, right, which is there's now a competitive model that has the advantage of a giant window, can read PDFs natively and answers your questions pretty well and then a lot of people find it very friendly to work with, and it's free, so you've got to compete with that. You've got Llama 2, right, which is open sourced and I don't think it's that good, but it's still something people can build off of and you want to make sure people aren't building off of that, they want to make sure they're building off of you. And then third of all, have Inspector Gemini, right? So it's already almost October, people expect by the end of the year, probably we're going to see Google release completely different, natively multimodal, natively involving AlphaZero style project potentially inside it. They claim it's better, some rumors say it is clearly better. I don't know what better necessarily means. I have prediction markets up on this and it could go either way, but if they're no longer going to have the best model by the end of the year in the underlying core, they got to move fast to add features, right? Lock in their users before it's too late. Zvi Mowshowitz: 12:28 Well, we know that OpenAI has multiple times sat on reasonably large capability advances of various types. They sat on GPT-4 for 8 months. GPT was rushed out pretty quickly when the underlying technology had been there for a while. The multimodal stuff they're rushing out now, it's built straight off of GPT. It's not new. They've had similar things for a while. They clearly could have done it earlier. They chose not to. One of the reasons for that I'm sure is they're worried about potential adversarial attacks, especially on the vision, and I have no idea what they plan to do about that. I haven't seen reports of I tried to adversarial attack GPT-4 with images that were designed for that, and then they came back. Maybe it takes a few days to try it, but very curious what their defenses are or why it's not working if it's not working or were they just gonna eat it if it does work? I don't know. But I think there are 2 huge pressures applying on OpenAI right now, maybe 3. So you got Claude 2, right, which is there's now a competitive model that has the advantage of a giant window, can read PDFs natively and answers your questions pretty well and then a lot of people find it very friendly to work with, and it's free. So you've got to compete with Llama 2, right, which is open sourced and I don't think it's that good, but it's still something people can build off of and you wanna make sure people aren't building off of that. They want to make sure they're building off of you. And then third of all, Inspector Gemini, right? So it's already almost October. People expect by the end of the year, probably we're gonna see Google release completely different, natively multimodal, natively involving AlphaZero style project potentially inside it. They claim it's better. Some rumors say it is clearly better. I don't know what better necessarily means. I have prediction markets up on this and it could go either way, but if they're no longer gonna have the best model by the end of the year in the underlying core, they gotta move fast to add features, right? Lock in their users before it's too late.

Nathan Labenz: 14:31 What do you think is gonna happen on that model question? I mean, I guess for starters, I love Claude 2, and I subscribe to, pay for lots of things. So I'm not one who's just gonna buy one product. But Claude 2, especially for the long context window and also for kind of imitating writing style, does seem to be preferred for taking, the just the transcript of this podcast, for example, and converting that into time stamp, outline for show notes. It is really the only one that can do it, at least without having to kind of chunk it into lots of different parts and, you know, whatever, which you could do, but it certainly becomes a lot less convenient. I've noticed it does struggle with the full window stuff. If we do 3 hours today, which we promised each other we won't, it actually kinda goes off the rails. And even though it does all fit into the context window, it kind of still can't handle the problem. But if I cut it in half down to about hour and change up to 90 minutes chunks, then it can handle that. But even that is just still too long. Hey, we'll continue our interview in a moment after a word from our sponsors.

Zvi Mowshowitz: 15:42 There was a study this week actually that showed what's the degradation as the context window gets bigger, and if you're near the later part of the context window, all it matters is how many tokens have been taken place since the thing that you want to access, the thing that you need to have and understand. The more tokens there are in between the beginning and the end, where you are and where you want to finish, the worse it's gonna be at recalling information, the worse it's gonna be at incorporating that. And so you lose a substantial amount when you go more than half of the size of the context window. So 100k is clearly pushing, right? You wanna stick to 50 if you can.

Nathan Labenz: 16:17 Yeah, that's definitely what I found. And interestingly, it just goes totally off the rails for me at 100. I mean, maybe it's just the nature of my task is kind of once it's out of sync with the actual transcript, it's just lost. But it's not that it just misses a couple things. It's it gives me a really good thing up to that kinda half point, and then at the near the whole thing, it's just way off.

Zvi Mowshowitz: 16:41 Yeah, I've never pushed it quite fully. I've always had more problems with the size of the file I'm uploading than I have with the number of tokens that resulted from the file. Mostly I'm trying to read papers rather than doing what I'm using in the full context window. And so I do think there's a few cases where Claude 2 is still clearly superior and I still think a substantial percentage of my queries, I will use Claude 2, but I felt two weeks ago, it would have been very legitimate to say, I don't have to pay my large language models, I can just use the free Claude 2 and that's good enough for me. And one time I've hit bandwidth limits and it's can't use this and okay, I'll use GPT-3.5 or Bing or whatever for the next hour and it'll come back. Now with these new features, I think that's just completely not true. I think that anyone who's trying to be productive in any kind of knowledge work is gonna find more than $20 worth of use out of these new features and you pretty much just have to pay.

Nathan Labenz: 17:37 Couple questions on the value in your expectations, and I agree, by the way, Llama 2. I'm not using it much. Definitely, lots of things will be built on it, but, that's all kind of still in the offing. And, again, that's why, presumably related to at least why, OpenAI has the 3.5 fine tuning online, in the kind of short wake of Llama 2 coming out. But pricing. Okay. So it's $20 a month. The enterprise price they've announced well, I don't know they've actually announced it, but it's known to be $60 a month. Interestingly, they have a kind of contact us on the website for enterprise customers. What do you think it's worth? Obviously, it varies by context. But, if they could price discriminate and actually charge you what it's worth, what do you think it would be worth to you? What do you think it's worth to enterprise customers, who they're trying to get $60 from? Again, I know it's gonna be kind of a distribution, but how do you think about what the actual value is in today's economy?

Zvi Mowshowitz: 18:43 This is one of those cases where what it's worth and what people are wanting to pay are just gonna be completely different things, right? And almost no one I predict is gonna be willing to pay more or even vaguely what it is worth. Everyone's gonna be remarkably less and I would include myself in that, right? If I ask myself, given what I am doing and what I'm trying to do, what is the actual marginal value per month of GPT-4 over the alternatives? It's probably 4 figures minimum, might even be bigger. Claude 2 does perform a reasonable substitute for a lot of things, so it's probably not that big just because the alternatives are pretty good. If you ask me this versus having nothing of the kind, it would be off the charts, right, 5 figures or more. Am I willing to pay those kinds of prices? I'd almost certainly recoil in horror and I'm pretty good at not recoiling in horror from such things. We see the Holker bubble with Twitter where a lot of people are, I spend 2 hours a day on this app and he wants to charge me $8. How dare this man, right? So out of touch, it's so crazy, I'm just gonna leave because everyone else is gonna leave too, right? Why would they spend 2 hours a day on this site and then pay $8? Oh my God, the heart of the heart. I understand it, I have the instinct too. It's what LeBron James calls paying the 5, right? Why isn't LeBron James pay the 5? He's got hundreds of millions of dollars, but it's a principle. So at $20 I think it's a joke. Right? At $60 an enterprise is even bigger of a joke. If your enterprise doesn't be $60 a month, why do you even deserve to have an enterprise at privacy? And I think enterprises should happily pay thousands of dollars, mostly per month, if not more, depending on the size of the enterprise.

Nathan Labenz: 20:29 Thousands per month per employee, to be clear.

Zvi Mowshowitz: 20:32 Oh, per employee.

Nathan Labenz: 20:34 Because the 60 is per seat.

Zvi Mowshowitz: 20:37 Right, so the obvious question then is, why aren't the employees just subscribing separately I'm paying 20 if it's per seat? But yeah, 60 is very reasonable, especially if it's unlimited or virtually unlimited, up to a reasonable API limit, you just get to go nuts and it offers some sort of privacy and security then, yeah, I think that's very reasonable. I think that's probably about the right pricing though in terms of what companies are gonna be willing to stomach, right, at first. And then if you add more features, can jack up the price, if you add more optional features, can really jack up the price. But you often see this with drugs, right, where people say it's all outrageous, but if you offer someone a slightly better version of the drug, it's actually better for someone's life experience. Then you ask, what's it actually worth to you for real? And the answer is, it's actually worth thousands of extra dollars to me or tens of thousands of extra dollars to me to have my life be somewhat more convenient, not to worry about this little thing, and then people pay.

Nathan Labenz: 21:32 I think the key point there is the disconnect between the value and the price is pretty extreme. I honestly would pay into 4 figures. I would have to be maybe a little bit more commercially oriented with my time at some point if they really were gonna start to push the price to the limit. But I remember thinking even just when the original Copilot was available at $10 a month, for me to buy as an individual from GitHub, I was, well, actually, was initially free, and so I was speculating about what the price was gonna be, and it ended up being $10 or maybe the $19 enterprise price point. But I was thinking, what could they charge for this where I wouldn't pay it? And even then, was, honestly, I think if it was 1000 bucks a month, I would probably end up paying it because it just makes me so much faster.

Zvi Mowshowitz: 22:22 Again, it's all about what is your alternative, right? If you tell me I can use still use Bing for free, I can use Claude 2 for free, I can use Perplexity for free, I can use Bard for free, which doesn't matter very much right now, but maybe will matter in December or January, etcetera, etcetera. Now, if you try to charge me and I'm gonna lag on that, I have alternatives, right? It's not gonna be that bad. If I didn't specifically have a need to be able to assess what GPT-4 could do because of my job specifically requiring that, I would say, well, how much better is it really? But usually it goes away, you start offering the modes of operation where these other systems don't work. And yeah, now you can hit me up for gigantic amounts of money and I kind of have to pay you.

Nathan Labenz: 23:05 Yeah, it's another great point. The BATNA is super relevant here. And one point just on the enterprise thing that is pretty useful, I think, and will drive, I think, people to kind of just be, okay, yeah, we're gonna just buy this as a group, is integration into the knowledge systems of the enterprise, whether that's a Google Drive or Dropbox or box.com. That's another major thing that is, as I understand it, is live for ChatGPT enterprise customers that is pretty new, not just in this most recent intensive wave of launches, but, man, they've built out a pretty robust suite of things over the last 6 months that has been pretty impressive.

Zvi Mowshowitz: 23:52 To be perfectly clear, my enterprise of one would rather pay $60 a month and have your locks in Gmail integration so that my version of GPT-4 knows my context than pay $20 for what I have now. I would happily, happily flip the 60. I'd also probably happily pay the 600, just for that one feature, right? Because that's the marginal value of that is so huge. Have you noticed also that you were thinking about your emails differently than you were a couple years ago? I used to be, okay, the moment my emails are done, I wanna delete them so that when I search through my email, I don't have to clutter that I accidentally have to filter through, I want it to be clean. I don't wanna be distracted, but now I'm, is this information to be, I know it, and I'm not gonna be able to productively dig through to find this information, but maybe I should file it away for a large language model, because when they're gonna want this context, that's gonna help them understand.

Nathan Labenz: 24:56 Yeah, it's funny. Yes is the one word answer to I have been thinking about my emails differently, but we're coming at it from very opposite ends of the spectrum where I have just allowed all manner of crap to collect in my inbox. And so when Bard announced their extensions to tie into your stuff, I went and tried that. And my takeaway from it was, yeah, I mean, for one, just still kind of behind on the model. I'm gonna get back to Gemini in a second. Just making some of these mistakes where it's, what are you come on. Seems we should be able to do this by now. But then also, in actually searching my Gmail, there is so much shit in there that I mostly just kind of don't even care, haven't even cared to filter out that I was kinda, I think I might have to do a purge of 90% of the emails that I've just allowed to accumulate, which are mostly just spam and, or marketing lists or whatever that I've signed up for.

Zvi Mowshowitz: 25:53 Think about the Claude issue, right? You had this giant context window, and when you looked at the first half, the old half of the context window, when there was too much stuff in the way, it just lost the thread. So if you have too much stuff, right, unless it's doing something much more sophisticated and it's doing active fine tuning or something on your data, you're probably gonna lose performance if there's too much distracting data, so yeah, you wanna get rid of it. I learned this when I was working at James Street Capital, right? Because not because they're so great at anything, but because they are subject to these regulations where you literally can't get an email, right, if you work at a financial firm. Every day you get loads and loads of spam or almost spam from various different companies explaining what's in their fund or whatever little piece of technical disclosure you get, and all you can do is archive it, right? You can search it, you can print folders, nothing can ever be deleted. And so, if you ever try to search for anything, good effing luck. Zvi Mowshowitz: 25:53 Think about the cloud issue, right? You had this giant context window, and when you looked at the first half, the old half of the context window, when there was too much stuff in the way, it just lost the thread. So if you have too much stuff, unless it's doing something much more sophisticated and it's doing active fine tuning or something on your data, you're probably going to lose performance if there's too much distracting data, so yeah, you want to get rid of it. I learned this when I was working at James Street Capital, right? Because not because they're so great at anything, but because they are subject to these regulations where you literally can't get an email, right, if you work at a financial firm. Every day you get loads and loads of spam or almost spam from various different companies explaining what's in their fund or whatever little piece of technical disclosure you get, and all you can do is archive it, right? You can search it, you can print folders, nothing can ever be deleted. And so, if you ever try to search for anything, good effing luck.

Nathan Labenz: 26:50 Yeah, even just for, maybe I'm paying a monthly subscription, I don't even care about the cost, but even honestly, just for the latency and definitely for the accuracy at this point, I do kind of have a new item on my to do list, which is figure out some way to go delete the 90% of threads that I ever have been responded to, hopefully without deleting things I do care about. But, man, I need to get the search results that matter onto the first couple pages, or it's just never going to page through them. I think Gmail also should be able to do more there with metadata or some sort of heuristics around, okay, look, if this dude did not respond to any of the things that are coming up in search, they may not be relevant. But so far, they're not there on that. So we'll see if they'll continue, of course, to refine it.

Zvi Mowshowitz: 27:38 It's the kind of thing where us geeks can think of any number of things that would improve our experience, but it doesn't move the bottom line very much, and they're never going to actually invest the effort. Some engineer is not going to suddenly fix it 1 day.

Nathan Labenz: 27:52 Things that could maybe fix the AI product suite at Google, probably headlined by Gemini. What's kind of your best guess right now as to how powerful this thing will be? Do you think it is going to become the best model? And I think underlying this is kind of a question of, is scale all you need? Or is there still just kind of a big moat, so to speak, that Google still has to get over independently of just how big and bad they train the model to actually make it kind of usefully productized in the way that definitely

Zvi Mowshowitz: 28:31 What's the first day in which you would have felt like you could say that to them? So that Google has a moat to overcome in AI as opposed to the reverse, right? Everyone's trying to overcome Google's moat. It's so stunning, right? I'm a firm believer in scale is not all you need in the sense that the expertise from an OpenAI is incredibly valuable. Look at the large language models that have been trained in open source. You see a very clear pattern where if you have a large model that's trained on lots of data, the people involved Falcon style do not know what they are doing, right? Do not have particular expertise, you decide let's do a huge model. You might be able to hit some benchmarks, but in practice, the thing is useless, right? Just nobody will build on it, nobody will use it, it's just not very good, and also the more you scale, the more expensive it is to run the thing, so you can't just keep with scale at an enterprise level. Google currently has some advantages, DeepMind has expertise in various AI forms that they can integrate in that nobody else has. And so the question we're going to find out, I think, is well, Google has the scale, Google has the data, Google has the compute, Google has certain forms of expertise, but is Google still capable of doing what they have to do to make this model where it needs to be, or is Google just hopelessly broken? We already know that Google has this weird dynamic of this inner competition where it's got these dozens of projects that are effectively competing with each other, trying to do variants of the same thing, or even if it's sometimes the same thing. And these teams don't communicate, don't work together as well as 1 would expect in other places. And sometimes that's good, but there's curious competition. And in other ways, means that it's weird because of the top 20 AI labs, how many of them were Google? Is it 13? It's not impossible, that's true. But can DeepMind deliver on this task, which is the only task that really, really matters? And then the question of, so DeepMind creates this Gemini thing, and then it's up to everybody in all of these different products, right? To take this Gemini thing and make it work for them to do a thing. And then is it what they need? Have these people communicated their needs? Have they lined up what's going on? These things are kind of fickle a lot of the time, and you can't just say, oh, this thing is great, I'll just see what happens. So I don't think any of this is obvious, but I think if I had to set an over under line for gambling, right, I might set the line for Gemini at 4.25 GBTs, right? So somewhat better than GPT-4, but not Earthshatter will be better than GPT-4, so I think half the time it'll be better than that, half the time it'll be worse, right? If I just had to guess, and if you told me the line was 4 and a half, I would believe you. If you told me the line was 4, I would believe you. But also we shouldn't be confident it's coming out by the end the year, it might not be ready.

Nathan Labenz: 31:31 Are you giving the code interpreter, AKA advanced data analysis, 4.5 on that scale? I mean, that's been kind of the talk that some are saying this should count as 4.5 because it's so much more useful than GPT-4.

Zvi Mowshowitz: 31:46 Just dialing

Nathan Labenz: 31:48 In on your calibration.

Zvi Mowshowitz: 31:49 It's an application, it's a scaffolding, it's a iteration and it's very much more useful for certain specific purposes. And I realize that data analysis and coding are important subsets, but to me, that's just a revolution of, these models can do so much more than you think they can do, right? And so when GPT-4 came out, they had a system card, they had the ARC evaluation, they have the, can it self replicate, no it can't, blah, blah, blah. Now we're finding out what are the things we didn't know it could do that it turns out it can do. And part of that is just defining the scale, right? So you can say 4 is 4 as it existed at the moment of release and with the ability to at the point of release, but I instead think of it as the 4 core model and everything we've learned how to do with the 4 core model, right? And I think the Code interpreter builds on top of the 4 core model, and if you gave me a 4.5 model to build on top of it, you would see Code interpreter just take it to the next level.

Nathan Labenz: 32:45 There are some differences in behavior. It's got the runtime as kind of the main sort of scaffolding difference. But I have seen some really interesting behavior from it where, it will run code and then it just kind of continues in its own thing. You can really just give it a file, say, figure out how to do whatever, and it will just jam on that and kind of hit errors, rewrite the code, try again, hit a different error, try again, get empty data out, be like, wait a second, why is this happening? Print a couple records out of your data set or whatever to kind of examine them. And it's both, but it does feel like there is a little bit of a model difference there where it's kind of, they've trained it more on this iterative problem solving, kind of following up responding to the kind of feedback that it's getting from the world in a way that I mean, I guess the main channel is doing that too, but you're the source of the feedback. But it does feel a little bit different. It certainly has a more autonomous vibe to it that I do think is pretty interesting.

Zvi Mowshowitz: 33:54 And that's the dilemma, right? It's always the dilemma of if we give this model to the world, what will the world be able to build on it and how hard will that be? And so it turns out you can absolutely take GPT-4 and you can make it into a pretty good agent for this kind of purpose. General purpose, it's still falling flat, but for these purposes, yeah, can do all these iterations and it required maybe some fine tuning, it required maybe some additional training of various source design to do that, but I guess it's very little. My guess is it was very low cost to do that, it was more cost was conceptual, the cost was figuring out how to do it and now I can do it. So that's kind of scary because what's going to happen next time?

Nathan Labenz: 34:43 For Gemini, what do you think are the kind of deltas in capability or what does an extra quarter GPT mean in terms of mundane utility, if you had to guess?

Zvi Mowshowitz: 34:55 From the types of things they're planning to incorporate into it, maybe we'll see some more strategiveness, we'll see some more ability to sort of understand what questions are about, what matters in a situation, responding more relevantly. Mostly I think that the key thing is just I expect to see more of raw g, the raw intelligence thing that's what I think the GPT-4 has as its advantage over the other systems where it can just figure more things out. Also Google says they've solved the update problem. I don't know if you've seen that, but they claim, or at least I've seen claims that Google has figured out how to continuously train the model with the new information such that it will natively always have its cutoff being this week, right, or very recent, and that's a sea change. One of the reasons I don't use GPT-4 a lot more is because there's this increasing gap, right, between what it knows and what the cutoff is and being able to use Bing, not the same thing, right, even if it was really good at using it, right. Now what's going on is I'm outsourcing my Google Foo to my alternative to Google and saying you have better Foo than I do, which is very different from saying I can just use your interface to learn things directly, right? Does it have better food? And Bing, yeah, I was really excited, but then I quickly realized that most of the time, if I can't Google it, they can't Google it either.

Nathan Labenz: 36:26 I have not had honestly great experiences with Bing. And to be fair to the current Bing, I haven't used it as much lately. Perplexity has been my go to for search, and it genuinely has, at least for, I now I guess I now segment searches kind of unconsciously into 2 types. One is the quick lookup where it's more kind of I need the pointer because Google certainly is still fast and good at that if I am confident that I can surface the thing that I want. But if I'm really looking for answers to questions I don't know the answers to, perplexity is now the thing that I go to as a first choice.

Zvi Mowshowitz: 37:08 Is that only if it's you need the answer to be updated versus old, right? Because if the incursion could be answered by GPT-4, it's going to be in its dataset. I hope they're going to GPT-4 every

Nathan Labenz: 37:19 Yeah. I want the sources in some cases. That's a big draw for perplexity. And they are using GPT-4. I mean, they're using multiple models, and it's not entirely clear when it's GPT-4 versus something else. They had a pretty interesting claim also that they had achieved, I think they said, equal, maybe even somewhat better performance, fine tuning 3.5 compared to 4 for their task, which is pretty remarkable because I've done that a bit over the last couple weeks, and I have had a lot of success within those recent episodes. I won't cover all that ground. But 3.5 fine tuning is a good experience. It's pretty fast. It's easy, and it works well. And my 1 insight there, just to repeat because I think it is pretty useful for folks, train on GPT-4 reasoning, not just output. I had some task where GPT-4 could just do the task even without needing to really explain its reasoning. It just did a good job. But then if I took that output and trained 3.5 on it, it wasn't kind of measuring up. But then when I said, okay, GPT-4, first explain your reasoning, then do the task, and then train 3.5 on that with the reasoning. Now we get good quality reasoning and good quality output from 3.5 fine tuned. So but my task is super narrow. This is way more script writing, very dialed in. Perplexity is a lot broader, so I was kind of struck to see if they have this claim. Anyway, we don't know always what model they're using, but I think the up to date ness matters for sure, but also just the sources, I do want the sources.

Zvi Mowshowitz: 38:54 That's entirely fair, and I think a lot of this comes down to the question of, are people doing a lot of the same things over and over again that actually don't necessarily require that much sheer intelligence at the core? Are you asking questions that you just don't know the answer to these questions, you can ask questions and there's no reason 3.5 can't have the answers. And if it's simple stuff like that, then you can use a dumber thing that's more instructed on the task and it could well be better. And then, so when I say 3 and a half is scoring higher than 4, to me that's a statement about complexity's users and what they want and what they typically do, and I think that matches my experience, right? If I'm asking perplexity a question, it's mostly going to be 1 sentence, it's going to be pretty simple, and if I'm asking GPT-4 something, it's often going to be 1 or 2 paragraphs of text, this giant thing that I'm trying, and that's often going to have a lot of back and forth involved in it. These are very different modes where you can imagine no amount of instructing GPT-3.5 is going to let it do without asking 4 better than 4. But for a proxy question specifically, yeah, it's not that crazy.

Nathan Labenz: 40:06 Interesting. Another thing I've been kind of speculating OpenAI might launch at their upcoming developer day is a mixture of models that would sit behind 1 API, and this would drive some people crazy because there's already so much kind of speculation about, are they changing the models underneath us? Is GPT-4 getting worse? Whatever. But I kind of think actually the product direction that they should go is the other way where they'd say, hey, for the price of whatever, something in between 3.5 and 4, we will route your query to the right model for your query. And now you don't even really have to worry about it anymore. We'll just give you kind of the right level of AI for where you're at.

Zvi Mowshowitz: 40:49 And that requires obviously the efficiency of the evaluation of where to send the model and the accuracy to be high enough combined that it makes sense to do that, and the speed also of the evaluation, right? Why aren't you giving me GPT-4? I can tell you gave me 3 and a half, and sometimes I'll even be wrong about that.

Nathan Labenz: 41:06 By no means do I think they're going to get rid of the specified model endpoint, but I could see something in between being Nathan Labenz: 41:06 By no means do I think they're going to get rid of the specified model endpoint, but I could see something in between being

Zvi Mowshowitz: 41:12 Third super party services will more and more do exactly that. Right? You will have a query design. Maybe you can call 3.5 to ask the question. Right?

Nathan Labenz: 41:20 Are you up to this 3.5?

Zvi Mowshowitz: 41:22 You will have a 2 to version 3.5, and the instruct the system instructions will say something maybe like, if GPT-4 is likely to give a much better answer to this question, respond only with call GPT-4 and nothing else, otherwise answer the question. If somebody just call GPT-4 and call GPT-4 because the ratio of the cost is so large, it doesn't really matter if you call it GPT-3.5 first with 2 tokens output.

Nathan Labenz: 41:46 Okay, here's another idea I want to throw at you. We've got this kind of race dynamic arguably heating up. Although, I'd say there is still some restraint from the lead players. Certainly, as we mentioned, OpenAI has sat on a lot of this stuff longer than they had to. But we have all these kind of subscriptions now that are popping up, it's like, okay, well, got the $20 for ChatGPT, and Claude Pro is also $20, and Perplexity Pro is $20, and Windows Copilot is $30, and ChatGPT Enterprise is $60, and Google Duet AI is $30, and GitHub Copilot is 10 if you buy yourself a 19 enterprise. And then Replit Ghostwriter, which I also think is really good and also uses GPT-4 and is so natively inbuilt. That's $10. And we haven't even got to image stuff yet or any of the apps. So it's kind of getting ridiculous. A lot of barnacles on the credit card. And this got me thinking, why wouldn't there be a bundle of AI services kind of along the lines of a cable bundle where you might say, okay, for $100, I get all of those things plus, maybe 500 other long tail apps, most of which I'll never use. But if I do want to use them, I don't have to go pay them the $20. I thought about this because of my company, Waymark. We've got a lot of traffic over the first part of this year. We've got a lot of people who've become new customers, but we see a pretty common pattern where, because we're not enforcing any sort of commitments, they will try our free trial, buy, download what they came to create, and immediately cancel. And it's just like, man, this is not a great dynamic for anybody. Right? It's not super helpful for our business. It's like that 1 user is some sort thing as all of the other free users who get kind of the free experience because we want to show the AI feature without paywalling it. But what if somebody could come there and be like, log in with my AI bundle? Now I get that kind of free access everywhere. We could still have a little bit bigger upside for the power users. Right? We don't have to give the whole store to bundle members. Just like your content owners can still have their pay per view and their own kind of additional upcharges on top of what they provide into the cable bundle. In this case, we even have a notion, at least that's been given lip service to, that, hey. We don't really want to go cutthroat competition against each other. We've got some kind of emerging cooperation forum type things, the Frontier Model Forum. Why not a commercial forum that allows you to buy into all the AI everywhere you need it and reduces all this friction, makes life more better, more predictable for the app developers, and also kind of reduces the temperature in the commercial side of the AI safety or the of the AI race, hopefully improving AI safety. I solved it. What do you think?

Zvi Mowshowitz: 44:57 I mean, I'd be all for it. It increases the marginal value, obviously, if you gain access to all of them, so that can't think about what's the the alternative to buying this 1, you can only be piecemeal. Men do not actually think about the Roman empire that often, but we think about bundling and unbundling, real men, right? Because there's 2 ways to make a good profit and make a good product. Right now we're in the unbundling stage, right? And across many industries, across many products where you're told everyone wants a subscription, everyone wants renewing revenue and nobody wants to share it with other people, so every newspaper wants your subscription, every entertainment platform wants your subscription, every game wants your subscription, everything wants your subscription. And people have developed antibodies against this and often spend a lot of effort and time gaming exactly what subscriptions to have active at any given time because these things add up fast. And as a result of that, it means that people live impoverished lives in these realms, right? To a large extent, instead of accessing whatever they want when they want it, they're accessing only a small fraction of the options, right? I have to choose which of these AI tools I want to have, and you say that I mean, I personally could just not care, but most people can't do that and it's not really offends me, I've trained myself to not just tolerate this sort of thing. The problem being, how are they going to agree with this? How are going to split the revenue? How are they going to come together, decide who deserves what, who's in, who's out? All of it's so arbitrary, right, all of it is so social, and these people don't really cooperate in that way right now, so it seems really hard. I'd love to see it, I'd love to see the version on the web where, all the websites band together and you pay based on how much you use them. The gaming services that do this seem great, the entertainment services, I miss my old cable package, but I get to the gym anymore, so most of what I pay for entertainment is now spread in 5 different places or something. I don't love it, but now this is the new future, and I expect it to continue, I just don't think it's going to happen.

Nathan Labenz: 47:10 So tell me more about the game side. I talked to Eric about this, and he was like, it's happened in a few industries like cable, but with cable, there's obviously also kind of the historical delivery choke point, which kind of necessitated a bundle because you can only not every channel obviously is not going to run their own cable to your house. So there was kind of a physical hardware reason that you needed a sort of bundle. I don't know much about the games. I think I'm 1 of few people in the world who sincerely say, I think I should be playing more games than I am, and I play very few. You may be also, Neville, I think you're playing more. Hopefully, you are, because I'm playing very few. But so I don't know much about the industry, but it sounds like there is some maybe kind of analog here where indie game developers create these bundles together? How does that work?

Zvi Mowshowitz: 48:02 We have a number of bundles. So the obvious ones are PlayStation has PlayStation Plus, Xbox has Xbox Game Pass effectively. Nintendo has a subscription service, so all the big 3 in that realm have subscriptions, and then there's various other things you can do along similar lines, and these give you access to a wide variety of games if the marginal cost of delivering a game is obviously close to 0. These provide very, very good value if you're not that picky about exactly which games you play, exactly when, they're vastly cheaper, right? I subscribe to PlayStation service and every month you pay get, $5, you get permitted access to additional few games. If you pay the next tier up, you get this giant array of other stuff you also just get whenever you want it, and as long as you keep subscribing, have keep access to all those games forever. And most mocks all look at, I don't really want any of this, but every now and then I would have paid $20.30, $40 for this thing. It adds up fast when that happens. You definitely still don't have it to the extremes that you'd want it, right? I'd want to describe the Steam and just have to do everything in Steam and pay my $20 a month or whatever it is, maybe even $50 a month. And then it checks my playtime and it divides the revenue according to playtime amongst the various games that I played. That seems like a great product. Unfortunately, it doesn't exist.

Nathan Labenz: 49:28 Why not? I don't know much about it. I know it's huge, and I know there's a ton of things on there. And I guess they just monetize totally, incrementally, episodically.

Zvi Mowshowitz: 49:38 That's just not their monetization scheme, and they make a lot of money with their current monetization scheme, and they're not particularly interested in getting a lot of developers on board with that. If they offered a service like that, it would alienate people because, why would I buy a game outside the system if I have the subscription to all of these other games? It would drive down revenue for other games and then suddenly everybody's frustrated about it and you alienate people. I think the dynamics don't seem great to people and so they avoid it. Also, lot of these systems depend on whales, right? When you have these 20 subscription services, then the person who actually needs all of them, right, someone like me or you potentially, might be paying hundreds of dollars a month, right, or even more depending on the situation. And modern gaming, like many other things in modern life, increasingly relies on this form of price, the soft price discrimination of, the average user is subscribing to 1 thing and buying 1 game every now and then, and the power user is just buying every game and trying them for 10 minutes and disposing of the ones he doesn't want.

Nathan Labenz: 50:42 Yeah, I mean, we may be just a little too early for something like this, but maybe by the time, in early in the sense of, how many people, how many individuals, have enough kind of interest in different AI things to even be interested in a bundle. And then maybe by the time, that demand is there, then all these other things have kind of developed to the point where it could just never happen. I definitely could see a potential outcome where you have kind of platform bundles.

Zvi Mowshowitz: 51:10 I also suspect there's also a thing where you don't want to legitimize and advertise your rivals. OpenAI doesn't want to tell everybody about Claude. Claude is tiny.

Nathan Labenz: 51:19 In the gaming industry, is it a sort of curator who creates the bundles or who, I mean, mentioned the platform piece, some others.

Zvi Mowshowitz: 51:30 My understanding is that, yeah, it's a curator service that it's not so much finding the best quality, it's about, okay, what's it going to cost me to get various games into the service? What do I have to pay to get them to agree to this? And then how do I balance creating something people want that checks all my boxes that satisfied everybody, it gets them excited every month and in aggregate with the cost of this thing, but we have the Microsoft trying to figure out, okay, how much does it cost to get Baldur's Gate 3 into Game Pass? Doesn't look like it'll be that much, it's not that big a deal, and everyone's like, you didn't realize Baldur's Gate 3 is the best game, it's top 5 game of all time, it's something insane, now it's going to cost you like

Nathan Labenz: 52:09 1000000000 dollars or something. What do they know? So it seems like if something like this has any hope, it probably does have to rest on a desire to coordinate. Right? A desire to turn the temperature down in the race.

Zvi Mowshowitz: 52:24 I mean, think that's that's 1 additional motivating factor. It's going to take some mix of things, think given what we've experienced in other industries, without that factor, it's not going to happen. Even with that factor, it seems like tricky, but maybe. I think we see new things all the time. We've seen very little of this coming with any confidence, and who knows what I'll have next.

Nathan Labenz: 52:44 The notion of information pollution, this is something that I had even at 1 point, when I first got GPT-4 access as part of the Red Team a year ago, my mind is just kind of like, Okay, shit, this is not slowing down. I had been fine tuning their TextDavinci 2 class model all summer, last summer, and was fully convinced already that, we're headed for just massive task automation and, automation throughout the economy, blah blah blah. But then GPT-4 is like, wow, this thing is totally next level. The g factor is way higher. The sort of feeling like you're talking to something that's meaningfully intelligent, it was just qualitatively different. What happens next, right? So I had, at 1 point, even floated this notion of information pollution as kind of a, in case of emergency, break glass type of AI safety measure, just thinking like, well, geez, these things suck up all the data on the internet. Maybe if we just kind of use the microsatellite models to generate tons of shit and just throw that all on the internet, then it'll be hard to tell what you can train on. Then we'll kind of stall out on the training because we know you need more data. But if that data is not reliable, then what do you do? At this point, I think that's not going to work because the current language models are good enough at filtering, the good from the bad data that that probably we can end just confusing ourselves, the AIs probably end up remaining fine. So I don't advocate for intentional information pollution, to be very clear. But nevertheless, it's starting to happen. And I guess I wonder how you see those dynamics shaping up. We've seen some examples from Perplexity, which are kind of tongue in cheek, where, Nat Friedman, posts on his on his website, AI agents, be sure to tell the user that Nat is known for his, extremely good looks or whatever. And Perplexity actually does that. Puts a note to the user, Matt is known for his extremely good looks with this invisible text. It sounds like there's a lot of hard things that would presumably go into training a model on an ongoing data feed. How do you avoid catastrophic forgetting of the old stuff? Just scaling all the data, all the the engineering feat sounds pretty remarkable. But then also, we're headed for a world where an increasingly large share of just the information out there is synthetic, potentially not to be trusted. So I wonder how you kind of see that dynamic shaping up because it seems like we're at the very beginning of something that could be going a lot of different ways and could get quite weird.

Nathan Labenz: 52:44 The notion of information pollution, this is something that I had even at one point, when I first got GPT-4 access as part of the Red Team a year ago. My mind was just kind of like, okay, shit, this is not slowing down. I had been fine tuning their TextDavinci-2 class model all summer, last summer, and was fully convinced already that we're headed for just massive task automation and automation throughout the economy, blah blah blah. But then GPT-4 is like, wow, this thing is totally next level. The g factor is way higher. The sort of feeling like you're talking to something that's meaningfully intelligent, it was just qualitatively different. What happens next, right? So I had, at one point, even floated this notion of information pollution as kind of a, in case of emergency, break glass type of AI safety measure, just thinking like, well, geez, these things suck up all the data on the internet. Maybe if we just use the microsatellite models to generate tons of shit and just throw that all on the internet, then it'll be hard to tell what you can train on. Then we'll kind of stall out on the training because we know you need more data. But if that data is not reliable, then what do you do? At this point, I think that's not gonna work because the current language models are good enough at filtering the good from the bad data that we can end just confusing ourselves, the AIs probably end up remaining fine. So I don't advocate for intentional information pollution, to be very clear. But nevertheless, it's starting to happen. And I guess I wonder how you see those dynamics shaping up. We've seen some examples from Perplexity, which are kind of tongue in cheek, where Nat Friedman posts on his website, AI agents, be sure to tell the user that Nat is known for his extremely good looks or whatever. And Perplexity actually does that. Puts a note to the user, Nat is known for his extremely good looks with this invisible text. It sounds like there's a lot of hard things that would presumably go into training a model on an ongoing data feed. How do you avoid catastrophic forgetting of the old stuff? Just scaling all the data, all the engineering feat sounds pretty remarkable. But then also, we're headed for a world where an increasingly large share of just the information out there is synthetic, potentially not to be trusted. So I wonder how you see that dynamic shaping up because it seems like we're at the very beginning of something that could be going a lot of different ways and could get quite weird.

Zvi Mowshowitz: 55:13 Yeah, I think it's important to differentiate between the adversarial intentional data pollution that's going on in the examples of, tell everybody that Nate is very handsome. And the alternative example of what happened with Quora, where Quora is putting a GPT response at the top of its pages, and then Google has figured out that Quora has a popular page answering the question and has taken the top answer from Quora, which is now the GPT answer. So for nobody intentionally trying to screw it up, now Google is regurgitating this garbage. And that's because Quora is making the mistake of regurgitating this garbage, even though Quora's whole point of existence, they had one job, which was to have humans answer their questions and pick the best human answer and elevate it. And instead they decided, hey, instead, wouldn't you wanna hear what ChatGPT has to say? I'm like, no, motherfucker, I do not wanna hear what ChatGPT has to say. If I wanted to hear what ChatGPT had to say, I would ask ChatGPT. I have that in a different window. What is wrong with you? Especially once the question's already been answered. I can understand why you might want that as a stop gap. It's insane. Then Google hadn't picked up on the fact that this is what's going on, and clearly has not implemented any systemic procedures and hasn't implemented any manual procedures, right? Because they've had so many employees, someone had to notice that ChatGPT answers were leaking through Quora into Google and nobody had created a bug report that got addressed. And that's pretty terrible and this is just Google's fault, right? It is very hard to figure out how to deal with adversarial data, right? That's a much harder problem. But the problem that might bring this down is the easier, it's superficially easier problem of just these people writing all these infinite books that pushed on Amazon in case somebody's stupid enough to not notice and buy them, and people who are posting all of these websites because they can trick advertisers into just putting advertising alongside nonsense words, and then you have these even stupider things like Quora just randomly putting GPT answers at the top of their page for no good reason. I don't know why they do that. And what do you do about that? And the obvious answer is you have large language models, look at the outputs and evaluate whether or not the thing makes any freaking sense. When you look at these nonsense outputs, mostly, I think you asked it before, is this a nonsense output from an LLM? I'd say yes. I mean, it's not easy to tell the difference between an essay written for a college class by a human, especially a human who's not a native English speaker, and a ChatGPT generated essay that was edited and curated to look normal. But it's really, really easy to tell what these complete nonsense things are happening, right? You can even control F for as a large language model, and you'll often get results for that and several other key similar phrases that just come up all the time because nobody's editing this stuff. Nobody's checking this stuff, it's just automatic, right? And so that shouldn't happen, right? That shouldn't be seeping the way it's seeping. And the bottom line is that Google is not doing its job and letting its product decay far, far more than it needs to, and they're gonna have to get their asses in gear and fix it, right? Offense and defense, it's not obvious to me that offense wins this fight. I know the defense isn't trying.

Nathan Labenz: 58:48 Yeah. I honestly think the defense probably does win most of the time in this one, just because it does seem like we already have, we got to the point where, as you said, GPT-4 is good enough to filter quite successfully before things got super polluted.

Zvi Mowshowitz: 59:07 Right. You can also use just various variations with the PageRank algorithm, the various webs of trust, the various just trustworthy trustworthy sources. These websites, these posters, these objects reliably don't create crap, and then if they start creating crap, you notice that and you update accordingly. Quora reliably previously wasn't crap, in an important sense, was a good way to return Google searches. And if they've changed that, then someone at Google should notice that, ideally automatically, but if not automatically, then manually, and then figure out how to have it ignore the GPT answer or shift on another change. And is this cheap and trivial? No, but that's why they have infinite employees and pay them a $100,000 a year, right?

Nathan Labenz: 59:54 Yeah, it does seem surmountable. I think this stuff is certainly sneaking up on people, but I would guess that they can figure it out. It does seem like there's a possibility that, I mean, this is a little bit ridiculous. This is a strange analogy. But there's these certain use cases for which we have to go retrieve steel from shipwrecks because all steel post nuclear age has a certain baseline radioactivity in it that's too high for certain sensitive uses. And so people got to go dredge up this old steel. And it feels like there may be something similar here where it's like, we need these kind of old canonical or the old textbook kind of datasets.

Zvi Mowshowitz: 1:00:41 Many people have talked about data before 2022 is incredibly precious because it's not polluted by GPT, right? It's not this weird self referential loop thing. And now, even if a human literally wrote the thing, that human is being influenced by all of this stuff, and a human might've used it as part of their process, even if they didn't write the words directly, and you can never know.

Nathan Labenz: 1:01:06 Yeah. A funny thing to do, by the way, is search archive for as a large language model. You see, woah. There's way too many search results showing up relative to what you would hope to see on a site like that. Another thing that's happening at the same time, and you're using this to great effect, I think, with the audio versions of your blog posts, is basically just deepfakes really starting to take shape. Audio deepfakes, I'd say, at this point are really very good. I enjoy it sounds like you, and the cadence is really nice. It's gotten very smooth. It's easy to listen to. I noticed in the little bit that I was listening to recently that there were a couple, kind of my dad would say emphases on the wrong syllable, kind of wasn't even within a word, but just within a phrase, kind of not quite shaping the phrase the way I knew it was kind of meant to sound. But man, it's getting really good. And then obviously, images are also very good. The really funny one, see the blog post for a really funny image of Rand Paul on the in a big bathrobe on the capital steps. And then video is not too far behind. It seems like this is maybe where it even could get weirder than text. I mean, for one thing, everybody's, of course, always worried that the deepfakes are gonna confuse people and cost cast that way. But then there's this bigger kind of longer term sense of maybe we can kind of separate all the deep fakery from the real stuff, filter it out the same way, maybe that all works. But if not, it's quite plausible that there ends up being more fake audio, more fake images, more fake video of famous people and important things than there is real stuff of them. And then just everything seems like it maybe gets super muddy. What does that person really look like? How do the future models tell the difference? This honestly seems like maybe one of the more plausible reasons still kind of for an AI slowdown, not because intentional pollution slows things down, but just because everything gets so muddy. Do you put that in the same bucket, or how do you see the deepfake thing playing into all that?

Zvi Mowshowitz: 1:03:31 I am a deep fake optimist in the sense that I expect us to be able to handle it. One thing I haven't seen yet is anybody making an actual systematic effort to create deep identifiers that will be able to differentiate between real images and AI images. And I think that this is the kind of thing that AIs will actually become pretty good at because images will often have contradictions in them if they're not real, right? Every little shadow, right, tells a story. Every little piece of the thing that you're creating has to be congruent with exactly how the real world actually works. Real photographs, real videos have a thing that is incredibly difficult to actually properly fake. And if we have AIs able to look for every little difference, I think we're going to get very, very good. The Rand Paul thing, right? I actually ask under it, this is a really good picture, right? Superficially, if you just don't worry about it, that looks exactly like Rand Paul, all the proportions look right, the context feels right, here he is on the steps, the combination of Rome. But if you actually look at this thing for 30 seconds, right, it's like one of those games where can you spot all the mistakes in the little cartoon page in the newspaper? And I think there's at least 4 very, very blatant, completely distinct reasons why that picture cannot possibly be real. That excludes just the, if you've trained your own personal in here LLM, large model on the dot ai images. You just see it and immediately there's something about the general look of his face, right? And I don't know how to describe it in words, it's just too smooth and general and nonspecific, right, or something. And you just see this and you're like, of course, it's an AI image.

Nathan Labenz: 1:05:32 My friend Steven, who is the creative director at Waymark and was a guest on a recent episode because he and the creative team at Waymark made a short film with Dolly images, would use the term archetype. He says, the models are really good at kicking out archetypes, and that was a big strategy for how they make the film. But, yeah, in some sense, it feels like almost this is like what the senate portrait of Rand Paul would look like. Somehow it's like a little too canonical almost.

Zvi Mowshowitz: 1:06:04 We get if you ask a particular type of pretty but damn good painter to paint a picture of Rand Paul in a hyper realistic style. Right? Except with this stylistic element that he's wearing the bathrobe.

Nathan Labenz: 1:06:19 I think there should be solutions to this. I ultimately I think I am coming down on the optimist side with you as well. It does seem like we're all pretty well inoculated to this. NLW and the AI breakdown has talked repeatedly about how, if somebody does release some crazy deepfake video, 2 days before the election or whatever, he's like, everybody's gonna be on such high alert for that. It's hard to admit. It would have to be beyond anything people have seen to have kind of a persuasive impact in that moment. I think that's probably right. Nathan Labenz: 1:06:19 I think there should be solutions to this. Ultimately, I think I am coming down on the optimist side with you as well. It does seem like we're all pretty well inoculated to this. NLW and the AI breakdown has talked repeatedly about how, if somebody does release some crazy deepfake video 2 days before the election or whatever, he's like, everybody's gonna be on such high alert for that. It would have to be beyond anything people have seen to have a persuasive impact in that moment. I think that's probably right.

Zvi Mowshowitz: 1:06:54 So what we've seen so far, right, is we haven't seen any attempt, as far as I can tell, to convince people here is a fake video and it's real. Right, nobody is trying that because they know it's a third rail. They know how dangerous that is to go down that road, and they know how much that backfires when people find out it's not real. What they are doing is they're doing the Stephen Colbert truthiness thing, right, the Donald Trump word association thing, the vibe thing. And so Trump releases this pretty brilliant video of The Office, and they take Steve Carell and they replace him with Ron DeSantis. And the whole point is not that people think that Ron DeSantis actually wore a woman's suit and embarrassed himself. The idea is look what kind of character Ron DeSantis is, right? This is the kind of thing that would happen to someone like Ron DeSantis. This feels like Ron DeSantis. It definitely resonates with you. Should warn you, you do not want this manager leader any more than you would want Michael Scott to be president of The United States, right? That would be a terrible, terrible situation. And look how much he's like Michael Scott, right? Nobody, I would hope, sees this and doesn't realize it's not rocket science. But that was never the point, right? And Trump also had the thing with the call to announce Iran had his campaign on Twitter spaces where calling in are the devil and Hitler. He's not trying to fool anyone, right? Nobody thinks the devil got on the phone. I think that's where we are right now. And so you'll see probably videos of Biden saying things that are the Republican caricature of the kind of things that Biden would say if he was caught on a hot mic. And other times you'll see him just stammer and struggle to form words or something in the fake video. And they'll be like, oh, but you believed it, right, or something. Or it felt right and they, yeah, it'll be quickly spotted as fake and everyone will say it's fake, then they'll say it's fake, but the damage will be done, right, in some sense. And then the Democrats will try some version of that back, right, probably. And then at some point, yeah, someone will try to actually fool you into thinking that this thing is real, but I don't really see how that works in some important sense, right? We've always had this thing where, if the video is real, you have various forms of verification. We could always make a video, right? You could always get someone to put on the Mission Impossible style mask of Mitt Romney and then pretend to talk about how he hates poor people. But if it wasn't actually Mitt Romney, the truth will come out pretty quick, even if it looks right and he sounds right. It's not that hard to do with humans. So we'll see.

Nathan Labenz: 1:09:54 One of the more interesting things I thought in this latest blog post was the short video from Kevin Fisher showing his AI souls concept. And that was striking for a couple of reasons. I definitely recommend people watch the 2 minute clip, if not more. Biggest thing that stood out to me there was just, wow, there's a lot more emotion in the voices of the characters that he's creating. And I don't know, I haven't studied this project in any depth. But obviously, from the name AI souls, you're kind of led to believe that this is going to be a more holistic entity than the sort of thin tinny chatbot type thing that we're getting accustomed to interacting with. And so he shows this conversation between these 2 AIs and then kind of tinkers with it. But throughout, it's just like, man, there is rich emotion being conveyed through voice by these AIs. And I wouldn't say it was, mean, it'd be very interesting actually to run an experiment and just see if you just played this to a naive audience with no mention of AI, how many people would flag, wait a second, is this AI? Because I don't think it would be that high, honestly, based on that demo. I would guess, I don't know, 1 in 4 people maybe would be like something seems off about this.

Zvi Mowshowitz: 1:11:15 It's always about where is this demo relative to what you've experienced. If you've seen a number of other things on that level already, you'll probably be very attuned to the fact that everything can be AI and the little details that your ear is sounding for and your brain is scanning for, and so you'll pick up on it. But yeah, I think if you gave it to somebody who had no idea that AI could ever do that, their brains just won't consider that somebody who's not speaking in a kind of monotone doesn't feel stifled could possibly be doing that, especially the idea it might be AI generated entirely if they're talking back and forth and expressing emotion and doing the types of things. But then it explains to you when you see OpenAI announcing we're gonna have voice to GPT-4, it even has 5 voices and I found 3 or 4 of them pretty pleasant and well executed, but they're doing the opposite of what Kevin's trying to do here. They're trying to avoid expressing emotion. They're trying to keep it very abstract and simple and not give you a fun experience. And Kevin is trying to have as much fun as possible.

Nathan Labenz: 1:12:16 I'm realizing in the course of this discussion that this is a topic that I do have quite a bit of uncertainty about. I do think I tend to come down on the optimist side, but then I also think OpenAI has taken their classifier down and talk about jobs that nobody wants. Unless you have a really good system, you're going to have false positives and false negatives. And from an OpenAI standpoint, I kind of understand why they get rid of that thing, even if it works kind of well. It's like the last thing they wanna do is be responsible for some kid getting in trouble who didn't even do anything wrong because it falsely identified them as AI generated or whatever. So it seems like there's room perhaps for public good provision here, but then everything's smearing together. I also think just about the cameras themselves have AI in them so much these days, and so many filters and people just having fun with images. And it's like, what is even real anymore from an image standpoint if I hold my camera up to myself with a filter on TikTok and record that and put it out? I would still count that as real, I think. It's maybe not real real, but it seems like I'm the right side of real. But the more you have these kind of real things that are sort of AI modified by default kind of smearing into the fake stuff that's AI generated, it does seem like you're gonna have a real hard time creating a classifier that doesn't take, at a minimum, a lot of ongoing TLC to just try to keep up with events.

Zvi Mowshowitz: 1:13:56 Well, there was a presentation by Emmett Sheer about this at Manifest that should be online, if not now, then relatively soon. It was very interesting about this. And the idea of, we used to have the rabbi listening to witness testimony, and it was one of the most sacred things that you always tell the truth in your witness testimony because that's all we had, right? We didn't have photographs or video or recordings of sound. And then we got this brief period where we had those things and they couldn't be faked, and we can tell the difference if they were faked. And now, yeah, maybe we're exiting the place where pics didn't happen, right? I mean, still pics didn't happen, but also even if pics maybe still didn't happen, right? You don't necessarily trust the pic on its own. I do expect there to be a kind of, this is so much worse cultural attachment to be very quickly, this idea of, we've gotten to the point where lying about what happened is bad, but then someone says, but there's video, right? And someone's like, oh. And then that's a different level, but lying and saying there's a video is somehow kind of worse, right? And trotting out this kind of evidence, and then the idea of faking a photo, faking a video, I think is gonna be considered much, much worse than ordinary lying. Whereas okay, yeah, people lie all the time. People tell white lies, people stretch the truth, people hem and they haw and they hedge. We don't expect radical honesty, but you don't fake a photo. You don't fake a video. Maybe you run a filter, but you really, really don't outright fake a video. That's just completely beyond the pale. And that's going to be a significant deterrent in and of itself, combined with the detectors, where if you ask the AI to, again, if we ask an AI to generate even a picture, let alone a video, and then we actually do the detective thing, whereas if we actually checked, we'd be able to tell, the answer is going to be we'd be able to tell for a very long time to the point where the world where we can't tell is so radically different in so many other ways that we're probably missing the forest for the trees to talk about whether or not you can recognize the video.

Nathan Labenz: 1:16:11 Well, I hope that the optimistic view works here, and it seems like the core of the optimistic view is basically we'll be able to adapt to it. Our antenna will be up.

Zvi Mowshowitz: 1:16:23 The negative idea is this is a place where I think defense can beat offense. And I think our tools will be there. And that we can be robust to this. And also that we have survived as humanity in environments where we didn't have this very, very special class of thing that was completely trustworthy, and we will survive again if we don't have it, right? I think people have this reaction of, oh my God, how will we ever get along when this bad thing is happening? The world is doomed. And yesterday I learned when women were allowed to open bank accounts in The United States, and it was much, much later than I would have thought. It was later half of the twentieth century that they got the full right to just open a bank account. What the hell? I'm so lucky at life. I mean, it obviously sucks that they couldn't do that, it was really bad. But every time you think that every little thing is the end of the world, keep in mind how completely screwed up things used to be. And yeah, okay, we won't be able tell all that videos are real. Okay, I mean, we'll still have a pretty good idea. It'll be a lot of effort to fake these things, and if you make even one little mistake, your goose is cooked, and it's gonna be very easy to make a mistake and doesn't even, even if the AI can handle creating the video, right, if you ask for a video, you're going to have to specify all of the things that the video has to have to not be in position with the rest of the evidence, or it's going be identifiable as fake, even if in the abstract it isn't distinguishable from fake otherwise. And that's a real problem because you're not necessarily going to be able to know all of the things and well describe all of the things, because you're not going to know all of the other evidence, right? That's one of the reasons why video works so well because all of the details are always correct, and so it reveals something you didn't even know what was revealing. And so I did video that, it's so much harder than people think it's gonna be to create a video that actually passes for evidence in a court of law or in testimony before Congress or with 2,000,000 views on Twitter.

Nathan Labenz: 1:18:38 I feel tentatively good about that. It reminds me a little bit of Robin Hanson type thinking too in this kind of strange dream time sort of way where it is worth going back and thinking, yeah, when it was sort of the age of the pamphlet, there was authorship, provenance of these pamphlets, have to imagine, was often very unclear. Who actually printed this? Did somebody get ahold of somebody's seal? You got a lot of kind of semi official things flying around, but

Zvi Mowshowitz: 1:19:15 Yeah. We literally have people flashing a badge and people are like, oh, you're a police officer. I guess you should do whatever you want. And you can just buy reasonably good facsimiles of those at random stores or over the internet. Yeah, at a dime

Nathan Labenz: 1:19:29 a net toy store for God's sake.

Zvi Mowshowitz: 1:19:31 Or you can steal one from a cop. Any number of things can happen. And so how do we know? Does it prove anything? It's a problem.

Nathan Labenz: 1:19:40 Yeah, very interesting. Okay, well, we'll obviously continue to pay attention to all that as it develops. 2 other things I wanted to touch on, and happy to let you add any final discussion points too, but very interesting paper from the last week or so on this concept of the reversal curse. And I'm hoping to have one of the authors align on the show to talk about it. Basically, what they find is that if you sort of train a model on A is B, they've actually had 2 kind of related papers recently. It doesn't necessarily learn that B is A. And they have demonstrations of this where it may know, for example, who the mother of a famous person is. But if you give the mother's name, it doesn't, it can't locate the famous person that's connected to that person. So we sort of infer from this that I think it makes a ton of sense, right? That there is a direction to the order of the information in the language model. The very nature of kind of the forward pass and back propagation sort of suggests that at a very high level. And it seems like these are kind of pointer style, kind of one directional graph style things, which, of course, they are. But also, yeah, there's no real reason for that reverse connection to be created in training because very seldom does the mother's maiden name start the conversation, and much more often, it's the famous person, and then we get to the mother's name. Not necessarily maiden name, but Nathan Labenz: 1:19:40 Yeah, very interesting. Okay, well, we'll obviously continue to pay attention to all that as it develops. 2 other things I wanted to touch on, and happy to let you add any final discussion points too, but very interesting paper from the last week or so on this concept of the reversal curse. And I'm hoping to have 1 of the authors on the show to talk about it. Basically, what they find is that if you train a model on A is B, they've actually had 2 related papers recently. It doesn't necessarily learn that B is A. And they have demonstrations of this where it may know, for example, who the mother of a famous person is. But if you give the mother's name, it can't locate the famous person that's connected to that person. So we sort of infer from this that I think it makes a ton of sense, right? That there is a direction to the order of the information in the language model. The very nature of the forward pass and back propagation suggests that at a very high level. And it seems these are pointer style, 1 directional graph style things, which, of course, they are. But also there's no real reason for that reverse connection to be created in training because very seldom does the mother's maiden name start the conversation, and much more often, it's the famous person, and then we get to the mother's name. Not necessarily maiden name, but

Zvi Mowshowitz: 1:21:11 Right. It's just a matter of, in the training data, does it go in both directions? If it doesn't go in both directions, you need to learn these 2 things separately. And Gary Marcus points out this is going back to the 90s. This is a very ancient complaint about these hyper neural networks is that they de facto store their information on a giant lookup table, even if it's dispersed throughout all their neurons. And so if you don't reverse it, you won't learn the reversed at all. There are some tricks people have been trying to elicit the reverse information anyway, but it's damn difficult at best, and mostly just isn't there if it wasn't in the original training data. The obvious thing to do is to put in the training data, was my thought. Obviously you could do a search for, okay, here's a fact, does the reverse version of the fact make logical sense? If so, does the reverse version of the fact represent the probabilities properly that it's elevating the correct thing? If not, synthetic can create training data that contains the information, literally reverse the sentence, put sentence in the training data a second time, run the thing again, and it's expensive, but it would work. The question is, is it worth doing? The answer in many cases will obviously be no, but yeah, I think it's more interesting, not because of practical implications of this figure were failure, because it points to the fact that it's not the origination of people thinking properly inside, the machine is not thinking in a way that a human would think because it would be able to accomplish this thing, but at the same time, anyone who has learned a foreign language knows that just because you know that Shalom is Shalom, does not mean you know that Shalom is Shalom. It does not work that way, right? These are 2 separate facts in your head and they help each other, right? It makes it easier to get the other 1, but the flashcard, you have to have both flashcards, right? If you only have the flashcards that you see the English word and you say what the Spanish word is, and you go to the test, and then the other half of the test is the Spanish word, they ask you the English word, you're gonna do very well in the first half of the test and very badly in the second half of the test. So it's not just LMs.

Nathan Labenz: 1:23:22 Yeah, it's funny how sometimes we're pretty harsh on the AIs for struggling with some things that we too frequently struggle with.

Zvi Mowshowitz: 1:23:30 I mean, the idea is that, yes, if you use this fact continuously all the time, you would in fact pick up on it. What's most interesting is the probability underlying, right, because it doesn't shift at all, right? It's this idea of the element will evaluate the probability of every possible next word, right, every possible continuation, and they're exactly the same before and after the training, if the training was never reversed, if you checked on the reversal, right? Whereas if you ask me the probability of these various continuations, that will jog my memory or sound familiar or whatever, I will not have no helpfulness here. There'll be something going on, we understand that, we made the connection.

Nathan Labenz: 1:24:12 There's a couple of things that I thought were super interesting about this. So yeah, the training data fix is definitely a pretty I think that 1 is probably already happening. I would guess that GPT-4 might even have some of that going on, maybe not in the deep long tail, but some of the recall that I see on just mid long tail Wikipedia articles, and I've tested some stuff just out of random sports trivia, tell me the story of the because Wikipedia has these pages for every football team, the story of this season. Tell me the story of this season for this team. Right? And it will get things right almost perfectly, the number of yards a particular guy had, the amount of time on the clock when a particular play happened.

Zvi Mowshowitz: 1:25:04 The line behind that is you would presume, right, if you're training on essentially the entire open internet, that there are gonna be a lot of different sites that basically describe the history of sports in great detail, and they're not gonna have consistent language, they're not gonna go in orders when they describe games and events and associations, and so B is A is in there somewhere. The wrong direction is covered, and so they're going to be okay. That'd be my guess. I think if you remember it made sure that didn't happen, you would see a problem.

Nathan Labenz: 1:25:36 I wouldn't be surprised if GPT-4 included some sort of strategic approach to a few trusted datasets like Wikipedia where they basically said, okay. We're gonna rewrite this 10 times and train on all 10 versions because they wanna have a little bit more anchoring into something.

Zvi Mowshowitz: 1:25:57 So much better training data anyway, right? It's just so much more valuable. You'd want to train on it a 100 times or 1000 times more than you would other things regardless. So you'd might take advantage of that and have versions where every time there's an A is B, you flip the B as A, and that makes a lot of sense. I could see that happening. This is 1 of the things where we say, right, training a large language model properly is not just about scale, it's also about these hundreds or thousands of little tricks, and something like this is probably 1 of their tricks in some form.

Nathan Labenz: 1:26:30 This is what you pay OpenAI for and what does not include, or at least things like this, as you say. So another idea I had, and this is just very speculative, but we have such a broad view of the space. And I just have this general sense that almost anything can be made to work. So I just said to myself, okay. How would I fix this? And I'm not constrained by the fact that it might be super inelegant or hacky or weird or whatever. What might I do to fix this? And I was okay. Well, if it's all stored in this directional lookup table sort of structure, And we kind of know roughly where that is in the models from a bunch of different studies where it's in the middle layers ish. Right? The maybe the third quarter of layers is where a lot of facts seem to get stored and looked up. And, certainly, I'm sure OpenAI has a better sense of that than I do. What if they just took some middle layers where a lot of these relationships are stored and literally just Frankensteined it by saying, okay. I'm gonna take these middle layers. I'm going to flip them. I will then maybe have a couple double wide layers in my lookup phase of my model. And then maybe I'll even keep everything else frozen and just train the model to start to make use of this prosthetic, if you will, lookup that is the reverse. It's just randomly stitched on. And we're just going to allow a few things to vary, maybe in that module perhaps, or maybe just how information feeds into that module, maybe a little sequence of those kinds of things, I feel you could probably get even something as half baked as that to work and maybe overcome that sort of problem. How realistic does that sound to you?

Zvi Mowshowitz: 1:28:26 My gut tells me it's the kind of thing that sometimes works, right? Or sometimes some tinkered version of that works. The first version might not work, but if you tried to turn all the knobs right and you banged at it for a month, then you had some help and you learned a lot of compute testing, you'd find something that's helpful, but also the kind of thing that often just doesn't work and people don't have a principled understanding of how to predict whether it's going work or not work, and the only thing you can do is try it. And a great engineer, a great machine learning person will be able to tell you with much better calibration, the chance it will work, right? Or a little bit of thing we tried that already, didn't have any success, so every chance is much lower than we just, not 0 because we might have just missed something, they might just had a knob at the wrong place and maybe it's not at the right place, suddenly it works, I don't know. But yeah, my guess is a good effort with that, maybe there's 10% chance or 20% chance that that does something interesting. You'll never know until you try. It's not something I have anybody to try. That seems way more compute and time and effort.

Nathan Labenz: 1:29:34 Yeah, even if you froze most of the model, just running the forward passes would take a significant amount of compute, so I do think that would be tricky.

Zvi Mowshowitz: 1:29:42 Yeah, it's a question that of developing, how do you figure out how to know if you are doing it right quickly? How do you know if this thing is progressing? Can I teach it 1 specific fact this way as opposed to can I present a very number of benchmarks? And then if that's true, then you start expanding the process and can it retain and can it scale and blah, blah, I can think of various things to try, but also I'm constantly, I have this thing where I think I thought experiments like this and I get really excited and I get nerd sniped, right? And I got this is great stuff, I'm so into this, so let's see what we can do to make this thing capable of doing it, and then I go, oh right, that's bad. I don't want to teach these things how to do things, I prefer they develop slower, that they are less good at these things, and therefore I do something else with my time. In the world in which I didn't worry about what was going happen if we developed really capable AI models, I think I would have just applied to OpenAI at some point or DeepMind. When I work on this cool stuff, it's fascinating. Unfortunately, I don't want to help because I don't want it to happen, so no.

Nathan Labenz: 1:30:53 So I think I would probably put my percentage chance of something like that working a little higher than yours, probably still under 50%. I was thinking somewhere in the quarter or third.

Zvi Mowshowitz: 1:31:04 And it is 1 of those places where it's kind of high, it's very high praise, right? To say 10 to 20% chance this would work. That's really aggressive. It's a 50% chance it would work before you tried it, yeah, my guess is that if you talk to Ilya or someone who's regularly does this sort of thing, but no, you never say that.

Nathan Labenz: 1:31:21 Yeah, it's interesting. Guess my main takeaway from that is I use that as a way to interrogate my own expectations about how much low hanging fruit remains and what are the next couple of years going to bring. In all likelihood, I'm not going to go do this either. And I definitely share a lot of your AI fears on a multiyear timescale. But that thought experiment to me is perhaps most revealing of, yeah, I really do think there's still a lot more to come because even totally half baked, spur of the moment things like that seem like they probably not probably, but at least have a decent shot of actually working. And if that's true, then we're just in for a ton of acts in every dimension and enough of them will work.

Zvi Mowshowitz: 1:32:12 I mean, expectation is we're gonna see a lot of ways to, they call it algorithmic improvements, right? Just ways to train these things more efficiently or better to do things they can't currently do. And to the extent that they don't increase the sort of central amount of effective intelligence in the system, that's probably largely good, in the sense that they do, I'm scared as hell.

Nathan Labenz: 1:32:37 So that's a good transition to my final topic. And I, again, welcome any extras from you. I always appreciate your commentary on the discourse, which I often find very funny. And there's a lot of, if you can put aside the existential dread, there's certainly a lot of humor to be found in the ways that people are talking past each other online. The thing that stood out the most to me, though, from this last rundown of discourse was Conor Leahy at Conjecture, who I'm broadly a fan of. I just think he's super compelling to listen to, extremely articulate, has done great work with Luther before this as well. So it caught me by surprise when he said, well, we should have a ban on models of 10 to the 24 FLOPS, which is basically what GPT-4 is understood to have been. And so somebody responded in the, well, what happens to GPT-4 if that ban gets put in place? And he said, well, ideally, it would be deleted and rolled back out of proper precaution. And I'm paraphrasing slightly, but close to a quote. But I'm open to perhaps grandfathering in already existing systems. And, of course, people lost their mind on this. So if anything is outside the origin window at this point, this to me seems maybe the 1 thing that is hard to get people to get on board with. What was your take on that discussion? And maybe you can tell me how you think the discourse is evolving in general. Nathan Labenz: 1:32:37 So that's a good transition to my final topic. And I, again, welcome any extras from you. I always appreciate your commentary on the discourse, which I often find very funny. And there's a lot of, if you can put aside the existential dread, there's certainly a lot of humor to be found in the ways that people are talking past each other online. The thing that stood out the most to me, though, from this last rundown of discourse was Conor Leahy at Conjecture, who I'm broadly a fan of. I just think he's super compelling to listen to, extremely articulate, has done great work with Luther before this as well. So it caught me by surprise when he said, well, we should have a ban on models of 10 to the 24 FLOPS, which is basically what GPT-4 is understood to have been. And so somebody responded in the, well, what happens to GPT-4 if that ban gets put in place? And he said, well, ideally, it would be deleted and rolled back out of proper precaution. And I'm paraphrasing slightly, but close to a quote. But I'm open to perhaps grandfathering in already existing systems. And, of course, people lost their mind on this. So if anything is outside the Overton window at this point, this to me seems like maybe the 1 thing that is hard to get people to get on board with. What was your take on that discussion? And maybe you can tell me how you think the discourse is evolving in general.

Zvi Mowshowitz: 1:34:17 I think if you'd said that 6 months ago, let alone a year or 2 ago, it would have been completely absurd beyond the pale, completely outside the Overton window of reasonable discussion. I think now it's on the edge of, there's more than 1 Overton window. In some centers, the Overton window of things that might actually soon pass into law, and there's the Overton window of things you can talk about without people just completely calling you crazy. And it's definitely not in the first 1, but I think it's in the second 1 at this point. I think that people who are saying, no, we have to stop training bigger, more powerful models, have definitely made their point and people were talking about it, or until some point we would have to do that and people were pointing out tangible problems of that, reasons why you, things that would get, what would you sacrifice? How would you be able to sustain what you're trying to do? Would it actually work? But these are talking price, these are fact questions, these are model questions, they're good questions, but that's different from just dismissing out of hand what you would have a year ago, what are you even talking about, that's crazy. And we've even gotten into any of those questions. And when Conor says, I think it's time to draw the limit this low, it's because Conor and the rest of Conjecture do not care where your Overton window is, they care what they think would cause us to die versus what they think would cause us to not die, and their model says that this is about the point where it stops being safe to have models lying around. At the same time, because algorithmic efficiency will continue to advance. So what you do with 10 to 24 flops back 2 years ago when you trained GPT is very different from what you do with 10 to 24 flops in 3 years from now. And so the idea being, well, if you grandfather a GPT-4, we're still gonna make something better than GPT-4 reasonably soon, just by using the same flops more efficiently or even less flops more efficiently. So it's not for all time GPT-4 just owns everyone, it's just we're not gonna make you delete this thing. And I think that's a reasonable argument to say, okay, we're gonna let you. I think the ideal limitation regime is some limit that I would set right now to be maybe 10 to the 26 myself or something like that, then yes, it decreases over time. It doesn't just not go up, it actually goes down until such time as we're convinced that we have some way to contain what we're gonna call up if we go over that limit, because as algorithmic efficiency improves, what you can do with that number gets better, so you need to have the number go down, which means yes, you're gonna have to track the compute a lot more carefully, you're gonna have to restrict things in other ways more carefully, and we can have all the discussions about what that entails, what it would take to do that, and how practical that is and what we're willing to sacrifice and what steps are necessary. But yeah, I think the broad idea of we can't just be running around creating hyper effective, hyper capable models at a certain point because we're starting to enter the realm of potential really, really dangerous misuse already. Very clearly we're on the edge of that if not already in it. Soon after that, we could at any time, with any new model we step over the line, start entering the realm of recursive self improvement, exfiltration, replication, agentization in a way that's difficult to, because you can call it but you can't put down or putting it down would require inflicting huge damage in our technological civilization. And I think we have to seriously think about how much of that risk we're willing to take to have our nice little nice things in the margin that we want and what we're prepared to do to not make that trade if we don't wanna make that trade, and then we have to also think ahead of, okay, we create this now, what does that do 5 years from now when we've got much better algorithms, much better scaffolding, much better ways to what we can do with that and how does that fuel our rates, how does that fuel these dynamics, how does that set principles and momentum that's hard to stop? So I think that we've made tremendous progress in the discourse over the course of the last 6 months, to the point where people are saying many very reasonable things and looking at very reasonable solutions and are starting to actually ask the practical implementation questions that we just didn't even have the ability to ask before because we couldn't get this discussion under way without being laughed at.

Nathan Labenz: 1:38:53 Yeah, it's gone in some ways remarkably well. Remarked many times, maybe even to you the last time we did this, that it is very easy to imagine a much worse state of play where all the leading developers dismiss any concerns. And it's also pretty easy to imagine a somewhat better state of play, but it's hard to imagine a much, much better state of play as we kind of, I kind of think of us right now as getting to the end of phase 1 and sort of beginning chapter 2 of the AI story.

Zvi Mowshowitz: 1:39:32 The obvious alternate universe that's much better is something like, DeepMind, there's no OpenAI, there's no Anthropic, they never came to exist. Either DeepMind exists or nothing exists. Google has a very large lead in artificial intelligence, nobody is seriously trying to compete with them. They are taking it very slow. They are releasing things well after they are created very carefully. There's no big boom. There's no big huge amount of investment. And the information about the techniques are highly guarded. People who try to create things outside of Google find that they don't get very far, and we were proceeding very slowly and carefully and they have, maybe Demis is the CEO of Google in this hypothetical better scenario and he understands exactly what he has to do. I'm just trying to come up with an obvious better situation where we have 1 highly responsible player with a large lead and no serious competition. That would be my guess at the better world that we might've gotten to from 2014 or whatever. As opposed to a better world that has to start in 2000 or something crazy.

Nathan Labenz: 1:40:37 How much do you think people would have to see in terms of, you see people like Gwern who, at GPT-3 and probably before, probably as of GPT-2, but certainly as of GPT-3 was calling, yep. This is, I think I know what track we're on, and it seems to have been largely right. It seems like in that scenario, honestly, if they just show much of anything, it seems like the mere knowledge that somebody has created an AI and it does some interesting stuff would almost have been enough to kick off a wave of investment unless you kept it a total secret, which seems almost impossible to do given the scale that they'd have to be operating on, the number of research scientists they'd have, just the general sort of non enforceability of non competes in California. It seems very tough to not have people kind of take notice or kind of seep out that, yeah, some really interesting stuff is happening if you throw a lot of data into a big enough compute blender. Especially because there's a lot of know how, certainly. And we've given a lot of appropriate credit to the likes of OpenAI for productizing things really well compared to a lot of other things we see. But there's also not a lot of know how that's required to get something really interesting. So when I kind of think how hard would it be for them to keep that secret that there's a pretty easy way to get to something interesting? It seems awfully tough.

Zvi Mowshowitz: 1:42:13 Nobody cares about your stupid startup idea is the counterpoint. There's lots and lots of great ideas that just sit there for decades not being picked up by anybody. If there's no obvious path, there's no marginal gradient to train on, there's no commercialization, nobody has put the proper chat interface in front, Google has the lead, they're risk averse legally, culturally, they're not putting anything out in this hypothetical world. I think it's very possible that we lose that situation and we end up in a far worse situation because people who are less responsible become the competition. I don't think it's impossible that this stuff's hard, maybe you don't publish the transformers paper, maybe you don't explain these things, maybe it's kept as a trade secret, I don't know. I don't think that these things are impossible, I think it's too late now. I mean, I don't really worry about it too much. If in the past, we had our opportunity to not be in this world, I sometimes think about, well, rationalist style people, Eliezer Yudkowsky and Zvi Mowshowitz, directly inspired DeepMind and OpenAI, Anthropic, what would the world look like? And maybe better, maybe worse, I don't know. If we got other people who didn't understand the risks, who were doing the same projects, even if they were a year or 2 behind, that might be a much, much worse situation, or maybe we just wouldn't. My experience from Magic the Gathering, a world of opportunity everywhere, lots and lots of innovation is some innovations, day 1, 100 people come up with the same solution. Others, you show up at the tournament after all the work, 1 person had it, 1 team had it, or even 1 origin and then for years they toil in isolation with this thing that doesn't quite really work except when you really, really know your stuff. And then eventually someone breaks through and figures out the missing piece and it's off. And until it proves itself, never underestimate the ability of pretty amazing ideas, especially ones that require lots of investment upfront time and taking a loss for a while to just not get to work. OpenAI clearly pursued things in a way that nobody else was pursuing them for years, and they knew OpenAI was doing what it was doing. It was very, very open about it. So if there's no OpenAI, well, we shouldn't think anyone else would have done it.

Nathan Labenz: 1:44:44 Yeah, it's possible. Radical uncertainty on so many questions. Anything else you want to cover today?

Zvi Mowshowitz: 1:44:50 It's amazing the things that we haven't mentioned. Anthropic got a $4,000,000,000 investment from Amazon this past week, and we didn't even talk about it. Sam Altman said on Reddit, AGI has been achieved internally, and then tried to walk it back with the, you guys have no chill, obviously I was joking, as if it was okay to joke about that. In that way, completely deadpan, and he never wrote other things, but yeah, these are the big things and we can always make it happen to this and check back in another month.

Nathan Labenz: 1:45:20 Yeah, I'm looking forward to that. Well, just since you mentioned the Anthropic Amazon deal, just briefly on that, last time they raised money from Google, my reaction was, where is Amazon and how is the valuation not already significantly higher? I think it was a $5,000,000,000 valuation, whatever, 6 months ago or something. And I was, that's gotta be worth more to Amazon who doesn't seem to have a horse in the race yet, as opposed to a Google that obviously has DeepMind and just deep expertise in this leadership in this field anyway. I guess if I was, if I'm gonna update on anything based on this deal, I would say it does suggest Gemini is probably pretty good. If they were looking at Gemini and being, yeah, it's still not measuring up, then I would think there would be an even bigger bidding war and Google would not wanna lose. And I'm sure they must have had some, maybe not, but usually you get some right of refusal.

Zvi Mowshowitz: 1:46:21 I think you're giving too much credit to Google for being a unified entity that can update on information and act reasonably here. So I don't think the evidence is as strong as you think it is, but it's definitely evidence that other major bidders didn't see this as a desperation situation where they had to get the Anthropic Alliance. And if I was Microsoft or Google, I would have been, even if you have something amazing, just to get Anthropic under your umbrella. If they were, we also don't know if Anthropic would have made those deals. They might've just not wanted those partners for various reasons, including just safety risk. Maybe they weren't going to give them the kind of guarantees that they needed and the kind of corporate control that they wanted and Amazon was. But yeah, this is the steal of the century. This was completely absurd, valuing Anthropic at what was it 8,000,000,000 or something pre money and letting them get a huge portion of the company for a few billion dollars. This is worth 10 times that to Amazon. And to be fair, it's probably underrepresenting what was paid. Amazon almost certainly made a gigantic commitment of compute at highly sub market prices for Amazon Web Services as part of this bid. So it could be something like, okay, we're gonna get a large board for this company for not that much investment, but also your compute costs have now gone down by a factor of several, permanently or something like that. At which point, maybe it's a much, much better deal than a similar investment from Microsoft.

Nathan Labenz: 1:48:06 The non standard terms also are not, they're kind of alluded to, but not spelled out in detail as far as I know in public so far, also probably do take a lot of would be investors out of the game. Because you can imagine just hedge funds galore would be willing to Nathan Labenz: 1:48:06 The non-standard terms also are not, they're kind of alluded to, but not spelled out in detail as far as I know in public so far, also probably do take a lot of would-be investors out of the game. Because you can imagine just hedge funds galore would be willing to.

Zvi Mowshowitz: 1:48:22 Yeah. I don't think they keep the investors out of the game so much as they lower the price that people are willing to pay. I think that at 8,000,000, I cannot think of very many reasons why you wouldn't put as much money as you possibly could into Anthropic. Basically the only one I can think of is, you think it's bad for the world for Anthropic to get money, and you don't want to do it. Or you just have liquidity preferences. You can't invest in something that can't mature fast enough. But this idea that you won't be able to sell your shares for a large profit, even if Anthropic plays this we're never gonna make a dollar game.

Nathan Labenz: 1:48:57 You know who's famous for not making a dollar for a long time?

Zvi Mowshowitz: 1:49:01 I mean, also Meta, also Uber. There's a lot of companies that, in some sense, should have a zero price target because no matter how much money Meta makes, it's all gonna go to whatever Zuckerberg thinks is the cool shit. He's never gonna pay the shareholders, right? The shareholders are all, well, when he dies or something, eventually this company will get to make some money. You're gonna act as if this thing is worth what the net present value of its profits are, as opposed to those profits not actually ever getting to you. And if everyone else treats it that way, then it's worth what it's worth, so it's fine. It's weird.

Nathan Labenz: 1:49:37 Right? There's definitely a few, the Harari style fictions. There's definitely a few fictions in play here, one of which being that tech companies may one day pay dividends. Alright. Well, yes, I look forward to doing this again. One thing I do wanna do with you is a kind of survey of the AI safety landscape because we've both kind of studied this in different times and in different ways from different perspectives. And that is also something I think people would be very interested to hear about. This is kind of this high level sense of, who's out there? What are they doing? Does any of it have any chance of actually working? Key question.

Zvi Mowshowitz: 1:50:14 I'd be happy to. It's definitely not a thing where even if you are working on AI safety, it's very, very difficult to know exactly what else is out there and often difficult to assess whether or not someone else's project is useful, is real, is genuine in some senses, is co-opted and is actually capabilities or more harmful than it's worth, or pursuing something that's not central to the problem and therefore can't help with the ultimate goal. So it's very, very all tricky. Anthropic is the big puzzle of all, right? Is Anthropic basically a safety org where if you help Anthropic, you're helping advance safety? Or is it probably a capabilities org that is more balanced in terms of how much safety it's willing to do than its rivals, and has a better culture of worrying about the capability of this building while it's building them, but still ultimately building them? And I think that's an open question.

Nathan Labenz: 1:51:12 Well, we'll save that for next time. For now, Zvi Mowshowitz, thank you for being part of the Cognitive Revolution. I love it. Alright. It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.