Full Transcript

Transcript

Nathan Labenz (0:00) Hello and welcome to the Cognitive Revolution where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week we'll explore their revolutionary ideas and together we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz joined by my cohost, Erik Torenberg.

Nathan Labenz (0:22) Hello, and welcome back

Nathan Labenz (0:23) to the cognitive revolution. Today, we have a timely and important discussion about SB 10 47, the proposed AI legislation that's working its way through the California state legislature, also known as the safe and secure innovation for frontier artificial intelligence models. To understand this complex and all too often misunderstood topic, I've invited Nathan Calvin, senior policy counsel at the Center for AI Safety Action Fund, which is 1 of the bill's cosponsors, and Dean Ball, research fellow at the Mercatus Center, to share their perspectives. We're also joined by Steve Newman, a self described recovering entrepreneur who is now working to raise the level of discourse around AI issues. To set the stage, I can say with quite high confidence based on my own close reading of the bill and related analysis, including that of friend of the show, Zvi Moschowitz, that contrary to some of the noise being made online, this bill is not some sweeping anti technology power grab nor is it the cynical ploy to protect industry incumbents from competition that some have alleged. As some evidence of that, consider that California state senator Scott Wiener, who is advancing the bill, is probably best known for his pro housing work. And he's also proposed legislation to hasten the transition to solar energy, to extend alcohol sales hours in bars and restaurants to 4AM, and even to decriminalize psychedelics. What's arguably most striking about AI discourse in general is how the strangeness of the technology creates space for people to think for themselves and how that often results in the scrambling of familiar political coalitions. At the same time that we have ostensible political opposites like Tucker Carlson and Snoop Dogg emitting similar sound bites, our discussion today is among people who are generally very pro technology and very skeptical of government regulation. As you'll hear, while Nathan and Dean do still ultimately come down on different sides of this legislation, they identify many, many points of general agreement along the way. Personally, while the bill may not be perfect, I appreciate that Nathan and his co sponsors are making a genuine effort to focus on the critical issue of tail risks posed by advanced AI systems and not placing an undue burden on startups and smaller players in the AI ecosystem, even though that means leaving other important questions, including the risks associated with algorithmic bias, the future of professional licensing, and the general impact on the workforce at large for another day. At the same time, as someone who has long identified as a libertarian and often been frustrated by government overreach, I'm definitely sympathetic to Dean's concerns that a new agency could exploit the bill's ambiguity to gradually expand its scope and power in very problematic ways over time. Ultimately, it seems to me that some form of government regulation of AI is inevitable. And so I feel that it's well worth genuinely engaging with and attempting to improve SB 10 47 as it goes through the legislative process. With that in mind, toward the end of this conversation, in search of a productive synthesis, I put forward and we discuss several possible amendments that I hope might make the bill a bit better. Overall, I think you'll find this to be an unusually respectful, nuanced, and insightful discussion on what might be the defining policy issue of our time. If you agree, I'd ask that you take a moment to share it

Nathan Labenz (3:40) with your friends.

Nathan Labenz (3:41) And on this topic in particular, I really wanna hear your best ideas for how government can help foster a positive AI future. I have an interview scheduled with senator Wiener for next week, and I will definitely read everything that you send me and attempt to bring the best ideas to him directly in that conversation. You can reach me, as always, via our website, cognitiverevolution.ai, or by DMing me on the

Nathan Labenz (4:04) social media platform of your choice.

Nathan Calvin (4:06) For now, I hope

Nathan Labenz (4:07) you enjoyed this deep dive with Nathan Calvin, Dean Ball, and Steve Newman into California's AI bill SB1047. Nathan Calvin, Dean Ball,

Nathan Labenz (4:17) and Steve Newman, welcome to the Cognitive Revolutions.

Dean Ball (4:20) Thanks so much. Happy to be here.

Nathan Labenz (4:22) Likewise. So we're gathered together for a discussion about the proposed legislation that is working its way through the California state legislature, which is known as SB 10 47, or as the authors want it to be known, the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act. This legislation would be first of its kind. Obviously, we've seen the executive order, but there hasn't really been anything passed through a legislature that would try to meaningfully govern what's going on at the frontier of AI. And it's predictably been the topic of a lot of discussion, debate, seemingly some miscommunication, seemingly some misunderstandings as well. So I wanted to bring this group together just to explore those and see what we can potentially hope to come to some agreement on. Also, because it's worth noting too that the we're not yet at the point of an up or down vote. So in terms of how to approach this today, I got a bit of outreach from senator Wiener's office in California, and the guidance there was like, this is probably gonna play out over the next several months. There are gonna be a lot of opportunities for people to make amendments. So we're very much still in the discussion phase. And, potentially, if we come up with anything that we all feel good about, we could even recommend that to the authors. So that would be a a big win in my book. For starters, Nathan, you are the senior policy counsel at the Center for AI Safety Action Fund. You guys are 1 of 3 cosponsors behind the bill. You wanna just start off by giving us a little bit of an introduction on who you are, who the organization is, your kind of background worldview, and why this is something that you're motivated to work on? Then, Dean, I'll I'll give you the a chance to give a similar story next.

Nathan Calvin (6:10) Yeah. Absolutely. So to start with the organization, so the Center for AI Safety, Action Fund is, like, the 5 0 1 before affiliate of the, Center for AI Safety, which others might know from the statement we put out about, issue risk for AI as well as a lot of the, technical research work that this vendor does, including by our executive director, Dan Hendrycks. The 5 0 1 3 4 was partially created because we were getting lots of inquiries and incoming from quality makers, including senator Wiener, and we wanted to have a vehicle that could do more direct quality work, which is something that 5 0 1 p threes aren't able to do. And, yes, typically for involvement in the bill, senator Wiener put out an intense bill mid last year talking about being interested in doing a a bill on these issues, and she approached us and the other cosponsors to help him really flesh that out in something in this space. But I think senator Wiener became concerned as a lot of us have that it seems like the pace of development in AI is really exciting, I think has a ton of benefits. But I think we are starting to see some of these glimpses of these really large models that have pretty dangerous capabilities. I think we've got similar discussion at the AI executive order, and I I think that's important, but I think there are things that you can only do in statute. And California is an important jurisdiction, and both seem like a really great opportunity. And, yes, been really impressed with him as an author. He's a really smart guy, he's been really into the details on this, and it's been really just a pleasure to work with, and I'll see our other co sponsors of the Build, Economic Security Project, and Incred Justice. Yeah. Just speak a little bit about my own worldview about this. I think that I am just very uncertain. I think 1 of the main things, which I think is that I have a lot of excitement about AI. I think these tools are really amazing. I use GPT 4 and on a daily basis, and I'm a big fan of the models and think that they have a ton of potential to improve our lives and work and to improve bring a lot of innovation that we desperately need to solve really hard problem. I'm also someone who in general is, like, very, I think, critical of unnecessary regulation or bureaucracy, big fan of kind of the remainder housing work and, trying to build additional your energy of trying to reform the FDA. A lot of the things that I think you and I on a separate podcast to probably chat about and not a lot to to agree on. I just happen to think that really the largest AI models that in addition to their potential to do a lot of good, I think they might be able to do just really powerful thing and that we should be having some amount of of precaution and kind of leaving it totally up to the companies and voluntary measures about whether they wanna be doing testing. I think we're you're in group both in the labs that there's just tremendous commercial pressure to get things out the door as as fast as possible. And I think there need be a role for governments to say, let's be thoughtful here. Let's get the benefits, but let's make not make sure that we're, you know, raking ahead and and causing catastrophe. I think the 1 thing I've, you know, wanna say, I'm a very big fan of the general structure of the bill and will defend that very strongly, but I think a lot of the details about the point, like, the complicated issue of the bill has already been amended. It will continue to be amended, and I really speaking in general, I'm gonna be speaking for myself and then I'm gonna be the action fund, but on this area, I'm confident the author also shared this that he's really interested in amendments. We want to get this right. There are a lot of different technical terms and definition. We have our clear intent, but we wanna make sure that those definition reflect what our intent is, and it's really to target specific, very dangerous, concurrent models while really generally let letting the AI ecosystem bloom and flourish has been continued. So that that's my general standing. I think I'm worried about a variety of different risks. I think I'm worried about, like, a super powerful, agnostic model that just empowers humanity. But I I think more and more, about, like, the ability for bad actors to use this for for malicious bio attacks, really severe attacks on critical infrastructure. I think some of the risks that were identified in the Biden executive order from, like, dual use foundational model that I think similarly novel DDRN developments, attacks on critical infrastructure, and models that, like, deceive or self replicate or or otherwise really hard to get back control of. So that's what we're, you know, broadly thinking. I also care about, and a lot of other issues also really important with with an AI and a lot with different of things about deepfakes and the future of work and racial bias in government systems, and all those are super important that I think are all important discussion we had. But this bill is really focused on kind of these catastrophic national security risks from the largest models and similar in DNA to what was attempted in the Biden order. But that's broadly where I'm coming from. I think that we do need to take some action here. I think that, like, we really are on an exponential curve, and I think that there's some temptation to be like, this is really complicated. We could wait a year or we shouldn't wait until we can make everyone incredibly comfortable with it. I think legislation takes a long time. We already have in the legislation, and as I I think some folks have noted some, like, delayed implementation or even after the bill passed, it takes a while to get government set up around these things. And I really do think that if it all possible to pass something this session so we can start that process going. And I really think that not how it being interpreted and on Twitter and think that, but I think there are already a lot of concessions in this bill and trying to be a moderate, not maximal proposal. We can get more into all this. That's broadly where I'm coming from.

Nathan Labenz (11:27) Quick follow-up question. When you refer to the author of the bill,

Nathan Calvin (11:30) is that state senator Wiener? Senator Wiener. Yeah. It's terminology is weird because at the federal level, you refer to the bill author as a sponsor, which is the opposite to when I say the author and senator Wiener. We had sponsored, provided, like, technical advising and not feeling bad amendment to language, but ultimately, him and his legislative director have been very involved in this and then make the final call at all the ingredients in the direction of the bill. And but, yeah, the I'm referring to the offer. I'm referring to senator Wiener and his team.

Nathan Labenz (11:56) Cool. Good. Thank you. Dean, you are research fellow at the Mercado Center. You've been a previous guest on the podcast when we talked about all things brain computer interface. So obviously, a big and pretty deeply technical engagement with technology, AI, all these sorts of things, though your day to day focus is on policy analysis. Give us a sense of where you're coming from on this.

Dean Ball (12:19) Sure. So first of all, I am at the Mercado Center. That is a think tank affiliated with George Mason University. We are a 5 0 1 c 3, and what that means is we're in the same tax status as a university. We try to educate the public and policymakers about matters of public policy using insights from academic research. So that's where we come from in a broad sense. Specifically on AI, I think that AI is a general purpose technology, quite possibly the most powerful general purpose technology that humans have ever invented. Certainly, the most powerful 1 of sort of the recent wave, at least it could be, of general purpose technologies, computing the Internet, etcetera. As a general matter, we don't tend to regulate downstream use of general purpose technologies with a centralized government regulator. That's just not typically how that's done. There's not a department of electricity. There's not a department of the Internet. There is FERC, the federal energy regulatory commission. They set standards for interstate transmission of electricity and all kinds of stuff like that, but they're not regulating my downstream use of electricity, which surely can be used to do all kinds of dangerous things, nor is there a department of computers. I think that if we want the AI revolution to be successful and to broadly benefit people, then even to speak of AI policy as a discrete area of public policy is probably going to seem a little bit anachronistic in 20 years. That's my hope. I wouldn't necessarily predict that, but that is my hope. Because I hope instead that AI is so integrated throughout our economy that it is regulated in all the myriad ways in which economic activity and and public policy interact with 1 another. And so I think that creating a centralized regulator is just likely to lead to all kinds of political economy problems that will hinder the impact of AI, the good impact of AI, in ways that I can't even predict. And I'm not really sure that it gets us that much more safety either in the long term, in particular, because most of the concerns we have here, CBRN, cybercrime, these things, certainly, they could come from domestic actors. Nathan Calvin, you you referenced national security concerns. I don't believe that the CCP cares about qualifying for a limited duty exemption under the state of California nor do they care about the NIST AI risk management framework. So I'm not really sure that this gives us safety in any meaningful way from those types of actors. And instead, I think to the extent it hinders AI progress, it actually creates danger, which is what regulation often does of these kinds. It creates a false sense of security, but in fact, has a tendency to lead to long term danger. I really think that nuclear energy is a good parallel to think about here. Except for at least nuclear plants, there were some nuclear incidents that raised a lot of concerns across the world. There is no AI Chernobyl so far. There is no AI 3 Mile Island. And we created the nuclear regulatory commission and illegalized nuclear energy effectively, after some major incidents. There have been no such incidents here. So if we're gonna illegalize AI progress, I'd at least rather do it after we have at least 1 concrete example of a major problem. I I just think not only is it premature, but it's the wrong way to go in general. So that is I just wanna be very clear, though. I am not opposed to governments passing laws with the word AI in them. I'm not opposed to a vigorous government role, in the regulation of AI. I just am skeptical of the idea that there is 1 big thing you can do, 1 major bill you can pass that will do what you want it to. Instead, I think the answer is every agency internalizing the risks and benefits of this technology in a serious and considered way and taking steps within their own jurisdictions and their own areas of expertise across the city, state, and federal levels. I think there's not 1 big thing. There are 5,000,000 small things, and that is the work that I hope we can do. And I think that, frankly, centralized regulators hinder that work.

Nathan Labenz (17:05) Thank you both. Steve, to introduce you briefly as well, you are a exited entrepreneur. Most people will have interacted from time to time with 1 of your creations, which is now known as Google Docs. And having exited that line of work, you're now focused on raising the standard of discourse related to AI, which is, obviously, a a noble pursuit. I think we're in a pretty good position to hopefully have some elevated discourse here because for 1 thing, I don't hear very extreme positions. Like, you will sometimes go online and sign people saying things like, oh, who cares if humanity goes extinct? The machines will be more worthy and some, you know, sense than us anyway. Not hearing anything along those lines. I'm also not hearing argument from protection of weaver employment, so to speak. I'm not hearing any anybody saying that the current mode of production and the current employment relationships and dynamics are, like, the primary concern. It seems like everybody on this call is broadly pretty enthusiastic about the upside of AI and technology in general and broadly and certainly this is true for myself as well, skeptical about government overreach into areas that are probably best left to develop on their own. Do you have any other thoughts on how we can make sure we have the most elevated discussion that we can?

Steve Newman (18:25) Well, I think the the first step is just bringing the other people who have a positive attitude and are interested in engaging and an understanding that we're gonna have a positive discussion. I think you've done a great job of queuing that up, and I I think really looking forward to seeing seeing Nathan and Dean dive in. And I think we've already

Nathan Labenz (18:41) seen just from the everyone's introductory remarks that we're here to kind of

Steve Newman (18:45) talk about reality and talk about things in

Nathan Labenz (18:48) a in a reasoned way, and and that's 90% of the battle right there. Hey. We'll continue our interview in a moment after a word from our sponsors.

Dean Ball (18:57) I I wanna pick up Nathan Labenz on something you said about protecting weaver employment. This transformation that is coming will be an economic transformation. I don't think anybody has intelligently modeled the labor market impacts of AI. I know some people who are trying. I don't know how possible it is right now, but it's gonna be a struggle because there are a lot of areas of the labor market that are protected with barriers to entry, and barriers to entry will be erected to keep AI from from entering, whether it's driverless cars, medicine, law. There's 1000 things where, you know, current industry groups are gonna try to continue extracting rents, and that's gonna be a real struggle. I think that will be a huge policy struggle over the coming decades. And if you don't believe a centralized regulator will be a weapon that will be used against AI for diffusion. I think that you have not been paying appropriate attention to the way that government has generally conducted itself at the state and federal level for my entire lifetime, at least. So I think that's 1 thing that's important is that I am very concerned about maximizing the benefits. I wanna do that, and I think a centralized regulator presents all kinds of challenges to doing that. That's just 1 additional point I would make.

Nathan Labenz (20:29) Yeah. I think probably you'll have agreement in the general sense that we can find plenty of examples of government gradually overgrowing the original intent of the legislation under which it's operating. Maybe we can approach that from the standpoint as we get into the details of, like, how can we permit that from happening in this case? I often say, well, people ask me, like, why should we want any of this? Isn't this all just gonna be terrible? I'm like, AI doctor. That's my first thing. When I first got my early PPT 4 access, I couldn't stop asking it all these medical questions now that I have that many fortunately. It was really good and so convenient. And that is, in my view, almost the canonical thing where if we end up in a world where we prevent people from getting quality medical advice from AI, whether out of irrational fear or van cost benefit analysis or protecting doctor employment, it seems like we'll have made, like, a very bad mistake. So if that's something we can agree on, then we can get into, that tendency, which definitely, you know, is out there, could be mitigated or prevented. Fair?

Nathan Calvin (21:34) Fair. I guess I just wanna say that I agree with you that there a lot of really large benefits there, and I think it be quite bad to not get them. And I think some of that will involve pages in the future work. I think among the coalition that we're working with here, there are, like, a variety of different use with with it among you. And I feel like to I'm gonna be like, this bill is not necessarily taking a position either way on the, like, labor market future and whether you could be trying to have unemployment. I I have my own views about that, but I think either way, I just don't think what this bill is about. And that if you're someone who is either concerned or optimistic about AI displacing human jobs, then I don't think the bill is necessarily gonna have a lot to say in either direction to be totally honest. Just wanna clarify that. And then also and I think we'll dig into a lot more. I would push back on the idea that this bill is a centralized regulator. It's not a licensing regime, and the only government body that has created a new division of the California Department of Technology is just, like, receiving a court and certification. It doesn't have rule making power. It doesn't issue licenses. Most of this is a liability regime through the attorney general. So that still is probably more centralized than you would like, but I think there are other proposals that really are, like, centralized AI regulator where you're getting preapproval to be able to do training run where they really are doing the talk themselves, doing a lot of that. I agree that there are really big challenges of that, of having government capacity to really understand all the things going on in the cutting edge and doing that effectively. And and I just wanna say, I also, I think, have confirmed about the efficacy of a centralized regulator, particularly given the existing technical expertise or lack thereof within governments. And so I'm sure we'll continue to debate about how we're characterizing the bill, but I I also have hesitation about a centralized, government regulator here, and I I don't think that's actually

Nathan Labenz (23:12) what this bill does.

Dean Ball (23:14) Only thing I would say is the frontier model's vision, in a recent amendment to the bill, does have the power to change the compute threshold. It has the power to change the specific standards for models that fall under its jurisdiction. So that already is starting to look a little bit centralized. And the other point that I think is under discussed here is that we talk about repeating the mistakes of nuclear energy. The frontier model division, because of California's current $60,000,000,000 budget deficit, the frontier model division is funded by fines and fees that are levied on AI companies, and that's exactly how the nuclear regulatory commission is funded. The regulators the regulated entities pay for their regulation. This obviously creates regulatory capture potential. Right? Like, on its face, their salaries are paid for by the people they regulate. But in addition to that, it insulates the frontier model division from the normal appropriations process and legislatures where maybe 5 years from now, we're not as worried about AI risk. Maybe all the concerns of folks like Jeff Hinton don't pan out, and we wanna lower the budget for that. But guess what? You can't because the FMD funds itself, or it's inconvenient to because the frontier model division funds itself from the AI industry. And so not only does that insulate it from the budget process, but in essence, it insulates it from the trade offs of the political process of democratic input. And I think it's probably worth California think maybe thinking seriously about why they have a $60,000,000,000 budget deficit before they create a new centralized regulator for the most promising technology in the world. But, also, I think that it would just create all kinds of problems, just the funding alone. So come

Nathan Labenz (25:08) back to this question of how centralized the regulator is in a minute. I I have done my own close reading of the bill and broken it down into basically 6 things that I would say it does that seem to matter. And we can then dig into the the details of of each 1, either 1 x 1, or you can add categories to the the list if you think I missed any. First 1, as I think about it, is it sets a standard for what are we talking about here? What is the actual risk that the bill is trying to address? There, it sets what I would say is a pretty high standard of either mass casualties or $500,000,000 or more in damages to things like critical infrastructure. So that, for me was like, okay. That's a pretty big thing. Right? Like, mass casualties, 500,000,000 in damages.

Steve Newman (26:04) I think if you were to

Nathan Labenz (26:05) go ask most people on the street, at what level of harm would you think the government should start to get involved in AI? They would probably come up with something that is lower than mass casualties or or 500,000,000. I don't know Dean would say that you would be worried would be subject to future discretion or lowering of that threshold. But for me, it's a high number.

Dean Ball (26:25) So I don't think I don't know this for a fact. I don't think that the bill allows the frontier model division to lower the hazardous capability threshold. I don't believe it can do that. However, I would say the following things. Critical infrastructure is an extremely deceptive term. Critical infrastructure is it's a little bit Soviet in the sense that critical infrastructure is in fact a larger category than infrastructure. So critical infrastructure includes things like hospitals, all financial services, all telecommunications infrastructure. It is the majority of US economic output. So when you say critical infrastructure, don't think that you're talking about a small subset of things like nuclear power plants or something. You're actually talking about most of The US GDP. $500,000,000 is a lot of money. At the same time, cybercrimes worldwide in 2023 were responsible for about $9,000,000,000,000 of damage. Oh, 500,000,000 not indexed to inflation? That could be not so much money. It already is not so much money. You could probably, depending on how you calculate it, if you knock a wastewater treatment plant out, you don't blow anything up, you don't do anything like that, you just knock its computers out for a couple of days, You've probably done $500,000,000 of damage to critical infrastructure. And, obviously, we don't want people knocking out wastewater treatment plants. I'm not saying that's, like, a light thing. I'm just saying that you should think very carefully. Wastewater treatment plants go down. Right? They go down a lot. Cities issue boil water notices on a regular basis in this country. We shouldn't think that's, like, some unprecedented thing. And just because it happens to be caused by a it's just a little, like, I don't know. It's the new thing. Right? But, oh, it was caused by computers. I don't know. Like, I think that it's a big deal, and we should do everything we can to protect our critical infrastructure and be very serious about cybersecurity. But I actually think $500,000,000 is not as big as it sounds.

Nathan Labenz (28:26) So to read the definition from the bill of critical infrastructure, it says, critical infrastructure, quote, unquote, means, and it is pretty broad, assets, systems, and networks, whether physical or virtual, the incapacitation or destruction of which would have a debilitating effect on physical security, economic security, public health, or safety in the state. Yeah. Yeah. So that is I think we are probably really relying on the number there more so than any sort of narrowness of definition because, as you said, like, a lot of things, you know, would have the potential for debilitating effect on physical, economic, security, public health, safety. Now this is maybe more of a nitpick than anything. I'm not sure it really matters. But for what's worth that 9,000,000,000,000 cybercrime number, I just came across that. And I don't really buy it to be totally frank because that would be, like, pushing 10% of global GDP if global GDP is a $100,000,000,000,000 and US is, like, $20,000,000,000,000. But where that number has been flying around, I've seen that actually used on the on both sides of the debate now, because I've also seen people on the AI safety side being like, look how big of a problem this already is. And I'm like, there's no

Nathan Labenz (29:34) way that

Nathan Labenz (29:34) we're losing 10% of global output to cybercrime. That just seems crazy. I actually saw it at a fundraising downstream 2 long ago for AI assurance tech company, and I was like, I'm gonna have to downgrade your market size a little bit. But

Dean Ball (29:49) Well, no. But because of I don't know how that number is calculated. But because of the broken windows phenomenon, the some of that $9,000,000,000,000 might be accretive to GDP. Right? It's not necessarily that it's a $9,000,000,000,000 reduction in GDP. Costs of things are accretive to GDP. Right? If if I break a window and I fix the window, that's GDP. I would just point that out. And I think but I think that that that $9,000,000,000,000 figure, it might be, like, totally. That's entirely possible. That should tell you something about $500,000,000 in the context of cyber attack damage. How do you calculate it? I don't know. You could there's probably ways you can fudge those numbers.

Steve Newman (30:28) Yeah. So let me suggest we take a step back for the moment. Dean, I'll address this question to you. Should we be regulating anything? Is anything bad at a large scale plausible to happen? Or in in other words, are we debating exactly what's the best way to write this bill, or are we actually debating, is there even a problem here?

Dean Ball (30:49) So I I would definitely say we should be regulating things. I think I have a rather circumscribed perspective on that which government can effectively regulate through a single statute. I think there are many different things that government should be doing to incorporate AI into various criminal statutes and scale up their enforcement of things and increase defenses and all that kind of stuff. But is your question, what do I think the outcome would be if the bill passes?

Steve Newman (31:19) So the bill doesn't pass if no bill passes. In in other words, we're talking about the wastewater plant being knocked offline, and from 1 perspective, like, that could be a really big problem, could cause, I know, half $1,000,000,000 in damage. From another perspective, it's a big economy in a big world and big problems happen. And maybe if AI was a little bit involved there, it doesn't really mean that that was fundamentally an AI issue. Taking a step back from the specifics of half $1,000,000,000, should we be worrying about AI? Is there a problem that we could be regulating whether or not this regulation is the right way to go about it?

Dean Ball (31:52) I get the sense you're asking about, like, catastrophic risk. Am I worried about sort of existential type major major risks from AI?

Nathan Calvin (32:01) I I

Steve Newman (32:01) I I'm not trying to go all the way to existential, but just big stuff. Like, like, where tech that brings the whole US grid down for a month or knocks all the wastewater plants offline or things that start happening in a big way that traditionally have not happened in a big way or just a lot easier to happen?

Dean Ball (32:18) Or rather Yeah.

Steve Newman (32:19) Should it be half 1000000000, or should it be 2,000,000,000? Just is there a here there in in in as you see it?

Dean Ball (32:26) So I share Nathan Calvin's and I think probably Nathan Labenz' general uncertainty, like, fairly strong uncertainty about about the the future trajectory of AI capabilities. So I would say, yeah. Like, we should take the the concept of a cybersecurity attack that takes major electricity grids down for a long period of time. I think that is definitely something that we need to be worried about. I question the wisdom of passing a law that says it is illegal to take down The US electricity grid because I suspect that it already is. And I think that, basically, if we pass a law that says it's illegal for the AI transformation to not go well, well, that doesn't do much for me. That doesn't advance the epistemic ball in any important way because we all want it to go well. We all want this transformation to go well. We don't know what that means. Nobody knows what that means. And if the government just passes a law that says it has to go well, then we all have this epistemic uncertainty, and now you've introduced lawyers into the equation. Not that lawyers shouldn't be part of the whole this whole dynamic, but it's, oh, great. Cool. We can go to court and talk about this. Sounds fun. I just I I question the wisdom and really the efficacy of laws of this kind, but I would definitely agree. The AI existential risk, inherent risk, I'm not all that worried about Claude 4 itself developing its own motives and trying to kill all of humanity or whatever. I think that's a little bit of a Jejun debate, but but I'm definitely worried about future models being used for all kinds of things. And cybercrime is at the top of the list for sure. So, yeah, I there's definitely a there there in my mind.

Nathan Calvin (34:07) 1 thing that I think could be helpful is that I agree with you that in a lot of settings, the right way to handle regulation is to focus on kind of the downstream uses of the technology. I think it's fair to say, look. We're not gonna, you know, place a bunch of burdens on email or the Internet. We're gonna focus on people who use those tools to do bad thing and go after them, and that's appropriate. And our laws are able to handle that. I think there are lots of setting where that worked, and I think that's appropriate. I think there are settings where that doesn't necessarily make a lot of sense. We have things like in in nuclear policy where we don't say, oh, we're gonna go after terrorists who use plutonium or something. We they know the people who are enriching nuclear materials have some obligation to procure those materials and to be thoughtful about it. I think similarly, I'm very enthusiastic about biology. Like, biology has tons and tons of benefits, and I'm very optimistic about a lot of different. I don't think that therefore means, like, we should not think at all about gain of function research and if folks are doing lots of gain of function research, then therefore our solution should be, hey. Let's just go after the individual terrorists or or folks who are who are doing that. I think that we have to have some process also higher up on the drain. I I agree that I think that this is atypical this way, but I think in a lot of cases, it made sense to focus on the downstream applications. I just think that fully advanced, the largest AI systems that have a potential of having these pretty devastating impacts are 1 of the small handful of technologies where we do need to be thoughtful about whether regulations are necessary.

Dean Ball (35:33) I I agree with that general framework you've laid out. The fiendishly difficult aspect of this, though, is that plutonium is not a general purpose technology. It's not a consumer technology that I might use on my phone. So that complicates it considerably. I think I agree with you that there is room for regulation and and a need for regulation somewhere between the downstream use and the creation of an AI model. I actually think AI might just be a little too low in the value chain or the production chain of threats, 1 might say. So a concrete example is, the Biden administration just a few days ago released what in my mind is a very productive framework for nucleic acid synthesis machines.

Nathan Calvin (36:21) Also a big fan.

Dean Ball (36:22) Yeah. Where they basically said, if you're gonna make if if you're gonna do federally funded life sciences research, you have to use machines, DNA synthesis or nucleic acid synthesis machines from specific providers that will comply with these standards. And those standards include that sequences that you wanna produce genetic sequences you wanna produce have to be screened against the database of known threats. Great. Excellent. But I think that sort of going 1 level removed from that to, like, the statistical analysis of genetic data sets and regulating that you know, I'm an American, and that does raise constitutional problems in my mind. Right? I think I do have a constitutional right to do statistical analysis on genetic data. I don't think I have a constitutional right to synthesize DNA, and I think that's, like, a pretty clear distinction that, like, at least most Americans can intuitively grok. Constitution as v Moshewitz has said to me, constitution is not a death pact, so I'm not gonna rely just on a constitutional defense. I will also just say that it's too low to be a productive regulation. It's like regulating metallurgy rather than missiles.

Nathan Labenz (37:32) Hey. We'll continue our interview in a moment after a word from our sponsors. Is it a good time for me to interject the second point of the 6? And maybe I should even just run all 6 down real quick. Okay. We covered 0.1, which is the standard of the scale of harm that would be of concern. That is, again, mass casualties, $500,000,000 damage to infrastructure. The second point is, how would you reasonably expect that you might be playing in that regime? Here, the bill uses the same threshold as the executive order, at least on the first definition, which is 10 to the 26 FLOPs. So if you're gonna train a model that is gonna use 10 to the 26 FLOPs going in, then that is presumed to be a covered model. That is to say, if you can't make a positive assertion for good technical reasons that this thing is going to be safe, meaning it's not gonna be capable of causing these sort of downstream mass casualty or $500,000,000 worth of harm, then you have to do certain things. And those things are you have to implement

Steve Newman (38:40) a lot

Nathan Labenz (38:41) of safety standards during the training process so the the weights can't leak to the public or to the Chinese government or whatever. You have to that includes having the ability to do a full shutdown. I'm not exactly clear on what would time to do the full shutdown, but the idea is, basically, you wanna have an off switch. So if things get crazy, you can flip that switch and stop the process. You have to at the end of that training process again, this is assuming you didn't have good technical reasons up front to say that it'll be safe. Then at the end of that process, you then have to do the testing and say, okay. Now that we've made this thing, let's see what it is actually capable of. And at that point, have another chance to say, okay, actually, we did all this testing, and now we can say, even though we didn't have a theoretical reason up front, we now have a practical reason based on testing that we can confidently assert that it's safe. And then if you still can't do that, you have to deploy only with surrounding safeguards, which are meant to prevent these large scale harms from happening. Basically, you have 3 tiers. You could say, I have a theoretical reason. And I really would love to see that emerge. Right? That would be the real silver bullet, which doesn't seem like anybody's super betting on at the moment, would be like a conceptual breakthrough that, hey, if we do these things, design the system in this way, create the data set in a particular way, whatever, then we can just be confident that this is gonna be safe from the get go. Now everything is awesome downstream. There's, you know, minimal imposition, right, from the bill, if you can make that kind of assertion. But if you can't do that, then you gotta hammer it with a bunch of tests, find out what it is in fact capable of. Again, you have a fork there. Hey. We tested it as much as we could. We determined that it can't do these things. So we're or if still not, then we have to have guardrails, which would be like moderation, monitoring of usage, and potentially also not releasing the weights into the wild, because this is probably 1 of the more contentious pieces of the build. That if you do release something into the wild, then modifications of it downstream are still your responsibility. So you have to be able to assert not only that it won't do this in the form that we release it, but also that it can't be like relatively easily modified to that state. And this is where I would refer listeners back to the FAR AI episode that we did with Adam Gleeve, because they showed that it really doesn't take much in a lot of cases. If the capability is in the model, then the sort of RLHF finishing that they are generally treated with the end of training is maybe enough as long as that stays in place, but very little fine tuning typically downstream can reverse those sorts of safety fine tunings, even in a non malicious way. That was 1 of their most notable findings is that even just taking a a small chat dataset and saying, hey, we know we wanna have you chat in a particular way. Not training it to do bad things, but just omitting the fact that it's supposed to refuse bad things, that that final fine tuning step can, in a way that the developers didn't even intend, reverse that. So it's on the the model trainer to make sure that not only is this thing not exhibited, but that it can't be easily unlocked with downstream fine tuning.

Nathan Labenz (42:02) So that's like the big 1.

Nathan Labenz (42:03) And then the 1 other thing that's worth noting there is, and I think this is also where a lot of the debate and the kind of heat is right now, is that 10 to the 26 FLOPs is like clear enough. But then there's also this other definition that's if you're gonna train a model that could reasonably be expected to be similarly powerful to what 10 to the 26 would get you, then you're also in that regime. And that's where we're like, well, I don't really know, you know, if I'm in that regime or not. So anything that you guys would clarify or refine on my description of that? Obviously, a lot of people have offered their summaries, but how would you grade mine?

Nathan Calvin (42:37) I think good summary. Definitely hear you on the point that, yeah, I think 1 part that that is harder to know is saying, I know I'm creating a model below 10 to 20, but I'm not sure whether it will be as capable as a model at beta buck we are at 10 to 20 20 24. How can I make sure whether I did not have do any things and the offer added vision, which says that basically, if you are surprised by how good the model is, that's fine? You're not in trouble just like impetude the safeguard after covering that you were surprised, and that's okay. And but in just some understanding that, like, maybe in the future, we will have more accuracy about exactly how high performing a model will be prior to training, but at least in the the current regime, we we recognize that you might not know that until after you started until after you trained the model and started doing various thing about the capabilities.

Nathan Labenz (43:24) Just to give people a sense of what is 10 to the 26 FLOPs. First of all, it's not entirely known whether any already trained model has hit that threshold or not. If it has, it's almost for sure happened at either OpenAI or perhaps Google. The biggest meta trained still in training, LAMA 3, is expected to be halfway there. It's funny because it's I think it's been estimated at 4 times 10 to the 25, which is sounds really close, but it's still only 40% of the way there because orders of magnitude. It's the last 1 is always the the 1 that matters the most. So what Meta is spending on this WMA 3 400 b still in training now is at retail price, something like $200,000,000 in compute. They have published a bit of guidance on how many h 100 hours they've put into the first 2 models that they put out. They put in over 6,000,000 h 100 hours into the 7 d b version. And so extrapolating, Andrew had a nice analysis of this, which I'm taking as closest thing available to ground truth for these purposes. And then just putting a price on what you're in the market today, an h 100, if it's not dramatically subsidized, push is usually about $4 an hour for an h 100. So Fair enough. Roughly speaking, you do the back of the envelope math. You're getting to about a $200,000,000 compute budget for the current Lama 3. So you're in you're in somewhere in the sort of hundreds of millions to half $1,000,000,000 just to buy the 10 to the 26 flaunt. Now I know, Dean, you may have some thoughts on this, because I know you've written a bit about how you think this is not a great way to frame this, in part because efficiency is definitely a major driving force here. I don't know if there's any alternative that you would propose. If we were gonna keep the general structure of this, is there anything other than a swap number that would make sense to anchor off of? But in any event, that that that's a sense for the budget.

Dean Ball (45:32) I I would say couple things about the compute threshold. First of all, yes. It is the case that oftentimes regulation will be triggered by a firm size, the revenue that they have, the number of customers that they have, whatever it might be. Firms as a general rule want to get bigger. When you impose thresholds like that, there's ample evidence in the economic literature that when you impose thresholds like that, you create all kinds of weird problems with firms where firms will just skate right below the threshold. The difference is that firms, as a general rule, want to be more compute efficient. So with a threshold like this, you are swimming against the current, and I think you're ultimately incentivizing the frontier model division to lower the threshold over time. And they can do that under this bill. And the other thing is I'm just, for me, it's not entirely clear what 10 to the 20 sixth is based on. It's just I've never really heard a scientific basis for 10 to the 20 sixth watch. So, Nathan, if you have any answers there, I'd love to hear it. But that to me is the main thing about the threshold. But I think we shouldn't be regulating at the model layer. And I think that the fact that it is so hard to come up with good thresholds is make this stuff legible to the state is, like, evidenced in favor of my argument. So I don't have a good alternative solution. My alternative solution would be not doing this and doing other things instead. It would be not doing nothing but doing other things.

Nathan Calvin (47:04) Great. Just just to briefly justify where the 10 to the 20 fifth number comes from. I I think the idea here is just that this is the next generation of models. So not g v 4 and cloud 3 outputs, but really the ones that companies might be releasing potentially later this year or or next year. And I think it comes from the understanding that, like, we don't think that any of the current generation model models are capable of causing these really severe harms, but we think that there are, like, some glimmers or warning signs that future generations that they they should be, like, keeping their minds open to the potential that they could. And I think that's where that is coming from. The stuff like open AI's study on biorests from g p 4 saying, look. Maybe it helps, like, a little bit, but it's really not very much. It is not the stuff that you couldn't find on on on Google, but saying that, look. As we create larger models, we keep our mind open to the fact that it really might provide quite substantial assistance to a layperson in creating a a really nasty biological weapon. The idea is that we have some sense about what the abilities are of of current models. I I think on the lower threshold thing, I think you're talking about it's guidance about what sort of threshold might be done, like, covered under, like, b. It's not a formal rule. It guidance about it. There was consideration about giving them rule making power to change threshold, and we didn't do that.

Dean Ball (48:17) Yeah. I guess I hear that, and I think if we were debating all of this and the current frontier was GPT 3 or GPT 3.5, we would 100% be saying this about GPT 4. We would and and in fact, people did. People said it about g p t 4 and g p t 3 and g p t 2. And not to say that it's not eventually gonna become a thing. I I I am not dismissing that entirely, but it does seem to me like a vibes based argument for imposing a threshold like this of it's the next generation, and we think that could happen maybe. But otherwise, we have no empirical or scientific basis for these claims. I just think we should not create an unprecedented regulatory regime for software, and that is what this is. It is an unprecedented we have never done precautionary principle, generally applicable precautionary principle, or liability regime for software. Never done it in the history of this country. This would be the first time. Software has given us a ton of economic and social benefits, So I think the bar for that is high, and I just don't think we think the next version might be dangerous is a sufficiently high bar for something so unprecedented.

Nathan Calvin (49:25) I do think it's fair that if we were having the conversation around when, like, GTG Freeway would really that people might say, maybe there are some warning signs. And I think, basically, like, eye and then profit and the companies basically did the things we're describing in the bill. Right? Like, they cut through them. It felt they determined, like, Antas, they do the nasty things, and we can release in the form of that. Okay. I think it would have been fair then to say, hey. There's a potential. They should do these types of things, which I think they back did and then released the model and everyone using it, including myself in it.

Steve Newman (49:51) Right? Mhmm.

Nathan Calvin (49:52) Because I think there's some level where you should just have uncertainty. Think at this level in the next generation, we're at that level. I think it's debatable, and I think from where they were sitting with g d 3, think they should have kept an open mind to the fact g v t 4 might have turned out it didn't. I'm glad it didn't, and maybe someone could have made a more bulletproof argument for why it definitely wouldn't. Again, I agree that this is a weird situation we find ourselves in different from a lot of the things. I don't know. We don't have with, like, email, the creators of email and the leaders of all the email companies signing open letters saying it might kill everyone. Fundamentally different thing here. We have these rules of thumb, and we didn't have have worked, but I think we really are, like NVIDIA is worth more than Saudi Aramco or what. NVIDIA, like, went down a bill. This is, like, a crazy, unprecedented, wild thing that people are promising might be able to solve all these crazy problems and do all these insane things. And I think it's not unfair to say that our, like, normal intuition for how to regulate general purpose technologies or software, we I think there's reason to think that these really incredibly large models that capability that we don't understand is is an area where our intuition should run.

Nathan Labenz (50:54) Yeah. 2 important things there from my perspective. 1 is that this is qualitatively an unusual situation. All the effort that is going into making the models agentic or capable of midrange planning and overcoming barriers, finding solutions to the the first problems that they encounter. This is definitely a a horse of a different color compared to normal software. We're we're only on the verge of really entering that moment. G p t 4 can't do much on its own. It it falls down at the kind of friction point most of the time. And yet, as Zuckerberg recently said in his darkish interview, like, you observe these weaknesses. You have this process of building a lot of scaffolding to try to overcome those weaknesses, but you basically have a TikTok development cycle where you build the scaffolding, and you also try to build the next generation of the model. You don't have to have all that scaffolding. And so it seems pretty clear that the next generation coming out of open AI, and this has been broadly reported on, is gonna be a lot more agentic. And I basically define that as saying, if you give the model short instructions, it can unpack those into a plan, and then go pursue that plan until it either succeeds or tries trying or somebody comes in and turns it off. Right? But the there is a qualitative difference to the sorts of things that we're starting to get into now. That's also measurable just in terms of, like, how close these things are to expert performance in a lot of different domains. But they're already better than the average human. They're closing in on human expert. And so there's not that much farther to go. Either progress would have to, like, totally stall out, which definitely doesn't seem like it's a bet anybody really wants to make either commercially or safety wise. Or something else really weird would have to happen for us not to be entering into very unprecedented territory. The other big thing is, key point, is the leading developers basically are doing these things. Right? We've if we consider the live players today to be OpenAI, Google, and Anthropic, those would be like definitely my top 3 global model developers. Anthropic has led with a responsible scaling policy. OpenAI had a pretty robust, I would say, could have been better, but nevertheless, nontrivial effort in GPT 4 safety a year and a half ago. Clearly, they're bringing way more energy to that now with the next generation. And Google, if anything, gets criticized for being too slow and conservative. So it seems like all of the leading companies are basically already doing something like this. So then you won't well, I would say, okay. Who is this bill really for? And it seems like it's for meta

Nathan Calvin (53:33) maybe

Nathan Labenz (53:33) slash, like, the next half dozen companies that might say, hey. We do wanna participate in this multi $100,000,000 training Dale game. And it's not that many companies that could plausibly do that and and expect to get anywhere close to the state of the art, but there are a decent number. So it seems like it's really for them and maybe also for the decisions that they might make to open source if they can't, which again comes really right back to meta. The decisions that they might make to open source if they can't make certain assurances about what might happen with light modification downstream. Fair analysis?

Nathan Calvin (54:18) I think that's fair. 1 thing that has been the subject of a lot of conversation online, and I just wanna let the record trade is that there's been a lot of folks saying you're trying to make it so that no entrance can compete with OpenAI and and traffic and Google, and that this bill is basically, like, secretly authored by them. They're the ones doing it, and it's just not true. They'll also have, like, provisions around, like, whistleblower protection, but different things they have to do that they're not half dead. Yeah. Wanna give them props and recognition so that they said that, like, I do think that they'd last have folks over the night who've taken these issues very seriously and have put out these practices that, like, we think are good in different independent experts with no financial association with them. And and like are saying are good. And we also you we have 1 in in view, which is, like, 1 of the upstart meeting with them, like, to work the bill and to do it all. I don't think, like, that crazy of a super radical thing. I think these aren't things that are doable. And what we're just saying is you're doing this now, but as the rate intensifies, don't stop doing it. So important. And also new entrants come in and think that you wanna not take these pretty basic common sense precautions, that's not okay. But that's some of what we're getting at, and I think it's fair to recognize that I think we really are trying to look at what different companies and folks are doing that being smart and appropriate and saying that, look. You've made voluntary commitment. The White House is not doing this. They know the things like that. We don't think that these the voluntary commitments, we don't think these can be then we just decide if you're not gonna do that if your investors are really pushing you, like, we may clearly want you to do.

Dean Ball (55:46) I think that's interesting because this bill doesn't have any this associated with it other than sign a document that allows you to be sued. Like, it doesn't there is no this. In this bill, there are no because we have no standards for what companies should be doing. If we had those standards for either red teaming or alignment standards, safety standards, whatever you wanna say, I would be so much more in favor of laws that say, for example, if you wanna commercialize an AI model in x y z industry, then you have to refer to these standards. To do that, to get to that point, which would actually be, like standards like that would actually be moving the epistemic ball forward as opposed to saying, you guys figure it all out. And if we don't like what you do, we're gonna sue you or potentially bring perjury charges against you. I know the perjury thing is complicated, but certainly there's a civil enforcement aspect of this bill. It's very European. That's very much how, like, EU regulation works, where the EU says, do x y z. You figure out how to do it. And then if we don't like what you do, we'll come after you, and we'll, like, we'll let you know we don't like your compliance by suing you or fining you, which, again, defines structure here. How big would the fines from the frontier model division be? We have no idea, and they can set them to be whatever they want. So that's not great from a political economy perspective. But I think that we're getting the order of operations for good AI policy or AI regulation entirely backwards with this bill. I think that what we need to make good laws are good standards, and what we need to make good standards necessitates active scientific inquiry into AI, and active scientific inquiry is best supported by open source invest by open source software, by open investigation in a decentralized way by many different people, and this bill makes all of that much harder. So I think that what we instead need to do is take a process of science to build standards to then create laws, and we're doing it in the opposite direction right now.

Nathan Labenz (58:03) Yeah. I guess my immediate reaction to that is that sounds almost like the case for a pause. If you feel like we don't know what the standard even should be.

Dean Ball (58:14) I don't make the case for things that won't happen. So I don't I don't make the case for things that aren't possible. So, no, it is not a case for a pause. It is a case for scientific investigation.

Nathan Calvin (58:24) I agree with you. I would love to live in a world where we could sit down for 5 years and figure out what these rules and standards should be and then write those clearly into law. I just think that if we sit down and do that process for 5 years, we might have horrible bioweapon attacks and critical infrastructure attacks and, like, crazy agentic models running around the Internet committing law crime.

Dean Ball (58:45) But in what world is that actually true? If agentic AI models are running around the Internet committing crime, society adapts. Right? There's this model that I think a lot of people that are are fixated on AI safety have where society is static and, like, things happen to society, and society is this inert object. But that's not true. Like, you're living in the symphony. You're part of the orchestra, and it's happening. It's unfolding. Our response is unfolding all the time. And so I I think that, like, society will respond. The biorisk thing is the AI models are not the enabler to do to to biorisk. There's biorisk potential on the table right now, major biorisk and and CBRN risk that exists regardless of AI. There are other bottlenecks to that that we need to be more proactive on policings, not just in America, but across the world. And we are. Right? That's part of what the Biden administration executive order did that I very much applaud. We are doing those things. So it's not that we would do nothing and that society would be inert and that then we would scale. We would pursue the scientific investigation while doing many other things to ensure downstream downstream security security and and resilience. Resilience.

Nathan Calvin (1:00:01) I really love that you're taking the idea around safeguarding society from some of the bioweapon. I feel like there's some fluffiness in some proponents. Oh, if you just have the open source models, you can defend the bioweapon with your own bioweapon model. And, like, the way to defend it is more of, like, actively having r u b c and gene separators printing and, like, affected vaccines and vaccine platforms and kinda agree, like, wish we lived in a world where maybe we just all stopped debating this and just all worked on trying to safeguard the world against horrible myoIP. I think that that is very important and and good. That still No. I agree. Society in general is pretty good at adapting. I think the issue is that this rate of change is so incredibly fast. It's just like so much faster than the previous levels where we have had a big them. And when I'm in Sacramento and I'm like talking staffers and legislators from rural California, like, they're just terrified. I'm not necessarily that optimistic about the ability of society to effectively adapt to the just letting it out at an maximum speed. And and I agree that some of it's gonna happen no matter what, and to some degree that could pay because out of the the 2. But I do think The US and California in particular is really leading the industry right now. We do have these export controls on China and just kind of their own problems with the tech industry. That means that I think if they don't have access to cutting edge open first models, it's unfair how much that they can actually be competing at the frontier. I think we also need to be adapting and putting in societal adaptations and realizing that stuff's gonna slip through the cracks, but I also don't think we can just rely on adaptation on its own given just how incredibly fast this is happening relative to other societal adaptations.

Dean Ball (1:01:34) Yeah. I agree. There's definitely policy steps. I just think models are too low in the stack for a variety of different, both practical and principled reasons, but it's a combination of policy and societal responses. I will also just say that there's I I hate when people use IQ to, apply to language models, but there is, like, a 1 30 IQ language model out there who's going around talking about its metacognitive properties quite freely. And it's called clot 3 opus, and I don't know. The that's happened, and the world the republic has stood, has it not? Nobody seems to care outside of me and 4 other people. No 1 really seems to care about it. I wrote about it the day it came out, and I was like, oh my god. Look at this thing. But nobody cares. It and it seems like it's been fine. I'm not saying that that's great and there should be much, much, much more of that. But, also, 1 of the things that AI safety advocates love to to say is that their regulations only apply to people that are very large companies spending lots of money. That's true today. Obviously, that's just not true in 5 years. If the thresholds stay the same, everyone knows that. And I've always felt like it's a disingenuous thing when anybody says this only applies to big companies because we all know that's not true. In the long term, that's not true. But it is also the case that currently it does only apply to big companies that are spending a lot of money. And they're big companies that have shareholders, and they're spending a lot of money. And so there are certain they have strong incentives to investigate the safety aspects of this and to not release things that do awful stuff. And maybe, yes, do they maybe do it by accident? Possible. I doubt it. I doubt not in an irreversible way. I think these companies are very capable. I really admire all of these companies for those things. I think it's great that we're leading. Right? I think it's great that American firms, people that are embedded in an intellectual community that has worried about these issues for a long time really worried about these issues before the technology was real. Right? That's great. Sometimes I worry, though, that because these concerns were initially born before the technology was real, they may be a little divorced from the technology in some sort of fundamentally reversible way. There's there's, like, a genetic there's, like, a genetic problem with the concerns that people have that just is, like, we had all these concerns before deep learning was a thing. We were not anticipating a statistical approximation of the Internet as being the thing. We were anticipating, like, a Bayesian pure reasoner that could infer special that could infer general relativity from 2 frames of an apple falling from a tree, right, without having seen any other part of the world. Right? That was, like, the original vision. This is quite different, and I just think that's generally important to keep in mind. We're not on the path that LAIZER thought we were.

Nathan Calvin (1:04:34) Is there 1 thing I will say is there's been some movement in the opposite direction where you had folks like Jeff who weren't initially particularly worried, who then, when working with the latest generation models were like, holy crap. You will have more than I thought it was. And it seems like if I thin out the last 5 years of progress 5 years forward, I am pretty surprised about that we're the doctor. I do think we need to make sure that we're continuing to think about, like, in the realities of these models and not in the higher level ideas about how intelligence might work. And I think that's why I think all of us, I think, for your credit, I think, for acknowledging that tremendous amount of uncertainty here. I think it's reasonable. I'm glad that we agree on that. Can I ask what You have heard

Nathan Labenz (1:05:12) from the leading developers? They've been strikingly quiet in the public about this. I haven't seen any official statements from open AI or Google or philanthropic on this bill. I wonder, like, why that is and what what do you think their attitudes are? Would you characterize industry as, like, working against this behind the scenes or saying, hey. This is okay with us. We're not, like, gonna be vocal supporters of it for whatever reason.

Nathan Labenz (1:05:39) I I do wanna talk a little bit

Nathan Labenz (1:05:40) about the whistleblower thing because I I do think 1 of the weirdnesses of the current moment is that this likely society changing and potentially society devastating technology is being developed in a way where it seems like not even everybody at the labs has visibility into it anymore, which is weird. But I guess for starters, how would you characterize what industry is saying or doing behind the scenes on this?

Nathan Calvin (1:06:04) Yeah. What I'll say is that the senator's office with the cosponsors included have had conversations with OpenAI and Anthropic and Google and Meta and Microsoft. And I think they have a variety of different views. I think there are aspects of the bill that they want changed. I think there are aspects of the bill that they do view and hey. These are things we're already doing. That seems okay. This doesn't necessarily seem that crazy. We have seen come out in public opposition places like the chamber of progress, TechNet, AI, which are tech associations, which have Microsoft and Meta and Google as members. And so we've seen some of the companies, like, speak through these tech associations. And that's sometimes how it goes where in general, the companies are happy to let their, like, practice speak through them, though I don't wanna necessarily pay with this super broad breast because I think there is just, like, actually a lot of heterogeneity even among companies that you might anticipate would have, like, similar interest here. The thing I can, like, very much say is, like, they did not write this bill. They're not pursuing, like, not to do that is just a 100% false. There's been talk about that, and it's it's just wrong. Scott Wiener got freaked out about AI risk and then came to sponsor saying, can I do an AI risk bill? It was not like anthropic approach. Didn't you do a secret thing? Can

Nathan Labenz (1:07:24) you share anything more about what industry wants changed?

Nathan Calvin (1:07:29) Yeah. I think they're wanting to be clear about some of the specificity of the language and saying, like, we understand your intent of what you're trying to cover, but are you being patient laid precise about that? This is something that, like, we're confident we can we can follow and understand what it is. There have been, like, some, like, micro debate bell due to the KYC provisions violate the stored communications act. There's some, like, fairly big provisions, but their view in general has been that we're not gonna or more post publicly, but that we want to weigh in and and provide technical information. I I think some of the amendment is the the technical way that talked about and is that accurate. I'd also wanna be careful to the fact that, like, the center's team is leading these negotiations. They're ongoing. And, again, I wanna push back very strongly on the idea that the motivations for this mill are to lock in these incumbents. That is just, like, a 100% false, and I can't empathize enough the degree to which it falls. I think, ultimately, a company do have and support because they think these are doable. I think that's We would love that. Again, you did that. The other ones did good. But, like, I just, you know, wanna work out that as strongly as possible, but it should fall.

Dean Ball (1:08:32) Yeah. I don't believe that OpenAI or whoever from the big companies wrote this bill. I think that's true. In terms of the motivation. In terms of the effect, this bill undoubtedly benefits OpenAI, Anthropic, and DeepMind at the expense of Meta, whose strategy relies upon openness. That is the explicit strategy of the company. And also most of the startup community, which broadly speaking hates this, and a lot of the scientific community and academic community, which also I know Jeff Hinton and Joshua Bengio exist, but there are other people. I don't wanna betray confidences because I didn't get permission from anyone to to say names, but there are very prominent researchers who are just as respected as Hinton and Benjio who don't like this. So, you know, the startup community has a lot of problems with it. Scientific community has a lot of problems with it. Those are important. And, no, I definitely don't think that the regulatory capture is the explicit aim of the bill. It is, however, very likely to be the effect. And that's how we should judge things, not by the motivations, but by the effect. California is repeat with legislation, which should be judged by its effect and not by its motivations.

Nathan Labenz (1:09:45) Seems like a lot of the discussion has focused around this is gonna make open source illegal, which is obviously a bit of a hyperbole, but does maybe have a kernel of truth to it. The main thing that I see is this idea of you release a model, and what people do downstream with it is your responsibility. That makes sense to me in as much as I think if you have trained this monstrous shoggoth lama floor or whatever, and it's just got a thin veneer that's, like, causing it to refuse to make your bioweapons when asked through the meta search bar. But with light fine tuning, that capability can be exposed again. That's gonna be a big problem if something like that does get released. I presume everybody would agree on that.

Nathan Calvin (1:10:37) I don't think everyone agrees on that. Sorry.

Nathan Labenz (1:10:39) Don't know if you can object to that, Dean. But if let's say we have a we train long before. We do not have a theoretical breakthrough. We can't make a good argument for why this is not gonna be a problem. The pre trained version, we can even say, hey, look. Okay. It'll do these various things. Right? It'll find j 0 exploits. It'll design bioweapons. It will maybe break itself off of its server. Now we RLHF it so that it refuses to do that. But we know from plenty of research that capability is still in there and can be re exposed with downstream fine tuning. That to me seems like it would be a big problem if that's released. Tell me if you think that's either not a problem or I

Dean Ball (1:11:20) would say the biological weapons thing, it it it's not an intelligible concept to say that LAMA 4 can design biological weapons because the design and software aspect of the biorest thing is an important part of the equation. But there are other parts of the value chain as I've already or production chain as I've already talked about. So there's other things you can do about that. In a world where Llama 4 or whatever can identify 0 day exploits and have that be fine tuned out of it or or otherwise removed by downstream open source users, I would I I think that the equilibrium we have reached right now is actually a quite positive 1 where the frontier of capabilities are generally reached by the closed source companies who take these things very seriously, and they test it. And Microsoft owns GitHub. And if g p t 5 and GitHub haven't already met each other, I suspect they will in the future, probably before the public knows about it. I think that there is definitely a lot of responsibility that Meta would be taking. And if it's not a responsible decision for them to open source it, I guess it's you're asking me what I would wanna happen. It's not my decision. It's ultimately part of this whole thing is that I don't think it should be any single person's decision. I think the world is too complicated to make decision for people on things like that. But, yeah, I would certainly hope that Zuckerberg and Dionne Lecune and others involved in that decision would think quite seriously about that and at least want to robustify society much more, our digital infrastructure in particular. And I think steps are already being taken along those lines, by the way. But, yeah, would I want that to be released tomorrow? No. I wouldn't if it were my choice. And I think that most people don't, and that probably includes capital allocators and people who are responsible to shareholders and things like that. So I think they probably won't. And if such a thing is even possible, which we don't know, I think a problem gets introduced when you legislate things, because legislation is different from conduct. When you legislate things, you lock certain dynamics into place and create sort of a freezing in amber of industry and industry dynamics, and that's just problematic for all kinds of downstream reasons that are hard to predict. Yeah. I guess, no, I wouldn't want it to be released, and, no, I don't suspect it would be.

Nathan Labenz (1:13:53) Yeah. I have a, you know, reasonable amount of trust in meta leadership these days. I think Zuckerberg has had some notable comments recently suggesting that he's not purely ideological about this, and is not not gonna YOLO an open source model if he thinks it's, like, genuinely not safe to put in the public. I wonder if there are things that we could do to allay some of these concerns. On on the open sourcing 1, I do think most everybody thinks that open source is good unless there's these crazy capabilities that are getting inadvertently exposed. So 1 proposal that I was noodling on myself and then got a more concrete version of it from z in a blog post that will be published by the time this is released, so you can go check it out on z's blog, is to create a little bit sharper definition on what counts as a derivative model versus what counts as a nonderivative model. The idea right now is that if you open source something and somebody makes a derivative of it, it's still on you. If they fine tune and reexpose some latent capability that you mask, that doesn't get you off the hook as the foundation model developer. But there's not like a limit to that. And you could imagine a scenario where somebody does 10% as much additional training as the original model had or 25% or 50%. And at some point, it's like,

Nathan Calvin (1:15:12) okay.

Nathan Labenz (1:15:12) That's no longer just re exposing some latent capability, but you're actually using this base to build something that is fundamentally new and qualitatively different and maybe should be considered non derivative. So 1 idea I might advance would just be maybe we have some sort of reined in notion of how much downstream modification is still the responsibility of the original foundation model developer versus at what point it becomes like, okay, now it's really on the new developer. You put in your own tens of millions of dollars of training, and now that's on you. Another idea would be maybe some sort of sunset to this bill would be a good idea. I think everybody agrees that we're in a regime of fast change. We would be very surprised if we don't know a lot more in even 2 years relative to where we are today. So, again, I don't know why this is never seems to be done, but 1 could imagine a, hey, for the next 36 months, this is what you have to do. And at the end of that, this bill will sunset. Obviously, it could be extended. But ideally, it would be, like, updated and improved upon, and we could maybe bake some sort of forcing function into the legislation now to try to create something like that in the future. Any thoughts on my my fixes? I think these are

Nathan Calvin (1:16:32) thought whole pedestrians that I think author really appreciates. Again, I think we agree on more than I think you might think for bitter. I think being open source was good. I'm happy about it. I think it's great. I think it's gonna be able to do lots of awesome things, and I think there will be harms, but those are best addressed at the application developer and other stages other than the developer. And do you think that there is some level of capability where just it out is not a great idea? And Mark Zuckerberg seems to be acknowledging that. I do worry that if we're in a regime, we're, like, relying on the will of Mark Zuckerberg to avoid catastrophe. And a lot of people are uncomfortable with that, and I think that possible there will be other people who maybe do think it's great machine thick over and do wanna just slow it out at some point. I think that is a a rough place to to be if that's what we're taking everything on. I definitely think it's fair to distinguish between a situation where you are fine tuning, remove thumb guardrail rather trivially versus, like, building your your own model with 10 or 50% of the training costs. And, again, I think it's definitely there to consider amendment that clarify that. I think that seems reasonable. On the sunsetting queue, like, there are things within the bill that try to make it future crew. All these references to, like, reasonable and myths all of these best practices, those will continue to evolve over time. But this is us trying to reckon with the urgent nature of the problem and wanting to put up what is a thoughtful and educated quiff at that, but I definitely don't prestigious. That actually does task of it being the end all be all of there being no later revisions or subsequent statute needed. I think that's super fair. Again, I think that open is great, and I am I genuinely think that there are, like, a ton of benefits and that in the vast majority of cases, the benefit outweigh the cost. Similar to how I work pretty bad, like, the biosecurity issues and when you're working around, like, any restrictions on gain of function research group that was opposing any restrictions on being front research was, like, scientists for science. And they they said, no. If you don't like gain of function research, therefore, you're anti science. You hate all of science. And the fact that I don't want there to be models that can shut down wastewater plants or design bioweapon or do things like that. I hate open I I love open thirds. I just think there's a very small category of very common things where the often and the opportunity for societal adaptation are are not there where I think that we should be more cautious. And honest not that different from where Mark Zuckerberg is at. I don't think it can just be them entirely making those calls on themselves and what mood Mark Zuckerberg wakes up on a particular day if you just feeling like maybe Elon put out a bigger model and I really wanna beat him today. They'll, like, look important. I don't know. That's my thought there.

Dean Ball (1:19:08) I'm I'm not sure if we read different newspapers or something, but I'm not really sure I I see anybody YOLO ing right now. This isn't, like, the least YOLO technology in the history of technology. It is such a profoundly unfun conversation. The unfunness and seriousness and fake everything about this conversation, is a form of regulation, I would argue, in some ways. But I guess I just don't really see where the person is who says, I I don't give a crap about humanity. I guess I fail to understand who is the person that, like, isn't thinking about these things, that doesn't care about the dangers of taking down critical infrastructure or wiping out humanity or whatever it might be, but who does care about the California frontier model division. I guess I'm, like, failing to understand what part of the Venn diagram do we think we're really moving the needle on here other than just maybe creating new forms of liability and new regulatory risks for the legitimate players. I that's what I guess I failed to see. And by the way, my comments should not have all been taken to suggest that we're relying on Mark Zuckerberg's whims. We're relying on a lot of people's whims. Right? That's the nature of the world. I can't help with that. That's just the reality. Some people have more power than we do. But but at the end of the day, what I'm relying upon is a system of incentives that tends to work. And by the way, which regulation often tends to break in weird, like, extremely high dimensional ways that are, like, very hard to diagnose or to foresee. And I think we all know that about regulation. Everyone's seen that. So it's not like what I'm saying is empirically unobserved. Everyone's

Nathan Calvin (1:21:01) seen that. I feel like sometimes there's an idea that currently California has no rules at all about AI, and then this is stepping into the void and creating those. There's lots of liability law. There's lots of actually existing uncertainty about, like, a situation where you did release a model and then someone modifies it, does something bad with it, whether you might have liability. I think to some degree, these are already questions and uncertainties to exist. I view this as actually clarifying what those are and some degree giving company more confidence that what they're doing satisfies their burden. I think there's actually some point back. There's no formal safe harbor in this legislation. Like we have a statute's clause that make clear that, like, all other remedies and rights and actions under law still exist. But I think there is some more confidence that come to companies about, like, what do I do if I wanna prevent these really bad things from happening? We're saying, these are some things to do. These are some good ideas. And this is giving some additional clarity to what it means to take reasonable actions as a branch developer. Again, I agree. So far, how people are acting is, I think, very in line with the bill and with what I would view as, like, responsible development. I guess I'm just worried is that I don't know if that necessarily extrapolate in the future. I do there are people who, like, the deaf cases and Richard Suttons of the world, who I do think have very different world views for myself or the vast majority of people. And I agree there are be some group of folks that are not gonna care about what the law says at all, but maybe they're gonna have investors who just think that they're making a bunch of money and they they're like, oh, I want the machine to take over. And some investors might not care about that ideological thing if they don't believe that, but maybe they do care if they're just violating the law. And so I do think there are other people for whom this does have force. And I also do think again, there's the question of what some of these developers think upon reflected in their better nature, it's in everyone's interest versus what is happening when there is incredibly intense competition and ego. And I just think we do shape their incentive and, like, having them push back against that drive. Think it's super important, and that's great.

Dean Ball (1:22:57) But also not killing the entire industry and running their own company out of business shapes their incentive. If OpenAI releases a model that brings down the Internet tomorrow, I don't envy Sam Altman's job. It sounds incredibly hard. The pressures are there. Right? And certainly, there's internal stakeholders within OpenAI that are very worried about these things, and and that should be the case. My point is that safety is a very hard thing to legislate. It's integrated into a lot of different economic activity at the frontier of technology. It's incentivized by a lot of different things. I think everything you're saying is fair. I just think that the dynamic we already have is a perfectly fine 1. The first thing I looked at this bill and said was if someone uses an AI model to do a hazardous capability of the kinds described here, you're gonna get sued, like, for sure, 1000 different ways. So, like, what is the marginal utility of this bill other than creating, again, a regulator whose objective is to ensure safety, but which does not actually do any safety work themselves and which does not build AI systems. That is a regulator whose incentive is to cover their ass. That's the public choice reality of this. And so you're creating, like, what will ultimately, I suspect, be a quite pernicious addition to the AI industry dynamic with the frontier model division. So

Nathan Labenz (1:24:27) 1 idea a simple person like myself might advance would be to say, why don't we just make this a lot simpler and just make certain things illegal? Like, what if we said it is illegal to certainly release or open source an AI model that can do x? And just have some concrete x's and say, you just can't do that. Boom. Simple. Now it's on you to make some if you wanna open source something, you gotta make sure that you comply with that standard of what it can and can't do. And there you have it. Now it's tough because these are general purpose technologies, and there's a lot going on there. But if we had a pretty high threshold for the sorts of things that we are concerned with, then it seems like a lot of people would be like, okay. That reduces a lot of ambiguity. I don't have to worry about whether I fit into this or not. I just have to be very careful about 10 different things, maybe even fewer than that. And at least it's clear what I have to do. Right? What would be the downside? Maybe it would be more inclined to support that. But what would be the downside of a sort of simplification, clarification, and just the following types of AIs are illegal. Boom.

Nathan Calvin (1:25:38) Yeah. I think definitely room to make the bill fairer in in part than you and others have given some good suggestions that I think, us in the offer are gonna think hard about. I think it is hard when you're trying to enumerate those and also recognize that you can't be a 100% sure. Like, we spent some time trying to think about, like, definitions of deceptive or self replicating systems or different things. Very hard to define in statute. And I think describing more in terms of the the near consequences that we're trying to prevent, we thought was clearer. It's easier to describe the process of you should do tests for these types of things, and you should have some level of reasonable confidence. I I hear you, and I think open to the typical language that tries to make that clear. I doubt that addresses Dean's parents speak today, but that's my initial thought.

Dean Ball (1:26:23) The bill you've described, Nathan, I'm not sure if it would be, like, a single piece of legislation, but that is the legal outcome that I would most prefer. The problem with it is that we don't actually have the knowledge to legislate an outcome like that at this time because of the lack of good technical standards and evaluations and safety best practices and all that kind of stuff. And that stuff simply takes time to build, and I think we should invest so much into building those things as quickly as we can. I think that this bill hinders our efforts to do that, not helps. It does take time. And I mentioned that earlier that it might take 5 years to do that. And Nathan Calvin, you said, we don't have 5 years. And in my mind, nobody has made a compelling case. That is true. The only case that I hear on that is pointing at charts that don't tell me anything concretely, calls to authority, etcetera. We don't know how much time we have. And so I don't think just I I like, this is a general it's not a critique I'm making of you in this conversation. I just wanna be clear. You've been, like, awesome. But it is a critique I have of the AI safety discourse in general. I'm not this is a societal negotiation. The AI safety people don't just get what you want because you were here first. This is a societal negotiation, and I don't personally like to negotiate with a gun to my head. And that's oftentimes how it feels when Mustafa Suleiman last summer goes on a podcast saying, we're all gonna die if we don't create a brand new regulatory regime from scratch tomorrow. That isn't true. And we're giving up a lot. We're potentially setting up radical precedent here. By the way, that is an important part of this. It's not necessarily specifically what this I mean, it is specifically what this bill does that I don't like, but it's also the precedent that it sets about government preapproval of software and killing that whole system that's been quite valuable. We're creating an and no one's talked about the enforcement infrastructure for this bill, also a major thing, especially as models of this level of power proliferate. Good luck enforcing it without creating a policing regime over the Internet, California state government, which is a long term outcome that that might be inevitable either way. But my only point is we're sacrificing a lot, and we're setting a pretty radical precedent by doing this. And we need really compelling evidence for that. And I just it could happen because of exponentials is so unbelievably far from good enough in my mind.

Nathan Calvin (1:28:56) Yeah. I think we are honestly in similar places on the epistemics. Like, when I say 5 years, you know, I put money in my retirement account. I do plan beyond 5 years. I'm not somebody who does definitely in 5 years gonna be this. But when I look at these graphs, I have uncertainty. They could turn off at any time. We could have another air winter. We could run into to block. All that could totally happen. I've I've talked to a lot of smart people who have different opinions about this. There's, I think, healthy uncertainty there, and there's a question of what to do in face of that tremendous uncertainty. I hear you saying that you think even if those risks do emerge that this will make us in a less good position to handle them. But I I do think that they are at a high enough level to be worth taking seriously in 1 direction or the other. The other thing is I a 100% agree with you, though. Just because a bunch of people on less wrong thought about this 15 years ago doesn't mean they get to decide what happens. I a 100% agree with that. I also don't think, Kate, that just because Mark Zuckerberg buys x thousand numbers of h 1 hundreds, that he gets to decide what the rest of the world looks like, and there's no opportunity for democratic input or things in that either. And so I think that that's what is happening now. And we're going through this multi month, tons of open hearing, lots of discussion on Twitter, talking to dozens and dozens of people. And I I think it's fair to be pessimistic about government and lots of respect, and there are things with good intention. I think this process of trying to figure out what should this should look like and taking seriously a variety of news about people inside the AI community and people who are terrified, who are out of the AI community. I think that is appropriate. It's the way we have to change things, and I prefer that to, like, just saying that these leaders are gonna make their calls, and I hope they woke up on the right side of the bed this morning.

Dean Ball (1:30:29) To a certain extent, I actually would dispute the notion that we make decisions about the trajectory of technologies or industries based on democratic inputs solicited from, like, anyone. Right? I don't think I have a vote in that. I have a vote as a consumer in the marketplace, but that's not something I exercise through my government. And I think that government thinking too much that it is responsible for setting the trajectory of technology or the destiny of mankind, and really anything even remotely adjacent to that gets you into all kinds of dangerous problems. It's a bad attitude for policymakers to have. I'm not sure that things in California are going so overwhelmingly great that Scott Wiener has a lot of time to be working on the destiny of mankind. I just I think that we make these decisions in more diffuse and complicated ways, and we don't do it through the democratic process, and history doesn't unfold by show of hands. Gabriel O'Princey did not ask everybody in Europe whether he should kill archduke Ferdinand. He just did it. Sometimes that's the way it goes. Sometimes nonlinear things happen as the results of 1 person. History doesn't unfold by show of hands. And so, no, I don't think our fate is in Mark Zuckerberg's hands. I don't think our fate is in any 1 person's hands, and I wanna keep it that way. And, actually, my opposition to something like the frontier model division existing is partially connected to that instinct I have of not wanting to put all my eggs in a small number of baskets.

Nathan Labenz (1:32:05) Let me ask another kind of hypothetical question. It seems like there was openness to this notion of if we could effectively enumerate a bunch of things that would be really bad for AI models to do, then we could all potentially get on board with a law saying, you can't make an AI that can do any of these bad thing. We have definitional challenges there. 1 thing that I see missing from the bill, or it's not entirely missing, but it's very limitedly addressed, is independent testing of the models. I would love to see a little bit more independent access to models during the training process. In particular, there's a lot of debate in the literature and community as to what should be considered an emergent property, and how fast do these emergent properties pop up in the training process. It seems like in general, there is time. That last order of magnitude is always the biggest order of magnitude, of course. And, like, it seems to take an order of magnitude for a lot of these things to come online to the best of my ability to synthesize the literature. So what about a scenario where we say, it's illegal to distribute an AI that can do these things. Distribute could be via API or could be open source. It's but it's illegal to distribute an AI that can do any of these things. And we will determine whether the AI can do those things by giving access to a bunch of third party testers who don't necessarily get the weights, don't necessarily get to know all your trade secrets, don't have a a sense for what the parameters are or whatever, but can just hammer at the thing for a while and satisfy themselves that either they are able to get a certain behavior out of it or finally give up. I know a lot of people who would be motivated to do that. I have a little, or all volunteer project red teaming publicly deployed applications. And I could tell you any of those people would be very excited to get the opportunity to go do some of this work on stuff that's not yet released. There's only 1 line in the bill that mentions third party audits as appropriate, least at that I was able to find so quickly. I think there's more in there. But I'm reminded too, we had Paige Bailey from Google DeepMind on the podcast a few months ago, and she was talking about the experience of doing these training processes at Google. And she described it as, like, coming in in the morning and, oh my god,

Nathan Labenz (1:34:31) we just hit new state of the art over here

Nathan Labenz (1:34:33) or this thing that we didn't expect to be this good for a while yet suddenly grokked something, and it's actually hitting these thresholds sooner than we expected. And I took away from that that there is not enough internal during training testing happening to really have a sense for what is going on. Open AI is not that big. Right? They have 1000 people ish. And, like, half of those are, I think, on the nontechnical staff side at this point. So we're not really able to put enough actual human cycles into trying to figure out. And meanwhile, you got plenty online who's jailbreaking everything. I'm pretty confident with current techniques, like, if it's in there, he'll pull it out. He's got the lead speak to to figure out how to to really talk to the models in the way that they wanna be talked to. So I'm trying to square the circle here by both giving you know, the best standard that we could have would be to throw a a motivated community at it, I think, and try to elicit problematic things. And then just say, hey. If we tint the crowd on it and we can't find them, then we guess they're probably not there. That doesn't give us a guarantee, but it would give us much better sense, I think. So what do you think? You can't do these things. You have to allow testing. The testers can show that the models can do these things. You can't distribute them. Nathan's SB 10 48.

Nathan Calvin (1:35:52) Yeah. I am also very enthusiastic about third party testing. I think 1 of the things as a co sponsor and and author thought about, we just wanna make sure that the incentives are set up right about who those third party testers are gonna be and what that will look like. I love the idea of, like, finding on Twitter going in and messing around with people's models and trying to break them. I think realistically, you're probably gonna get, like, Deloitte, like, going for a checklist. I think I think there's some concern that, like, companies or if they just hire Deloitte or Booz Allen Hamilton and they give them a check, and then be like, okay. I'm good. Let's do it now and not do that more, like, deeper bolt on searching and sense of responsibility. And but I think that there's some type of not wanting to roll that out. Feel like companies can just push the responsibility onto those folks before we're confident that they will actually do a good job of it. We do have some language in the road, Bill, around the the decision being able to, like, establish a accreditation process for third party testers to do them could be credited by the FDA to be able to do tests. But I think requiring them to then use those specific testers, I I just think we want a little bit more information about, like, who counts as a third party tester? How are you gonna enumerate the provision by which they're doing it? You're gonna rely on accreditation process. How is that accreditation process judged and not opportunity to capture weird things. Yeah. I'm enthusiastic about it. I'm very open to the idea of there being, like, a role for greater amounts of third party testing in the bill, and I really were important part of it. And some of our whistleblower protections also do apply to folks that labs are bringing in to to do testing. And I think we definitely view a important role for them in the but just like the attitude, I think, that led the failure to feel for this is responsible, but very humble about their ability. I think that they should feel like they have a role and important part of those systems will protect California residents, but then I think they should be pretty thoughtful about what to what degree government is gonna be able to go to toe to toe in an effective manner with the company. That is what we have tried to build into the bill saying that, like, let's look at what the companies themselves are doing, what they think they're currently publicly saying they're doing. Let's say, no. You actually have to do it. And I think that has been trying to take that approach of not having, like, a penalized regime that trying to do preapproval model releases, which is not anywhere in the bill. I've been trying to go in and, like, how the government post data testing themselves because, like, they don't know how to do it. They they were a very far place to, like, government being able to pay the 7,000,000,000 salaries of folks at these firms, be able to stand and go to sell with them. I a 100% agree with you that there have been lots of problems and lots of legislation about the government, you know, like, we're gonna save the world in this tech government office and have that go better. And that's part of how we've done stuff like, Scott Wiener has spent a decade fighting, and I'm very happy to end it fighting. I think that the last thing we would ever wanna do is create an AI sequel. I think that would be a disaster. And I just don't think that's what this is. So anyway, I think it seems like part of your concern is not necessarily that, like, this itself is doing those things, but that it will be grown over time to create something more like centralized AI belief in the future rather than the, like, the specific language and that actually does that today. Anyway, this is a side 3 third party thing, and I think that the author quite open to suggestions there, and I agree there's a super important role for them. It's just tricky to figure out exactly how you wanna specify it.

Dean Ball (1:39:06) Yeah. To make some of my contentions clear, contention number 1, I do not believe that if this bill is passed, current open source models become illegal. Right? I would never make that claim. There might even be, like, another generation ish of open source models that are legal, but they become illegal eventually. They become essentially unlawful over time, and that's how the bill is designed.

Nathan Calvin (1:39:33) There is some level of capability where I do think that that lest you come up with methods like self destructing model weight, the resistance of tuning. Yeah. There have those capability.

Dean Ball (1:39:41) It's a legislated gradual death of open source AI. I think that's just basically the case. And it's designed so that it doesn't happen like immediately. It's the opposite of whatever Sam Altman says, iterative deployment. It's like iterative destruction of open source. I mean, like, existing open source models, llama 3 will always exist, but it's a sort of gradual devolution. But to your point, Nathan, I think third party evaluation is great. 1 of the things that I support at the state level and at the federal level too is actually safe harbor for good faith third party and, like, truly independent. Not like I paid Deloitte, but, like, truly independent, no financial relationship research into AI models. Right now, a lot of that work violates these companies' acceptable use policy. And so I think that fixing that would be a great move. I would note that in the truly independent, again, not like we paid you, but actually independent research into models is harder without open source. Right? Like, that's a that's a trade off that you're willing to make that I am less willing to make. I think that's, like, a fair characterization. I just wanna say 1 other thing, though, to go back to your hypothetical SB 10 48, Nathan, which is just that we have 2 problems with saying x y z is illegal. First of all, it's what is x y z? How exactly do you define it? But number 2, I actually think the more serious problem is that we don't have a way of actually making that effective. Right? Like, you could say right now, if you make an image generation model that is specifically designed to produce child pornography, then that's illegal under current law. But if you say it's illegal to make a DNA foundation model that can't produce a dangerous bacteria, I don't know. Can we train that out? What is a dangerous bacteria? Dangerous in what circumstances? How do you model that? How do we know? How would you ever actually legislate that in in practice? And then on a technical level, in as much as you can legislate that, can you actually remove that specific set of things from the model's capability without damaging anything else? Right now, the answer to that is no. It might always be no. Maybe it'll be yes. Probably, it'll be some complex combination in between, which is why I think that we need to just accelerate research into all of that kind of stuff. We need to be doing as much as possible. Why I support national compute, infrastructure. If California wants to do it, that's great. I don't think California has the money, and I would prefer to see California spend the money on, like, addressing their overwhelming deficit, their overwhelming budget deficit. I am a fiscal policy guy at heart, so I look at that, and I'm like, I don't know. You should probably be thinking about that. That's probably the primary threat that your citizens face right now. Probably not so much CBRN from AI. But, yeah, like, I'm in favor of national compute infrastructure to give academic researchers all over the country the same, at least a modicum of the kind of resources that that they have in the big labs. I want Lee to have a 10000 b 1 hundreds to do whatever she wants with. And Stanford's not gonna buy that for her. I I think that the public should in some sense, for all researchers. So I I again, yeah, like, I I think that's what we need to be thinking about, and I think that, like, this work this kind of a bill just slows down the scientific research.

Nathan Labenz (1:43:05) So if I had to synthesize this, it sounds like general agreement on it would be good to have a discrete list of things that you can't make an AI that can do these things, but for the challenge of actually defining that with clarity. And there's also agreement on third party testing is good, and obviously, we don't want it to be a credit bureau type of situation vis a vis the mortgage companies from 15 years ago with an example of how that can go wrong. But that does seem very overcomeable to me. I think there there are plenty of people now who would volunteer or who would take a grant from a philanthropist to do that sort of work. And really, the only barrier to them doing that sort of work right now is access. And so that seems like the kind of thing that legislation could, like, rather easily address and say, you have to provide some amount of opportunity for testing to happen during Flash after the training process, but before deployment. That seems like quite easy to do. And the only thing I would wonder about transparency on the results of those tests this is a personal hobby horse for me because in the process of doing the GPT 4 red teaming, I was under an NDA where I was not allowed to visit talk to anyone about anything related to this entire thing. And they wouldn't tell me anything about their other safety plans other than the fact that I was in a black channel with a couple dozen people, and not many of them were doing much of anything. So I was like, I feel like I'm, like, 1 of a couple people in the world that is doing this testing, and I got a little freaked out. Not so much because of what I was seeing. My conclusion to them at the time was that I did think it was a safe thing to deploy. But just the yeah. Because if it's just me and like a couple other people here that are active in this 1 Slack channel, like, that doesn't seem to be enough. So I went and talked to a few people in the AI safety community to see what they thought about what I was seeing. And when it was discovered that I had done that, then they kicked me out of the program. Now that doesn't seem great for a lot of reasons, and I do understand that they viewed me as a bit of a wild card, potential loose cannon, whatever. But this is where, again, I think legislation could really help. 1 thing I've considered in the past is a special whistleblower provision for observed model behavior. If you saw an AI do something, you should be able to say what you saw the AI do. Right? They the sort of, what did Ilias see? Now that that could be distinct from what was the technique. Right? I think there's still a place for trade secrets. At least I'm not trying to make a a case that there should be no trade secret. But if you saw an AI do a certain thing and it freaked you out, I think you should probably be able to tell the public that. And I don't really see what the harm would be to the companies that, hey, it did x. Right? This thing broke itself out of our test server and got onto our other test server, and I'm freaked out about that. That kind of thing doesn't seem like it should be swept under the rugs. If I was gonna take 10 48 to 10 49, it would be like, you have some things that you definitely can't do. We could make those maybe, like, fairly narrow and as concrete as possible for now. We might even feel like that leaves something on the table. But in the interest of clarity, try to make it narrow. Insist on access for testers to be able to do this sort of thing, and also make it clear that those testers can disclose the things that they see. And then if we could do all of that, I think we're headed for a scenario where I would feel a lot better about it as opposed to nobody really knows what they're baking. And they'll show it to us when they're good and ready, and who knows what they did in the meantime that didn't get any light of day. And again, I don't even know how much visibility there is in the labs at this point. There have been some really interesting, somewhat cryptic treats recently from people that have departed open AI, where 1 who's now at Google said, what's really nice about Google's culture is that I have visibility into what's coming. Somebody responded, did you work at another place where that wasn't the case? Like, man, what is going on there? So I don't know. I'm building my own bill here, but I

Nathan Labenz (1:47:23) feel like I'm on the

Nathan Labenz (1:47:24) verge also of getting through a a point where you guys would both support this, and it feels like it might be pretty good?

Nathan Calvin (1:47:31) I do think there are areas where you and I have some substantial agreement, and I think some of them is around trying to figure out these appropriate transparency measures. And I agree there's a problem where there are just far too few people actually investigating the safety of these models. I think 1 solution to that is just to release the models in an open weight form, which then enables anyone to then tinker and understand that, which I think is, again, good and very beneficial for current generation models. I think there becomes some point in the future where that worried me and where I don't think that is sustainable. But then when we do enter that regime where it is maybe not safe to to real relief in an open way form certain models, then I do think you need some way of, like, actually allowing other experts and people outside of it. Not even just because the companies are, like, selfish and profit motivated, but because there's not enough bull and not enough, like, brain there and folks actually evaluating it. I think you need ways to take the wisdom of the crowd and important respects. I think that is an area where there just is a lot of tension among companies of getting access and of disclosing trade secrets and what exactly those things look like. In the bill as written, we have some provisions around the company's reporting AI safety incidents, like, to the FMD, and there's some mechanism by which they can then be, like, made available to the public. But I think it's fair to say that should be cleaner to really go and want some of this just stuffed in a somewhere when it really would be vital for the broader, public and technical community to be aware of. We have talked about this briefly, but I just do think it is super important. 1 area of agreement that I've seen among thoughtful people who are taking both sides of the open source for frontier models debate is thinking seriously that potential that advanced AI models are gonna lower the ability for bad actors to make bioweapons. I think some of the open source folks then say that risk is gonna exist anyway and exist today, which I agree that it does, and that would be not an effective way of combating it. I feel differently, but I think you can go back and forth. Clearly, are good faiths and thoughtful people who disagree. But I just, like, really wanna say there are people listening to this podcast on either side of the debate. Part of you see Gene said this is screening. Just dealing with the potential in the future. We're going to have a worse delivered bio incidents. We let's all agree on this and do something about this. I worked in Congress in 2022 a bit after COVID and there was just no energy to do anything on pandemic prevention because it was like all like everyone's COVID fatigued. It doesn't want to talk about this. These are just like incredibly wise and good investments. Do you think we are pretty likely to see some intentional bad news of biology in the next 10 years that could just be really horrific? And I just think that we could make just incredibly wide investments now for that, and it's good to see that scenario where it being I agree.

Dean Ball (1:50:07) Just as a response to your idea, Nathan, I think the whistleblower provisions are great. Safe harbor for researchers is, I think, a really smart move to make. I don't know if it's actually possible. I in a world where it's possible to legislate the kind of things you're talking about and actually technically possible to do to say, oh, you can't make a model that does x y z, and x y z is a thing that everyone can agree is bad. Sure. If that exists as a thing, then it will be an example of a technological free lunch, and I'll be very happy for that. I'll be very happy if that's the outcome that we get. I think we also need to be prepared for a world in which it's really not epistemically or technically possible to talk coherently about that kind of thing, to say a biological foundation model that can't make a bad DNA sequence. I think we have to be ready for that world, and we have to be ready, yeah, for a world that will be radically better in so many ways that we can't understand, and it will also have higher stakes, bigger dangers, more capable actors that are smaller, where intelligence and agility and speed and flexibility matter matter greatly, that's gonna be an adaptation for all of us. We're transitioning into that. And I think it will change the nature of statecraft. I think it will change the nature of business. I think it will change many things. And it's just I when I look at bills like this, what I see, frankly, is a rather desperate attempt to to freeze the present in Amber and to not go into that new future. And I don't know about you, but I'm not all that happy with the current status quo in the world. I'm open to changing the status quo in quite radical ways. And the trade off to that is that it might change in some ways that I don't like. So I think that contending with the reality that we have rather than casting an anchor into a seabed that's not actually there, into a seabed that is of our imagination and exists purely in a legislative fiction is I think that's ultimately gonna put us in a lot more danger than simply contending with the reality and building capabilities to be able to thrive in that world. And that latter thing, that building is what I work on primarily every day.

Nathan Labenz (1:52:38) Sounds like a good closing statement for you. Do you wanna give any final high level thoughts, Nathan?

Nathan Calvin (1:52:44) Sure. I guess what I wanna say is that I think the proposition that this bill is standing for is, like, a lot more intuitive and common sense than I think I've seen it deframed. I think that we're saying that, like, companies have to test for extremely dangerous and hazardous capabilities. I think we welcome continued iteration on how exactly those are defined and making sure that they're not overbought and in what respects. But, like, I think we can talk about there being really dangerous thing. AI models shutting down wastewater plants and doing bioweapon and, like, the people in going around doing crazy thing. Like, the very bad things companies should test for them, and they should make, like, appropriate and reasonable mitigations, not asking them to do things that are impossible, but asking them to take seriously what is possible that is happening in other companies that is happening at at nest and other places and to do that. I think that seems like pretty fair. And I think the thing from the open source community is that if you're saying at some point in the future, you think it might be the case that there's going to be a model of or whatever that can do really dangerous things and just say it's just completely impossible for us to prevent it from doing those incredibly dangerous things if we open source it. And just like everyone else, you guys gotta deal with it now. Sorry. This is the immutable, like, march of progress, and you're just gonna be crushed under its wheels. I don't know. I just think that's, like, pretty intolerable. And I really welcome, I think, other solutions. I also am, like, really enthusiastic about trying to invest in in methods that, like, send things around ways to open source that still allow additional open inquiry and transparency in those models. Again, I think we don't know about what those research directions are gonna look like, but I think there's some promise in ways to open source that are resistant to adversarial fine tuning. I think there's a long way to go, but I don't think, like, fundamentally against the laws of the universe for that to be possible. And I guess I just wanna say that so the large number of people who are, like, worried about this bill and worried that we're going in the wrong direction, but who do believe that dangerous capabilities of future AI models are real. Like, I genuinely say reach out to the author, reach out to myself. We genuinely wanna get this right. It's complicated in a perfect world. I would love to just, like, raise where we're at for a few years and have a really long deliberative period so we can get this exactly right. I think that's not the world we live in, and we have to be setting these processes up and having this conversation now. And I think they're just, like, sufficient enough fundamental disagreements that I don't think Dean will ever be able to get to the point where he supports this bill. But, like, that doesn't mean that he still can't have valuable insights about ways that the bill could be better or problem that, like, that we should try to fence. And but my message that the appropriate action in face of uncertainty and in face of trade offs is not to just freeze and just accept that's what's coming is coming. And so I which I know is not what you're advocating for. Think 1 other thing as well. But I think that, like, e even on the developer level that I think that we are, we need to be thoughtful and we need to make sure that it's not creating perverse incentives and not causing harms, but I don't think it's fundamentally impossible. And just the fact that, like, government has been bad at things before doesn't mean that, like, we shouldn't weigh in here where I think it's quite enormous. Anyway, it was a really pleasure chatting. I really appreciate it. I feel like I've been just spending too much time on Twitter watching things go by and wanting to actually have a conversation with that person and realize that they're engaging with a caricature of Scott Wiener and myself. We're opening eyesholes who just wanna lock in the incumbent for evil reasons. And and I just made clear that we are both thoughtful people, want things to go well, and are trying hard in a challenging world to do that. And so really appreciate you convening Nathan and being again, like I said, even if I even if we don't think that you can ever support this, like, know your feedback, and would love to hear any additional ideas. That goes for, anyone else with the name who's who also, you know, wants to make it better.

Dean Ball (1:56:23) I think what matters is that we had a respectful, civil, and nuanced conversation.

Nathan Labenz (1:56:27) I'm glad we did as well. Steve, any closing thoughts?

Steve Newman (1:56:30) I just wanna express my appreciation to everyone. This has been a fascinating conversation. I wish we had 50 more hours, literally.

Nathan Labenz (1:56:36) Thank you guys for being a part of it. Certainly, the conversation will continue. But for now, I will say Nathan Calvin from the Center for AI Safety Action Fund and Dean Ball research fellow at the Mercado Center, thank you both for being part of the cognitive revolution.

Dean Ball (1:56:51) Thank you.

Nathan Labenz (1:56:52) It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at tcr@turpentine.co, or you can DM me on the social media platform of your choice.