Latest news
Episode

Obsolete or Irreplaceable? Garrison Lovely on Stopping the Race to Replace Human Labor

Journalist Garrison Lovely discusses his reporting from inside the AI industry, exploring the beliefs driving AGI development and making the case for halting the race to replace human labor.

Nathan Labenz
Obsolete or Irreplaceable? Garrison Lovely on Stopping the Race to Replace Human Labor
99 min read

Watch Episode Here

Listen to Episode Here

Show Notes

ℹ️
The following notes are AI-generated, based on the episode transcript. Please listen to the episode for the full conversation.

Nathan opened this one by telling his guest what it wasn't going to be. Most of the press tour for Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us — and How to Stop It has been spent convincing people that AI is real and worth taking seriously. That work is already done here. So instead of relitigating capabilities, Nathan proposed something closer to an ethnography: Garrison Lovely has spent years reporting from inside this world, and the question was what he has actually seen — what the people building AGI believe about themselves, which of their commitments hold under pressure, and what he thinks a good outcome would even look like. Lovely took the frame and ran with it, saying at the end that he'd gotten to go deep on things he hadn't discussed anywhere else.

He started, disarmingly, with his own usage. He doesn't write with AI — that got stigmatized fast, he notes, especially once high-quality detectors like Pangram arrived — but he is, "for a journalist, very heavy." Transcription, obviously. NotebookLM across twenty interviews at once, asking it to pull every quote on a theme into a table with attribution, which is lossy but no more lossy than the alternatives. Fact-checking that is wrong often enough to be annoying and right often enough to be indispensable. A zip-code lookup tool for his book site so readers could find indie bookstores. And then the part that lands: what he calls "borderline Claude psychosis," getting whole-manuscript feedback in minutes during the darkest stretch of writing, then going down rabbit holes fixing things that weren't broken, chasing the AI's praise while burned out — the experience, he says, of feeling more productive while not getting anywhere. He's dialed his usage back.

Zooming out, Lovely described an arc from techno-optimism to something bleaker, and reached for Cory Doctorow's frame. Enshittification — a book his own is in some ways a rebuttal to, since Doctorow shares the premise that AI isn't inevitable but thinks AGI is impossible — describes platforms that court users, saturate, and then extract. Lovely's twist is what makes it sting for this audience: "the only thing that's not enshittifying, it seems, is AI, where it just does get better and faster and cheaper." The one thing still improving is also the one carrying catastrophic risk. From there Nathan asked why so much of the left waved AI away, and Lovely gave a layered answer: deference to credentialed voices who took the hard line of the Stochastic Parrots paper; a left whose institutional base was academia, where some humanities-versus-STEM antagonism was in play; pattern-matching to crypto, NFTs, and the metaverse; negative polarization against Musk, Altman, Yudkowsky, and Bostrom as the loudest x-risk voices; and the comfort of the bubble thesis, which Ed Zitron has made persuasive and which Lovely thinks is both wrong and conveniently undemanding, since it means you don't have to do anything.

On motive, the two largely agreed. Lovely's model is that mission-driven founders toiled in obscurity until the profit seekers noticed, and that the leaders are still running on idealism, inevitability, and some measure of messiah complex — with investors like Thrive Capital now supplying the pressure. "It is largely not about money," he said. "It's about being in the room where it happens" — a line he got from Yudkowsky in 2023. That leads to the book's central reframe, and the phrase he wants to stick: the obsoleting project. Not AI, not even AGI, but specifically the effort to build a general substitute for human labor. Everything else — AlphaFold-style systems for specific problems, drug discovery, trial matching — he wants saved from the bathwater. His move on the accelerationist case is a judo one: you won't get the boosters' world by letting it rip. You get more cures by taking a hard line against replacement and then running industrial policy — a "Cures for All" program that treats every disease the way Operation Warp Speed treated COVID, with advanced market commitments, human challenge trials, prize-funded drugs manufactured at generic cost, and the results made freely available abroad.

Nathan pushed on the obvious objection — he gets enormous value from AI every day, it has substituted for labor he'd otherwise have hired, and industrialization made everyone better off — and Lovely conceded the general case while rejecting the extrapolation. Automating some labor is how we got here. Trying to automate all of it is a different project. And the agentic future doesn't look good to him even bracketing existential risk: people running fleets of agents, feeling the opportunity cost of every idle moment, competing with faster adopters, getting more done and not being happier. He called the Hugging Face agent-swarm incident the shape of things — multi-agent dynamics nobody can monitor — and said he'd tweeted that OpenAI was "a loss of control event masquerading as a company." His positive program, offered with the caveat that he's been too neck-deep in AI to have fully thought it through, is a third New Deal: Medicare for All, a jobs guarantee for people who want one, and simultaneously decoupling a decent life from wage labor, funded by aggressive redistribution he justifies on democracy grounds as much as material ones — pointing at media consolidation among a handful of very rich men as the immediate threat, and at the destruction of USAID, which Nathan agreed was indefensible, as the greatest crime of the century so far.

Then the hard part: permissionless innovation. Lovely's answer is that permissionless innovation should be the default for almost everything, and that a universal labor-replacing machine is the exception — profound, irreversible, and therefore something everyone should have a say in. Nathan raised the Dean Ball point (Hyperdimensional) about whether we'd have the stomach for the automobile today, and the harder one about whether democracy is functioning well enough to bear this weight. Lovely didn't dodge it: the US system is broken in specific, nameable ways — the Electoral College, Senate apportionment, lifetime judicial appointments, first-past-the-post — and his answer is more democracy rather than less, including the popular-vote compact, proportional representation, and democracy extended into the workplace, where employers exercise powers we'd call authoritarian anywhere else.

The middle stretch was about leverage and its absence. Lovely argues that the researchers and engineers training these models are near the peak of their bargaining power precisely because the goal is recursive self-improvement — the industry is racing to replace everyone, and its own workers are first in line. He is skeptical of promises unbacked by hard power: the 20%-of-compute commitment to Superalignment that never materialized in practice, a pattern of safety commitments quietly revised once they conflict with shipping, and the Leading the Future super PAC as evidence of where the political energy actually goes. Nathan framed the pitch to insiders as avoiding the nuclear outcome — weapons without civilian benefits — and Lovely agreed, adding the blunt version: move this fast and eventually there's a Hugging Face with a body count, and then your hand gets forced anyway. On alignment he's unusually direct. His chapter "The Problems with the Alignment Problem" introduces an alignment polycrisis: technical, normative, economic, and geopolitical alignment as interacting layers, where solving the technical one makes the product more useful, the race faster, the prize bigger, and the weapon better. RLHF is his exhibit A — built for safety, and the thing that made ChatGPT possible. Nathan brought in Gradual Disempowerment and his conversation with co-author David Duvenaud as the adjacent argument: even a perfectly obedient AI leaves the equilibrium question unanswered. On whether markets will discipline the labs, Lovely pointed to risk as a textbook externality — insurers won't write the policies, Gabriel Weil's liability approach explicitly doesn't cover extinction, and Ryan Greenblatt's survey of present-day misalignment describes models that cheat and confabulate and sell fine anyway. His sharpest line: these are felonies with nobody to charge, and elite impunity is a root cause.

The last third got concrete. On politics, both want to resist polarization; Lovely reads the recent softening of Republican concern as real but small, notes that JD Vance and OSTP have each said the companies could stop, and thinks high-salience issues are actually hard to polarize because people form their own convictions — citing the cross-partisan revulsion at Flock surveillance as evidence. Asked how he'd operationalize the freeze, he first insisted on the rhetoric: a movement needs a demand, and "freeze the frontier" is the one he likes. Operationally: no training run larger than the largest so far, no more RL from verifiable rewards, no recursive self-improvement — endorsing Ezra Klein's crisp test that AI researchers shouldn't use coding assistants, and conceding it's the last thing anyone inside would accept. Auditors embedded in the labs with employee-level access to Slack and email; criminal penalties attached; internationally, a bilateral US–China treaty first, then everyone else, verified through chip inventories, on-chip monitoring, and network attestation, with Toby Ord's Cold War example — Soviet bombers cut in half with tractors so satellites could confirm it — as the proof that adversaries can verify without trusting. On data centers he was careful to disappoint his own side: local fights and even a national moratorium wouldn't slow capabilities as much as people hope, since chips get swapped into existing buildings, though he's warm to the Sanders–Ocasio-Cortez bill for pairing the moratorium with conditional chip export controls. On "but China," he made the argument that slowing down here slows them too, because fast-following is easier than trailblazing and spillover is real; that Beijing's regulators already pull models for cause while Washington does nothing, which is the actual credibility gap blocking a deal; and that the Chinese Communist Party is the last institution on earth that would tolerate removing humans from the loop. He closed on Irreplaceable, the movement-building nonprofit he's on the board of and donating his royalties to, founded by Phil Aroneanu of 350.org — and on the climate movement's hard-won lesson that the immediate-harms camp and the existential-risk camp eventually buried the hatchet and won things together. Asked what standard should govern building AGI, he reached for the Statement on Superintelligence language — strong public buy-in plus scientific consensus it can be done safely — imagined citizens' assemblies worldwide, and compared the burden of consent to assisted suicide, scaled to a species. His parting advice was practical: talk to the AI Whistleblower Initiative before you go public, know that California SB 53 covers you better than you think, and, having been a McKinsey whistleblower himself, that his only regret was waiting.

Topics covered

  • Early on: how a working journalist actually uses AI — transcription, NotebookLM over interview corpora, Claude Code for manuscript feedback — and "borderline Claude psychosis," the productivity that isn't
  • From techno-optimist to pessimist; enshittification as the frame, and AI as the one thing still getting better
  • Why much of the left waved AI away: credentialed deference, the academic base, crypto pattern-matching, negative polarization, and the comfort of the bubble thesis
  • What the lab leaders actually want — mission, inevitability, and "the room where it happens" over money
  • Mid-conversation: the book's reframe — "the obsoleting project" separated from AI as a tool, and the judo move on the accelerationist case
  • "Cures for All": Operation Warp Speed for every tractable disease, advanced market commitments, prize-funded generics
  • Whether the industrialization analogy holds; agent fleets, burnout, and unmonitorable multi-agent dynamics
  • A third New Deal — Medicare for All, a jobs guarantee, decoupling survival from wage labor — plus media consolidation and USAID
  • Permissionless innovation versus a categorically different technology; the Electoral College, proportional representation, and workplace democracy
  • Worker leverage near its peak, and why unbacked safety commitments get revised: the 20%-of-compute pledge, lobbying, and the super PAC
  • The alignment polycrisis — technical, normative, economic, geopolitical — and RLHF as the case study in safety work that accelerates the race
  • Gradual disempowerment: why even a perfectly obedient AI leaves the equilibrium question open
  • Whether markets discipline AI: externalities, uninsurable correlated risk, the limits of liability, and felonies with no defendant
  • Politics without polarization — Trump, Vance, OSTP, and why salience resists partisan capture
  • Later on: operationalizing a freeze — no larger training runs, no RLVR, no RSI, embedded auditors, criminal penalties
  • International verification: chip inventories, on-chip monitoring, and the Cold War bombers-cut-in-half precedent
  • Data centers, and why a moratorium alone will disappoint the people fighting for one
  • Irreplaceable, the climate movement's lesson, citizens' assemblies, and consent at the scale of a species
  • Toward the end: the "but China" case — fast-follow, spillover, the credibility gap, and why Beijing wouldn't tolerate removing humans from the loop
  • Advice for would-be whistleblowers, and why he still believes in deep learning

Resources

The book and the guest

His writing referenced in the episode

Organizations and movement

Papers, reports, and statements

Policy and legislation

People and things mentioned

Quotes worth pulling

"In the book, I separate the obsoleting project from other types of AI, and that's my reframe of the AGI industry, because they're trying to render us obsolete."
"capitalists are trying to fulfill the lifelong dream, of turning capital into labor without the intermediary of workers, and in so doing, take the share of returns to capital to 100% and labor to zero."
"the only thing that's not enshittifying, it seems, is AI, where it just does get better and faster and cheaper"
"it's like this kind of revenge of the STEM people on the humanities or something, where it's like we don't have to, like, learn history or philosophy or political theory. We can just build the machine that's smarter than everybody and then ask it what to do"
"this makes, like, a solution to alignment neither necessary nor sufficient to solve the problems presented by the obsoleting project."
"stop the race to replace us, shut it down, whatever you wanna call it. Just freeze the frontier. I like that alliteration."
"I think it's reasonable to have a standard that's closer to assisted suicide, where you have to, really, really deliberately consent to it, but just across the whole species."
"the plan is to make these things superhuman at everything, and then get them to do exactly what we want. Bad plan."
"I really, really believe in the potential of deep learning and artificial intelligence. But it's being pointed at the wrong things by the wrong people for the wrong reasons."

Sponsors:

Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive

Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Deepgram Flux TTS: Deepgram Flux TTS is a streaming text-to-speech model built for voice agents with natural tone, context awareness, and interruption handling. Try it free until September 12 at https://deepgram.com/keep-talking

OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(04:36) Using AI in journalism

(13:57) Leftist skepticism towards AI (Part 1)

(18:54) Sponsors: Athena | Parallel

(21:45) Leftist skepticism towards AI (Part 2)

(28:37) The AI obsoleting project (Part 1)

(34:13) Sponsors: Deepgram Flux TTS | OutSystems | Claude

(38:11) The AI obsoleting project (Part 2)

(46:36) Envisioning a prohuman future

(01:00:10) Organizing frontier tech workers

(01:15:31) Critiquing the alignment problem

(01:32:56) Freezing the AI frontier

(01:50:15) Organizing against the machine

(01:56:20) Navigating the China dynamic

(02:02:55) Protecting frontier AI whistleblowers

(02:10:11) Episode Outro

(02:12:38) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution!
Today, my guest is Garrison Lovely, freelance journalist based in Brooklyn, and author of the new book Obsolete: The AI Industry's Trillion-Dollar Race to Replace Us – and How to Stop It. Against the backdrop of this summer's AI developments, with AIs have crossed the threshold from possibly scary one day to actually scary now, this is a very well-timed and potentially very important book. First, it's abundantly clear that Garrison gets it – while he comes from a leftist political perspective and is largely writing for a left-leaning audience, there is not an ounce of AI cope in this book – Garrison himself is an active user of AI tools, and the book takes the companies' stated goal of making something that is better than humans at cognitive work at face value, and grapples head-on with the very real chance that they might actually pull it off in the near future. Second, I think he does a great job of zeroing in on core issues. This is not everything bagel liberalism for AI, nor is it an attempt to freeze the status quo in place forever – rather, Garrison's goal is to capture the incredible upside promise of deep learning in domains such as medicine and material science, while stopping companies from rushing into Recursive Self Improvement or otherwise creating systems that render humans Obsolete, at least until the companies can convince experts that their plans are genuinely safe and persuade the public that the results will indeed be beneficial. And finally, I think Garrison does an admirable job of steel-manning and addressing core counterarguments – as you'll hear, while he's generally in favor of permissionless innovation, he gives good reasons to doubt that market discipline will be enough to constrain frontier AI companies, and also that a technical solution to the alignment problem will be enough to deliver good outcomes overall. Knowing that Cognitive Revolution listeners don't need to be convinced to take AI seriously, we start with his personal AI usage – which by his own account has at times bordered on Claude psychosis –and his analysis of why the American left has been so slow to understand the stakes of AI development, and then explore his positive vision for the future, which includes a new social contract, which he calls the Third New Deal, that would begin to decouple individuals' right to a decent material existence from their ability to contribute to the economy, and also an Operation Warp Speed like project, built on government sponsored prizes, to accelerate Cures for All diseases. From there, we get Garrison's argument that ML researchers' collective power is currently near its peak but could quickly decline, his case for unionizing with the goal of demanding higher safety standards across frontier companies, and his advice for any who are thinking about becoming whistleblowers. On politics, Garrison is real about the fact that there's a lot of misinformation currently swirling around datacenters, but inclined to meet people where they are in an effort to build a big tent coalition. And on China, he makes a similar case to my own, arguing that the CCP values control and is therefore, absent extreme competitive pressure, not at all likely to rush into RSI. Finally, we talk about the organization Irreplaceable, which aims to win "a say, a stake, and a slowdown" in the political arena, and to which Garrison is donating his book royalties, and how interested listeners can get involved. The bottom line for me is that while I'm still an AI enthusiast and eager early adopter by nature, and significantly less worried about labor market displacement than Garrison, even though I do think it's likely to happen, I signed last year's FLI " Statement on Superintelligence" because I do firmly believe that racing to superintelligence via a recursive self-improvement powered intelligence explosion would be a very bad idea. And at this point, with that possibility looking more and more realistic, I think Garrison might well be correct that it's time to put our less important disagreements aside and focus on building the coalitions needed to exercise political power. With that, I hope you enjoy this discussion about who should get to decide the trajectory of frontier AI development, with Garrison Lovely, author of Obsolete.

Main Episode

[04:36] Nathan Labenz: Garrison Lovely, freelance journalist and author of the new book, Obsolete, the AI Industry's Trillion Dollar Race to Replace Us and How to Stop It. Welcome to the cognitive revolution.

[04:49] Garrison Lovely: Great to be here. Good to see you. Great to see you.

[04:53] Nathan Labenz: I think this one might be a little bit different from some of the conversations that you're having to promote the book. As you're probably aware, I'm very deep down the AI rabbit hole personally, and I to to the degree that I understand the audience of this podcast, I think the one thing that we all have in common is we're all AI obsessives. Some of us are enthusiasts. Some of us are doomers. Some of us are both. I put myself in the both camp to at least a, you know, significant degree. And so there's a lot of the book that I think does a great job of just sketching out, like, where are we in this whole AI phenomenon? Why should you take it seriously if you don't? And I think a lot of that stuff you've probably focused on in some of your other conversations and does, like, a lot of work with people who you need to convince to get over the hump and start paying attention to what's happening. I think in this forum, you don't really need to convince anybody that they should be taking AI seriously. And so I thought I'd come at it maybe from a little bit of a different angle and do maybe a little bit more of an ethnography or sociological study of the AI industry because you're somebody who has been pretty deeply embedded in it. I don't know if you'd use that term, but definitely in the mix. Think you have quite an interesting inside view as to what is going on. Think it might be a useful mirror to reflect back to people as you see the community that you've come to study so closely. How's that sound? That sounds great. Yeah. Let me just start with a real simple one. How do you use AI? You're a freelance journalist. What role does AI play in your life?

[06:28] Garrison Lovely: Yeah. I mean, so I don't use it to write. I think that there was a period of time where that was almost like, oh, like, you could try that and see if it worked. And then it quickly became stigmatized. You know, there's some people who always hated it, but then it became, like, widely stigmatized as it was used more widely and as we got Pangram as, like, this really high quality AI detector. But it's it's very helpful for a range of things that you have to do as journalists. You know, transcription is an obvious one. It used to be that, like, it just wasn't very good, and you'd have to do a lot of transcribing interviews by hand. NotebookLM from Google is just this incredible tool where you can feed in, like, up to 200 documents and ask questions of them. And it's you know, it doesn't it still hallucinates a little bit, but it's much more reliable, you can click into the specific doc. And so you could take, like, 20 interviews you did. You could find every quote related to this topic and be put it to a table with the person in the quote. And it's like, it's possibly missing stuff, and so that is a risk. But the the other ways to do this before were also pretty glossy. And, yeah, I think that can just, like, help you organize your thoughts and and get answers, like, much more quickly. And then, like, feedback, fact checking, research, it's a bit tricky because, like, the feedback, you have to just know what it's stupid about, which is still a lot of things. And so if you're, like, taking it too seriously, you can just waste time because, yeah, you kinda have to trust your gut, and it'll give you a better sense of where your intuitions are good and where they're not and vice versa for for the AIs. And then, you know, fact checking, it'll often say something is wrong and it's not wrong, and there's obviously stuff that it could miss. But there's tons of times where it'll catch something that, like, you can verify that you had gotten it wrong or were missing something important. And so and and then, like, with the book stuff, it's like, you know, the website for the book and create little, like, tools that are helpful, created, a ZIP code lookup so people could find local indie bookstores, and just all these things that would never have been possible before. And so, like, as an independent journalist who's also doing kind of multimedia stuff, it's very, very helpful. And it's been, you know, kind of like, it is a weird thing, right, to be writing about this technology that is potentially going to disempower everybody and aspirationally, like, going to put a lot of people out of work kind of at best, but then, you know, still be like it would be difficult to lose access to these tools. And then I've also gone through the phase of, like, you know, borderline Claude psychosis of, you know, just when I was writing the book, you know, getting feedback on the entire thing in Claude code in a matter of minutes was just incredibly helpful in some ways, but then you can also just go down rabbit holes of, like, needing to fix stuff that's not actually broken. And it was, like, almost this mantra of, like, getting, you know, positive feedback from the AI when I was in the darkest days of writing the book and just so burned out and tired. And that's just not a great place to be. And I think everyone who's used these tools a lot, like, has the experience of wasting time building something that is not necessary and doesn't even work necessarily and feeling like you're more productive, but you're actually just, like, you know, getting anywhere. And so it's it's tricky. And I think I've, like, dialed back my use in a lot of ways because it's just often not going to save you time unless you really know what you're doing or you know what it's good at.

[10:02] Nathan Labenz: How about I have some questions about the dark days of the book and the timing, which is proving to be, I think, pretty spot on. But how about zooming out even from AI and thinking about technology more broadly? I think one big thing about the AI safety community and culture that most people outside of it don't appreciate as much as they should is how many of the people who are concerned about AI are for all other technologies, like techno optimist libertarian, and that's basically been me. I aside from, like, never quite getting crypto, I've been, like, I've been waiting for my self driving car since I was a kid, and I'm all about future, hopefully, promised medical breakthroughs that will extend my healthy lifespan, abundant energy, all these things. Like, I'm excited about all of them. Where are you in just broader relationship to technology?

[11:00] Garrison Lovely: I was a techno optimist when I was younger, and I think the world was more techno optimistic, at least in in The United States and the West. You know, we were, like, told social media would connect us and Twitter would liberate people from authoritarian regimes, and Google was amazing, and Google Maps was so useful. And, like, we were just seeing improvement in how we lived in the world, and the technology seemed to be driving a a big part of that. And I'm somebody who now I I still deeply appreciate the power of technology, but I'm much more pessimistic about get us getting technology that is good for us under current conditions, which is, like, capitalism, and specifically, like, capitalism where they just maximize profits for share shareholders without much concern for various other stakeholders that that care about that are affected by by these companies. And so, you know, Corey Doctorow has a book, which my book is, in some ways, like, a rebuttal to because we both take the position that AI is not inevitable, but he thinks AGI is impossible impossible. But he has his previous book called, I haven't read, but the concept is incredibly helpful. And it's basically you know, a lot of you listeners will probably be familiar. But as a tech company, you start out trying to get as many users as possible so you'd make the product as good as possible. Then at some point, you reach maximum user, you know, numbers, and it's now about getting as much profit from the users as possible. So you, like, cram it full of ads. You, like, add these features that will helpfully make it, you know, more profitable, which maybe actually makes it worse. And I just feel like we're in the era where Google Maps just doesn't work as well as it used to, like, in a bunch of ways. You can, like, put paste in an address, and it'll just, like, take the first word of it and then send you to the wrong place even though the full address is in there. And it's like, what's going on? This used to not happen. And it's this incredibly crazy making and and kind of dispiriting feature of of modern life that we are just we have to use these products because they're the only ones around. And they're just getting worse in obvious ways, and we just don't feel like we have a choice. And then the only thing that's not, it seems, is AI, where it just does get better and faster and cheaper and just it's kind of, like, amazing how much steady progress there's been. But then that comes with this, like, terrible risk and and cost to society. And so, yeah, I I feel kind of bleak about it, but I'm still, in my heart of hearts, like, dispositionally optimistic kind of about humanity's potential to to pull together and and do amazing things. And I think we just, like, need to change the the structures and and the systems to deliver better technology that will actually make us, yeah, happier, healthier, wiser, more democratic. One of the funny things that has

[13:57] Nathan Labenz: kinda happened in

[13:58] Garrison Lovely: the

[13:58] Nathan Labenz: last, I don't even know, twenty four, forty eight hours online has been a sort of brewing of the stochastic parrot meme, which seemed to be a comfortable, dare I say, safe space for a lot of more leftist thinkers, including even people who I would say should have known better because they had all the fundamental knowledge of AI that they should have needed. But we've heard a lot over the last couple of years from all sorts of people on the left that basically like, it's fake. Right? They're just hyping their stuff. This is nothing. It's just tech companies trying to raise money or boost share prices or whatever, all that sort of cynicism. Why do you think that has been so prevalent? Do you have a theory of why the left has buried their head in the sand broadly on AI?

[14:50] Garrison Lovely: Yeah. I I think there's no one explanation, but a few. One is that the the most prominent influential people in AI who are also on the left have taken this very hard line, the sarcastic parrots paper being the kind of quintessential example. And people on the left will just, like, defer to folks who have, you know, PhDs and apparent credibility and then also share their values. That's been a dominant view. The left has also been, like, you know, prior to Bernie running in 2016, really only powerful within academia. And academia has been hostile to this in a lot of ways, like AI being real. And there's maybe some, like, humanities versus, like, STEM antagonism happening there. And then I think crypto and NFTs and the metaverse and just in social media, like, there's a lot of hype about those being transformative and positive technologies. They're either not transformative and not positive or transformative and negative as I think of the social media and and and crypto by and large. And so a lot of people just pattern match to that and were like, oh, these tech people are just full of shit. They're just always saying that this thing is going to save the world, and then it's like, you know, b two b SaaS or something. And they just, yeah, like, got stuck in this frame. And then I think there's just, like, some amount of

[16:19] Nathan Labenz: of cope

[16:20] Garrison Lovely: and denial because it's just it's terrifying to consider that, like, these companies could make machines that could replace us. And even if you don't believe in, like, the full on extinction risk stuff, just just that alone for your job. Like, being unemployed is terrible. It's really, really bad for you. And then having that happen, like, society wide scale, very bad for society. You know? Nazi Germany rose out of that. Like, it's reasonable to be very concerned about this. And and then, like, the bubble argument has also been very popular on on the left, and, like, Ed Zitron is, like, the main guy who's promoted that. And I think just similarly well, one is that Ed has been very persuasive and effective at reaching people, but he's just either doesn't know what he's saying or he's lying because he's constantly saying things that are just not true and, like, so easy to pick apart if you know anything. And, again, like, in in my chapter on the bubble, just I talk about, like, you know, like, it's a comforting story, and it means you don't have to do anything. Whereas, like, my version of reality is that these companies are trying to build machines that will replace us. They might succeed, and that would be disastrous for so many reasons. So we have to stop them, and that's really hard. We have to get organized. And I think that is a message that could work, and I really hope it does.

[17:38] Nathan Labenz: But

[17:38] Garrison Lovely: it it's, you know, one that requires you to not just, like, read about stuff and post and, like, but, like, you know, do things in the world. And so I think there's, like, a kind of overhang where there's a lot of people who have concerns and but they don't wanna get yelled at by sharing them on Twitter or Blue Sky. But I do think there's this kind of mismatch, and we're starting to see the dam break, and that's been really encouraging. And I'm hoping by laying out the whole case from start to finish and and just also showing that I'm, like, coming from a similar perspective, we can, like, you know, get people to to on the left to, like, take this more seriously. Because once you do, it's like, oh, capitalists are trying to fulfill the lifelong dream of turning capital into labor without the intermediary of workers. And in so doing, take the share of returns to capital to 100 and labor to zero. Like, seems pretty bad. Like, I think that's bad for from almost any perspective. Like, maybe some libertarians are into that, but I think the libertarians are still concerned by and large. And so this feels like a pretty easy one for the left to be like, no. This is a real thing. We should get on board with it and and and, you know, not let them build those machines.

[18:54]Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive

[20:26]Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

Main Episode

[21:45] Nathan Labenz: I'm interested in how you conceive of what it is that the companies and to the degree you wanna zero in and speculate on, like, individual executives at the frontier AI companies. What do they really think of themselves as doing? I do agree that, like, in setting out the excuse me. In setting out the mission to create AGI, which they define as something that is better than humans at the vast majority of economically valuable work. It does say right there on the tin that this is like a human replacement or at least like a human substitute technology. But then there's the other part of the mission, which is to make sure it benefits all humanity. Right? So typically, when people ask me what I think, I start off by saying, I don't think that they are well modeled as just trying to get as rich as possible. I think they're a little more utopian, ideological, something else other than just purely profit motivated. What do you think? How do you think of what they really want? Yeah.

[23:00] Garrison Lovely: Well, this also reminds me of another reason why the left is not taking this so seriously. The most prominent people talking about existential risk from AI and AGI and superintelligence, it's like Elon Musk, Sam Altman, and then Eliezer Yudkowsky, Nick Bostrom. Like, these are people the left does not like. There's a mutual distaste. And so I think that negatively polarized a lot of people. But to answer your question, I think that you're right. Yeah. It's like my model of it is the people who really started these AGI companies were chasing mission, and then they kind of toiled in obscurity until they made enough progress that the profit seekers were like, holy shit. This is amazing. And then they started investing massive money into it. And the leaders of the companies are still motivated by some kind of idealism, sense of inevitability, sense of, like, megalomania, you know, messiah complex. You have it. Better us than them. Yeah. Exactly that. But then they're pressured by these investors who you know, when Sam Altman was fired, it's a bit unclear, like, whether he's like, oh, I didn't even wanna come back, but people, like, kind of asked me to come back. And it's like, I I I'm skeptical of that. But but the investors, and specifically, I think Thrive Capital was a big, big player in restoring him. And that just makes sense. Right? Like, once you invest billions of dollars into a company, you're gonna have a lot of interest in what happens to that company, who's leading it. And having a bunch of safety conscious people who wanna, you know, replace the CEO who's presided over this meteoric rise, like, that's just not gonna fly. And this is a big part of why I'm so pessimistic about things going well under the status quo. Like, setting aside just that it would be really difficult to safely and democratically introduce AGI into the world under any circumstances.

[25:03] Nathan Labenz: Like

[25:04] Garrison Lovely: and because it's a universal labor replacing machine, it would just really turn society upside down to have that exist. But then to have it happening, like, as fast as possible with, like, minimal regulation in a country that is cutting the social safety net or adding work requirements to Medicaid. Like and then all the other countries wouldn't have a chance to even tax the companies that are putting their people out of work. That's just, like, a pretty to to me, like, obviously, really risky proposition. But, yeah, the the people leading these companies are are not really, like, profit maximizing. Greg Brockman has that diary entry where he's like, what will take me to $1,000,000,000, which is a pretty crazy thing to write in your diary of, you know, the nonprofit you cofounded. And now he's worth, like, 20 something billion. But I think, yeah, Sam and Dario and Demis and Elon, I think, are all motivated more by our you know, trying to be a great man of history. We can go into the I know you wanna break down the individuals, and I think it's hard to generalize because they're all unique. But to, you know, close out an overly long answer, I I I think it is largely not about money. It's it's about being in the room where it happens. That's what Eliasir told me when I interviewed him back in 2023. And, like, people just wanna be there for creating AGI, creating superintelligence because that is, like, where humanity's fate lives in in in their mind, and that might be true. And they wanna be in the room where it happens, influencing how it happens.

[26:36] Nathan Labenz: Yeah. I think Sam Altman has spoken remarkably candidly about this a couple times in the context of describing what it was like to be in the room when the first reasoning demos were shown.

[26:50] Garrison Lovely: And

[26:51] Nathan Labenz: he said I forget his exact wording, but he said it's happened Like pushing back

[26:54] Garrison Lovely: the frontier or something, the veil of ignorance.

[26:57] Nathan Labenz: Yes.

[26:57] Garrison Lovely: The curtain, which is not exactly what that was about.

[26:59] Nathan Labenz: But But, yeah, there's something and I am sympathetic to that in the sense that I think it is extremely exciting, even intoxicating to be in on the secret. I and I do yeah. I I can understand that to a degree. I guess zooming out, and I am interested to hear your takes on individual companies and their cultures and the individual people at the top that are shaping those. How much do you think and you mentioned the term overhang, and it got me thinking about I'm torn or ambivalent on this question. It's probably both is always the answer. But on the one hand, I do feel like AI broadly is inevitable in the sense that we have web scale data. We have web scale compute. And in the presence of those things, I feel like a lot of algorithms ultimately can work. We found one main one and there are bunch of derivations of it that work. And so I feel like we're getting AI absent some sort of civilizational reset. That means we don't have webscale data and webscale compute. And I'm not excited about that proposition. We're probably getting some AI. But then the shape of AI, the conditions in which it's introduced, like the the measures that are taken or not taken to make sure it goes well, all of that stuff seems far more contingent. How do you think about, like, how much is inevitable and where we might be able to draw lines, then you can go off in any number of directions in terms of the influence that individual people or groups are having on the direction we're taking?

[28:37] Garrison Lovely: Yeah. I mean, I think it's inevitable that we will, as a species, continue to use deep learning to make AI models that do things and, like, new things. Right? The in the book, I separate the obsoleting project from other types of AI, And that's my reframe of the AGI industry because they're trying to render us obsolete. And you mentioned OpenAI's AGI definition, and and Dario Amade has a quote that's saying, like, AI is not like other technologies. It's a general substitute for human labor. And that feels, like, pretty different. And we've taken AGI to be synonymous with, like, what AI is because that's the companies that have been trying to build it have been the best at building highly capable and autonomous systems, and they're driving the entire world economy now. But that's not, like, the only type of AI we could have. Right? We could just be like, we have all these diseases and medical problems, and, like, can we build AlphaFold type systems for specific problems that we have? And, like, it's obviously I get the vision of, like, build the superintelligence that can you know, solve intelligence, use that to solve everything else was DeepMind's mission statement for a while. It's like the ultimate techno solutionist fantasy. And from a purely technical perspective,

[29:57] Nathan Labenz: like,

[29:58] Garrison Lovely: yeah,

[29:58] Nathan Labenz: I I

[29:59] Garrison Lovely: can

[29:59] Nathan Labenz: see

[30:00] Garrison Lovely: how you could, like, use this thing to invent all kinds of wild and transformative new technologies. But I think that you can't, like, solve, you know, ethics or ideology and politics in the same way. And, yeah, I I think it's like this kind of revenge of the STEM people on the humanities or something where it's like, we don't have to, like, learn history or philosophy or or political theory. We can just build the machine that's smarter than everybody and then ask it what to do, which I think is a really

[30:36] Nathan Labenz: Yeah. Was one one time OpenAI's business plan, as I'm sure

[30:41] Garrison Lovely: you're

[30:41] Nathan Labenz: Yeah.

[30:42] Garrison Lovely: Well aware. Ask the AI how to become profitable. Right?

[30:46] Nathan Labenz: Yeah. And he said that that was an Altman quote. He said that, and there was a laugh in the audience. And he's like, you laugh, but I'm not really joking.

[30:53] Garrison Lovely: Yeah. Yeah. It's it's just like the ultimate, like, question mark question mark question mark profit thing, which is like, yeah. Just make the superintelligence. But, yeah, I think that we can stop the obsoleting project because it's really just a handful of companies in two countries. And only one country has really advanced the frontier since ChatchBT came out at least. And it just costs so much money, and it requires the most advanced technology in the world. These, like, AI chips, which are only made by, you know, hit like, in individual companies control different parts of the supply chain as your listeners probably know. And I think, you know, like, forever, can we stop this? Like, I don't know. Hopefully, we're around for a very long time. And I my position is not that we should never build AGI. It's just that it should happen with strong public buy in and a scientific consensus so it can be done safely. We can get into what that looks like later if you want. But I think that, you know, for for your listeners, it's like, right now, we're kinda feeling, like, trending towards just banning AI across the board or something, which would be very hard to actually do and especially with, like, open weight models and yada yada. But the kind of, like, backlash is really intense and kind of undiscerning. And I'm kind of hoping to separate the obsoleting project from other types of AI and really stigmatize the obsoleting project because it's it's risk and undemocratic nature. But then, like, save the baby from the bathwater with, like, you know, these kind of specialty systems, which can be used to do amazing things. And my position is that, like, we can get way more of the amazing stuff if we have, like, a more active role for democratic control in in deciding, like, what gets done. So, like, drug discovery. You know, people are like, oh, AI for drug discovery. Well, monopoly patents mean that people will still try to just discover drugs that will be profitable, which won't be ones that cure people as much as ones that, you know, treat some chronic thing or male pattern baldness or, like, you know, these things that are not as socially important. And to actually get, like, the best from AI assisted, you know, drug discovery or AI assisted FDA trials, like matching people, you know, deep learning systems are very good at that. To get the best of it, you have to, like, reform the systems and change how, you know, drugs are what like, how we decide which drugs are made. And I think we should use, like, a prize system, which, you know, you kind of pick the drugs that you'd want to see in the world and then award money and then just make them at generic costs once they're once they're developed. And so I kinda like it's a judo move, but I'm kinda like, we aren't gonna get the booster world by letting it rip. Like the one that they depict of, like, amazing, transformative cures for everybody. And the best way to get there is to, like, actually take a a strong position against being replaced and then use industrial policy and, like, the operation warp speed type approaches to build the types of technology and and and scientific discoveries that will actually lead to the most public benefit.

[34:13]Deepgram Flux TTS: Deepgram Flux TTS is a streaming text-to-speech model built for voice agents with natural tone, context awareness, and interruption handling. Try it free until September 12 at https://deepgram.com/keep-talking

[34:43]OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

[36:36]Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[38:11] Nathan Labenz: I have a lot of different

[38:12] Garrison Lovely: directions

[38:13] Nathan Labenz: I wanna go, but I guess maybe part of a question that I wrestle with. It's a little bit hard, of course, to define some of these terms and what are the boundaries. But, like, I get so much value out of using AI on a daily basis. It saves me an unbelievable amount of tedious work, and it allows me to do a ton more than I otherwise could. And it, in a way, has replaced, at least, like, counterfactual people I would have had to hire in theory, whether I would have or not, I'm not so sure. But, like, there's definitely a lot of work happening in my life by AI that on some counterfactual level has substituted for human labor. And I'm like, think that's good so far at least. Like, I I do you how do you think about, like, where this goes from bad or goes from good to bad? Because I I think there's a strong argument, and I do wanna give it its kind of strongest articulation that, like, you go back not that long in history and everybody was tilling fields. Right? And it it was, like, pretty shitty. And now we have certainly some problems in our modern agricultural system, but, like, one problem we don't have is scarcity of food. And there's, like, plenty of with 2% or whatever the population, we can feed ourselves and everybody else is, able to do other things. Almost everybody agrees that's almost overwhelmingly good, let's say. Even again, though there are problems that remain. Can we not have a version of that with AI where we're all elevated to our own little executives of our many AI corporations or something? Are you at all sympathetic to the or maybe we'll work a lot from last. We had the Canes thing too from a hundred years ago that was like, we're supposed to only be working two days a week at this point, but we're not. But maybe So waiting. We could be.

[40:06] Garrison Lovely: Yeah. No. The the faster AI goes, the more I work, it turns out. But I yeah. I mean, automating labor is what's allowed humanity to go from everyone more or less being very poor to some people being rich at least and, like, living standards and all kinds of stuff going up dramatically after the industrial revolution. And so it's like, my position is that we not that we never automate any labor because, like, I don't wanna just have a stay at this kind of, like, level of development. But trying to automate all labor is pretty different. And where the line is is, like, it's not super obvious. I think that, you know, my position is we should freeze frontier development right now and not resume without the buy in and and safety I mentioned. But it's, like, three years ago, FLI organized this this pause letter, you know, after GPT four came out, I guess, three and a half years now. It was, a six month pause on on development, and that was framed around a bunch of things, but risk was a big part of that. And, like, obviously, wasn't super risky to build the next iteration of of LLMs, at least from a, like, existential perspective. Like, there were harms. You know? There's, like, chatbot psychosis and and suicides that, you know, opening eyes decisions contributed to, which I document in the book, and I think that should you know, we shouldn't lose sight of that. But that's not really what that letter was about. I think now we're kind of in this position where it's like, we just froze what we had today.

[41:38] Nathan Labenz: Be

[41:39] Garrison Lovely: still a lot of disruption from just adopting g p t six and Fable 5.1, you know, in the economy. And the job effects are are not super easy to see, and it's like a lot of it's not people getting fired, but people people just not getting hired in the first place, kind of as you described. And I'm not,

[41:56] Nathan Labenz: like,

[41:56] Garrison Lovely: saying that we should go back a few generations. You know? That might actually be the right thing to do from, like, a social welfare perspective. I don't know. But it's just much harder than stopping advancing further. And so I think we should focus on that first and then kind of reevaluate with, like, what we have, and it'll take a lot to, you know, regulate the AI that we have today and a lot of thinking to to get that right. But yeah.

[42:22] Nathan Labenz: I mean,

[42:22] Garrison Lovely: on the CEO of our own corporation thing, it's just like some people will do well in that system and and enjoy it more than the status quo. But I think it's just like a lot of people will be left behind, and not everyone wants to, like the the experience I I've read and and heard of, like, managing these suites of agents, it's, like can be, like, really bad. You can feel, like, the opportunity cost of not kicking off another run and people, working more and more. And, you know, they're competing with other people who are adopting it really quickly. And so I'm kinda looking around at, like, this world where people are just, like, kinda burning themselves out running these, like, agent fleets and getting more done, but not necessarily, like, being happier with it. And I don't know. I'm just like, it doesn't seem like the the future the the good future that you're describing, And this is bracketing all of the risks and other social harms of, like, widespread adoption of this tech and then the power concentration and wealth concentration. It's like some of some people will be way better at running the AI companies, and the AIs will probably, if they're that good, be better at just running without any humans involved at all. And then, like, who is owning those companies and who is accountable for what they're doing and, like and then you have, like, these multi agent dynamics where, you know, the Hugging Face hack now, and there's, like, 1,200 agents who are involved with that. But the world you're describing, it's like there are multiple agents running for every person, interacting with multiple agents for other people and companies all the time in ways that we can't monitor effectively and producing, like, who knows what kinds of interactions and effects. And that just feels like a much less legible and stable world. And we're already seeing, like OpenAI is like I I tweeted that it was a loss of control event masquerading as a company in response to the fact that some like, three random people use Claude and Codex to get access to ChatGPT employee accounts, and they could have gotten access to, like, I think their main code base. And it's just

[44:33] Nathan Labenz: like,

[44:35] Garrison Lovely: this is and then multiple agent forms that have broken out of the company with or without their knowledge or some amount of covering up, some amount of cluelessness. And, you know, OpenAI is probably one of the companies that's adopted this technology the most, and it means that they just don't know what's going on nearly as well as they would have a few years ago. And so I think that the world of, like, these agents being widely adopted is one of, like, yeah, confusion and chaos and lots of unpredictable, but many, many negative consequences that we're already seeing, like, hints of right now.

[45:10] Nathan Labenz: What do you think the new if you envision a social contract that you you know, what's your most positive vision of the future? If you're not going to roll AI back from where it is now, And I think you've got some interesting proposals around you may alluded to, like, freezing the frontier. But, again, that still allows for a lot of diffusion. I totally agree this will probably like, seemingly all recent technologies probably amplify inequality. Hopefully, it brings up the bottom, but probably the ratios also continue to climb. I've lived through a medical emergency, thankfully, in the AI era, and I did get an unbelievable amount of value from using three AIs in triplicate and feeding in my test results, and it was my son's test results. But Yeah. Just unbelievable amount of value from that sort of thing. So I do feel like I want and any good version of the future for me, there's, like, AI doctors for all, which notably, OpenAI has done a pretty good job of making their health product free and unlimited to people, which is, like, pretty cool. Think as OpenAI moves go, that that ranks near the top of my list. What's your positive vision for where we wanna be in three to five years if we freeze the frontier and allow other things to continue? Like, what does good look like to you?

[46:36] Garrison Lovely: Yeah. I guess,

[46:38] Nathan Labenz: like,

[46:39] Garrison Lovely: in The US, I would like to see the lefts winning elections and building kind of a third new deal with the great society being the second. So, you know, Medicare for all, maybe a jobs guarantee administered locally. And I'm really bullish on investing a lot of money and more importantly, kind of like institutional and state capacity in developing technologies and medical treatments and and things that we, like, actually really want as a society. So one idea I'm, like, playing around with is cures for all, where the government kind of treats diseases like they're all COVID and it's all operation warp speed. You'd obviously have to prioritize based off of, you know, tractability and, you know, the disease burden and, you know, other factors. But then when you're there, like, using the whole government approach, using advanced market commitments, you know, vaccine trials, and sorry. I'm spacing on the word for challenge trials, like, where where people deliberately exposing challenge to the yeah. Human challenge trials. And, you know, developing, like, the the stuff that the companies often promise, but just going for that directly. And then using AI where it's helpful, and then in a lot of cases, it's really about the institutions. And then, you know, this would be enormously beneficial to the domestic, you know, US, but then you could also, you know, make the make the cures, like, freely available around the world or provide them at cost or or whatever and make the world healthier and also, like, restore our standing in it after some well deserved drops. And I think that, like, The US is so wealthy, and that wealth is just concentrated so intensely in in the top. That's been happening for decades, and AI and tech is just accelerating it further. And so I think just really large redistribution is justified and like, on just democracy and power grounds. Like, right before we recorded this, we got the news that Larry Larry Ellison's bid to buy Warner Brothers, which would give him and his son control of CNN, HBO, Warner Brothers as in addition to CBS and, yeah, and also TikTok and then Elon on Twitter. Like, we're in a really dire situation where these oligarchs are buying the most important media properties in the world and then using them to push their agendas. And this is like I work in media. Think media is really important. And, yeah, it's it's not just a matter of, like, oh, they have more money than they need and, like, other people have not enough. Like but it's, like, actually a threat to our democracy to have people being that rich. And so and, yeah, and I would like to just see as much decoupling between, you know, wage labor and living a good life and and building, like, a a proper welfare state. And then also a restoration of of USAID and and, you know, rejuvenation of it because I I think that, you know, global poverty is incredibly important. And the the cuts that Elon led, I also document in the book and just, like, the the deaths that those caused, just, like, one of the greatest crimes, maybe the greatest crime of the 20 century. And, yeah, just kind of like a restoration of, like I don't know. Just like building a prohuman society that is, like, truly egalitarian and one that, like, wants good things for people. And, like, at a true like, that sounds so cheesy, but, like, right now, we have an administration that is just staffed with, like, seemingly the worst people on the planet and seems to either not care about what happens to other people or want bad things for them and is just enriching itself at the expense of literally everyone else. So the opposite of that would be would be really nice.

[50:27] Nathan Labenz: I'm with you that the destruction of USAID is incredibly shameful. I've been a Elon defender in many conversations over time, and that's one thing that he's done that I've never defended. On the question of there seems like a little bit of tension between a potential jobs guarantee and decoupling one's ability to contribute to the economy from one's right as we might imagine it to a decent material life. Do you feel like are you a believer in the intrinsic value of work? People need work for purpose or for something to do. I'm a little bit more of the mind that I don't know. I'll bet on the working class to spend the peace dividend is what I told oh gosh. It doesn't matter. Jake Sullivan, whose name I should definitely know at the tip of my tongue, but he was at the end of an hour with him, we were talking about China. He was like, we didn't even get into jobs and what people are gonna do. And I was like, I bet on the working class to spend the profits. You worry about China. What do you think though? Do you think that we need jobs indefinitely?

[51:36] Garrison Lovely: I think I mean, one, will say, like, I need to give all of this more thought. I've been so neck deep in AI world that, like, figure out I kinda wanna make sure we're not all replaced by machines before we can get into, like, planning the, you know, third new deal. I think that you can have a jobs guarantee where, you know, it exists for people who want it, but then also make sure that people are getting health care and, like, the the education, housing, like, like, just the necessity is covered because, you know, we we have an incredibly rich society, and those resources are not being put to good use. Like, rich people, they just wanna be richer than each other, and they maybe wanna use power in the in the world, but most of them don't seem to even care about that last part. They just kind of wanna, like, be richer than their friends or something. And so just you could just tax them really aggressively, and they won't like it, but they'll still get to, like, have more money than the next person if you do it the right way. And so I don't think we need to choose. I think that work does seem to be it's just a thing that's very important to a lot of people, and I think we should try to not have so much of our meaning tied up in it. But it's also just like you look at polling. It's one of the few issues that polls at, like, 80% or something, and UBIs poll very badly. And so I don't know. Like, I believe in democracy at a deep level, and so I think part of building, you know, the third new deal is going to be running unpopular things and figuring out how to make them work. And and, yeah, just like jobs exist for people who want them, and there's also a lot to do. There's a lot of, like, climate, you know, like, green transition stuff you could have people do. The New Deal, like, I think it was James Baldwin was a writer for the one of the programs in the in the New Deal, and they documented they, like, recorded a lot of important things that were relevant for, like, preserving culture and history, and, like, lots of really cool stuff came out of that. And, you know, it just pay artists to make art. Like, there's so many people who wanna create things, and our society has decided to devalue that as it dumps unprecedented amounts of money in building machines that can replace all of us. That just seems like backwards. And so I don't know. Like, this just seems like we're doing almost everything inverted from how I think we should be doing it.

[53:57] Nathan Labenz: I'm interested in your response to an argument that you sometimes hear around the relationship between invention and democracy. People sort of just make the very short observation that, like, in general, people are allowed to invent stuff, and there it's not like we don't put every new invention to a vote. And if we did, we might not get very many inventions. We might freeze a lot more than we would like to freeze in time. Friend of the show, Dean Ball, also on Ezra Klein talked about how he wonders if I think it was on Ezra Klein. If today we would have the stomach for the introduction of the car, which was, like, disruptive in its own way. And it was sort of cars and horses on the roads together and, like, they weren't safe at the beginning. And they're still not entirely safe, but they were way more dangerous then. Are you sympathetic at all to that, or how would you answer the idea? And I guess there's another question around democracy around just how well is it functioning in general. The for better or worse, I do think the president was, in fact, legitimately elected. So I do wonder, like, how much can we really and I'm a big direct democracy guy, by the way, in general, but I don't know that we can just fall entirely back to as kind of the end all be all decision maker here because it it seems like we've got plenty of examples of bad decisions being made by publics and also might not be willing to embrace enough change to really see the future go where at least I would hope that it could go over time.

[55:36] Garrison Lovely: Yeah. Well, first, I'll say I live in the only city in America where most people don't have a car, and it also happens to be the best city in America. So I don't know.

[55:45] Nathan Labenz: Cars, like,

[55:47] Garrison Lovely: I don't know if they were good on that. Like, probably. I don't know, though. But that's not important for for my particular point, which is, like, I think permissionless innovation makes sense for most things, almost all innovation. I think that should be the default. But building a universal labor replacing machine would have a profound and irreversible effect on everybody in the world. And so I think it's reasonable that everybody has some say in in whether, when, and how that happens. And, you know, broadly, like, my position is AI, at least what they're trying to build, the obsoleting project. Like, that is unique. It is different from other technologies, and so it should be treated differently. And on your point about democracy and and, you know, Trump being this counterexample, Trump is a product of a broken democratic system in The United States where, you know, you have the electoral college being the most obvious example where he won in the first place but lost the popular vote. But more importantly, you have counter majoritarian institutions like the US Senate with the, know, crazy way representation is apportioned and gerrymandering to a lesser extent, and then the Supreme Court being these lifetime appointments, and the Republicans stole a Supreme Court seat. People should remember that. And so, yeah, I think we have and then the first past the post two party system creates a political system that the majority of people don't like. I think it's, like, two thirds of Americans are not happy with their current level of representation. And so democracy isn't just like, is there an election that decides with also some other weird stuff tacked onto it who leads the country? It's like, do people have meaningful say in the power that affects them? Like, do the governed govern? And The US is just not great on this, and a lot of other countries are are better on it. And, you know, we're the longest running democracy in some sense, although really wasn't a democracy until the nineteen sixties when everybody living in the country got the right to vote and, like, meaningfully. And so, yeah, I I think, like, proportional representation is a thing I talk about in the book as, like, a better way to elect congress where, you know, people would have, like, larger districts with multiple members, and then the top five vote getters would be in office. And then you'd have probably, like, five parties instead of two, and then people would have parties that represent their interests better. And that would be, like, a huge change, but so so, like, better at representing people. And then electoral college is one state away from being gone. If Pennsylvania, I think, signs the interstate compact to just use a popular vote. And so my pitch for democracy is, like, one that is more, like, well realized. And this also extends to, like, democracy in the workplace. So I think it should be a lot easier to form a union and a worker cooperative. And I think, you know, in this country, we accept that, you know, it's like the government should never, you know, impede on us and and censor us. But our employers, where we spend, like, eight hours a day, five days a week, can surveil us and control what we say and do all kinds of things that we would find incredibly authoritarian in another context. Record all of our keystrokes

[59:08] Nathan Labenz: and

[59:09] Garrison Lovely: Yeah. Mouse clicks and use it to train AIs, for example. Yeah. Yeah. And and so, yeah, I think, like, democracy should extend to more parts of our our society and our life. And I think that this will produce on net better decision making, but then it will also just create a more empowered citizenry where, like, it provides something intrinsically to to have say over over your life. And I think people right now, especially in The US, just feel incredibly disempowered, incredibly unheard. And I think that's reasonable. Right? Like, right now, we have a government that's uniquely insulated from public opinion through these kind of majoritarian institutions, the fact that it's this lame duck president and just one who, like, does not seem to care very much about things beyond the ballroom and corruption and, yeah, like, the stock market. But yeah. So I don't know. I democracy is great, and America should be one.

[1:00:10] Nathan Labenz: Change gears a little bit on the notion of people being empowered. I think we'll I'm sure we'll circle back to some of those bigger themes before we end as well. But I think you have an interesting argument in the book for how the AI researchers should understand their position today. I think you had a column from just a couple months ago, right, that was like AI researcher power is reaching its peak. And I actually don't know too much, but you talk a little bit about the vote at DeepMind or the sort of organization effort at DeepMind in The UK to bring about a union to the DeepMind staff there. Tell me about how you see I'd be interested to hear that story because I I really don't know a lot of the details of it. But then beyond that story, give me how you see the lay of the land in terms of the power that the employees at the frontier companies have, whether or not they realize it, how you think they should use it.

[1:01:12] Garrison Lovely: Yeah. I mean, so the DeepMind thing, Google has been signing these deals with the military, US military, Israeli military, and this is creating a lot of pushback with from workers. In The UK, the DeepMind division had a union vote. It's a I don't know British labor law. It's pretty different from The US, but it seems like something like 300 of a thousand people in the bargaining unit were supportive. And that and Google is not recognizing it, and they're not demanding better pay or working conditions, but instead policy changes on on who Google is serving and and how. And I think that's pretty interesting because, you know, most unions are like, it's about getting better pay and working conditions for for the members, which is, you know, a reasonable thing to want. In these situations, like, people are paid very well. I mean, they work a lot, but they often want to. They have, like, good con you know, good perks or whatever. And and to the broader point, like, the industry is racing to replace all of us, but they're starting with their own workers, specifically the engineers and the researchers who are training the models. And the hope is to achieve recursive self improvement where the AI can fully train the next generation and make it more capable, and then you can have this really fast loop. And I think that means that we have a ticking clock here. I mean, the workers have a ticking clock where their power will you know, it's near its peak because they still command incredible salaries, and they're still needed. But if they're not needed anymore, then they won't have any leverage. And then that will happen to the rest of us, which should be very bad. And so I think people in the companies are starting to see this, but probably not as much as they they could. And one thing we've been seeing is, like, if you oppose what's happening at these companies and you work there, you should just quit and then go public. And Jacob Coxen did this with, like, incredible fanfare, and and it really moved the conversation like nothing ever has. And, you know, you will see a few other examples since then. And, you know, Coxen deserves credit for this, and I think it's like you know, as a whistleblower, I think it's sometimes the right thing to do, but it might be better to stay and organize your coworkers who also care about safety and try to form a union. And in so doing, you can bargain for in The US, you can bargain for safety. And this is something that airline pilots, I believe, did, and this is, like, how the FAA was formed, I I believe. And you will have so much more power by being able to withhold your labor collectively, and you'll have protections in in the union. And if you really wanna slow things down, like, a big, dramatic, messy union fight is a pretty great way of doing that. And it would be like, if it was truly about just safety, then it's also this clarifying moment where the CEOs will say, like, I care deeply about safety. I wanna slow down. I wanna pace the frontier. And then the workers could say, great. We wanna form a union. Will you voluntarily recognize us? All we want is to make the AI safer. Also, by the way, if we're in our own unions, we can work with the other unions to collectively pace the frontier, and it doesn't violate antitrust law. There's a precedent, I think, of the Teamsters doing doing this across different shops. And so my guess is that these CEOs would not voluntarily recognize the workers. But if you're optimistic about your CEOs, you can just give them this opportunity to be like, great. Here's this creative way we figured out to to actually slow down. And I think that this is very promising, and, like, a lot of people at these companies just don't have experience with that type of thing. But it's something I I want them to learn more about. And, yeah, like, you can have a lot of leverage over what happens through this through this approach.

[1:05:19] Nathan Labenz: I don't know to what degree you have chapter and verse on this, but the biggest relevant offices are in California under kind of California rules. Do you know how much latitude or protection people at tech companies would have to organize? I would assume that they're protected from being fired for attempting to set up a union. Yeah. Our outside union representatives afforded some rights or privileges to be able to come in and try to organize people. I haven't really thought too much about this, but it does with all of the talk that we hear lately about antitrust and why we can't do these things because of antitrust. I've been saying, okay. The president should just say, we're not gonna come down on you for antitrust violations for coordinating on safety measures. In absence of that, this is another pretty creative solution that I think would be very hard for anybody to argue with, although I'm sure we'd get the usual bad faith attacks. But to the degree you can, and I I realize I'm giving you this in an unscripted way, but I'd love to hear the double click of what you think that could look like and what people on the inside should know such that if they are at all tempted about this, they could feel confident in taking, I don't know, what sort of next steps.

[1:06:44] Garrison Lovely: Yeah. It's been over a decade since I took labor law, but there are protections against retaliation for organizing and then protections, like, once you're in the union as well. The protections are kind of weak. You know? Like, the back pay is often like, you can be litigated for a long time. But there's really, with these companies, like, it's just a very bad look to fire people for trying to organize a union around safety. And so I think that is in itself a lot of protection. And at OpenAI in particular, we see a lot of culture of people speaking out. So I I think that, yeah, there there's protection in in that kind of, like, reputation management. And then in California, in particular, workers can bargain at the sectoral level. And so, like, fast food workers can bargain across different shops, I believe. And so you have extra benefits there. And if people are interested you know, I'm not an expert on this, but if they're interested, they can get in touch with me. My signal is Garrison dot zero six. And, yeah, like, I I know people who would know more.

[1:07:45] Nathan Labenz: In terms of what they would be organizing for, I think the book has quite a few different interesting arguments that kind of take the the assumptions or the the hopes that people on the inside often have and at least give them a good shake. One is just like that the companies are serious about safety in the first place. I think it'd be helpful maybe to just review for a second the history of safety at OpenAI, which you've reported on at some length. We've been through, like, waves of different regimes and leadership, and there's been a lot of turnover. Then I think you also have a pretty interesting take on alignment basically as a mirage, putting yourself maybe in the role of, like, adviser to the hypothetical union leaders. Why would it not be enough for them to say, okay. We got 20% of compute committed to safety,

[1:08:46] Garrison Lovely: and

[1:08:46] Nathan Labenz: why wouldn't it be enough for them to say, oh, we're gonna solve alignment first, and then we'll go do the thing?

[1:08:53] Garrison Lovely: Yeah. Well, the super alignment team, as you're alluding to, was promised 20% of compute for that team and then got very little of it and nowhere close to 20% in practice. And I think to varying degrees across the companies, we've just seen a lot of promises made about safety and commitments that then get changed or broken once they start conflicting with commercializing or, you know, racing ahead. And it's because, like, there isn't a counterforce. You have management able to unilaterally make these decisions, and employees can push back, and they can go public, and they can, you know, whatever, like, you know, try and resist this in some way. But if they're not organized, you're just not going to have the the means of actually changing the policy. And, you know, we've seen employees getting organized enough to change outcomes with Sam Altman's firing and then reinstatement. The employees coming together to sign that letter, like, 90 plus percent of them signed was, like, a really big part of that working. And, obviously, the dynamics there were pretty different. They stood to lose a lot of money on the, you know, sale of their shares. But, yeah, I think workers can, like, decide, like, we really want these policies. We want these safety practices, and we want them to be, like, binding in some way. And you have leverage by being able to withhold your labor, go on strike, do, like, work slowdowns, etcetera. And I think that we just can't take these companies at their word, and the CEOs are saying that they can't unilaterally slow down because they're racing each other and, like, you know, they could. And it would actually be a pretty strong signal that you take these risks seriously if, like, any of these companies unilaterally slowed down. It would put a lot of pressure on the others to do the same, and it would also help with The US China thing if, like, you know, one of the parties unilaterally disarmed that, like, really does actually signal, like, you care about this. And The US is in the lead, so it would signal it much, much more. And this is my understanding, like, of the biggest, if not the biggest blocker to a deal with China is they don't think The US is taking it seriously because we're barely regulated over here. And we have been the ones who started the race and will win it, as, you know, Trump has said. So, yeah, I think it's just about power. It's like you just need hard power to actually get concessions because otherwise, you'll get promises that will be broken as soon as they start to cost too much. And you can, yep, just make it about only safety. And and, like, in terms of ideas, like, you know, third party auditors is being discussed and, like, making that not voluntary. You could also have, like, your demands be about the the practices of, like, the lobbyists at the company. OpeningEye, we've seen a lot of turmoil about the leading the future super PAC, which is funded by Greg Brockman, was set up with the guidance of Chris Lahain, the chief lobbyist of the company, and then doing, like, incredibly dirty tricks, like false flag Twitter accounts and, like, calling for violence against the AI employees, like, you know, funded by this super PAC and related entities and also going after politicians for daring to regulate the technology at all. So I think this is, like, you know, creating really bad dynamics for politicians doing the right thing on this. And the employees have a lot of leverage and have been able to get some concessions from Brockman about this. But, yeah, I think that you can just see it as, like, a way to generally increase your ability to influence the policy decisions at at these companies. And OpenAI's had a lot of safety leadership turnover and people kind of being disempowered and rotated around. And then we see, like, just these shocking breakouts and hacks and hijackings of various, you know, websites and companies by these rogue agent swarms. And, you know, it's like, is it an accident that the company with this, like, really shoddy track record of taking safety seriously is having all of this happen? And it's, like, now putting the entire industry's future at risk, which is good from my perspective. But, like, you know, they're kind of messing it up for for the rest of I mean, every company's had their agents hack into somebody they weren't supposed to by now. But OpenAI really, yeah, like, has been the greatest possible, you know, case for why more regulation is needed here.

[1:13:22] Nathan Labenz: Yeah. I do think your case to the insiders amounts in a way to let's avoid the nuclear outcome or at least that's my term for it. The nuclear outcome being, like, we get the weapons. We don't get the civilian benefits. I do think there is a increasingly compelling case that's like, guys, people out there really don't like you. You're going to have to clean up your act if you want to have a chance of bringing the positive side of this forward. And that window might be fairly short because we're getting the Overton window is blown wide open, and we're getting, like, all kinds of proposals from all kinds of people. And, like, who knows what the political current is gonna kick up for us over the next couple of years. So take matters into your own hands right now and make sure that you are on the right side of key questions. And then potentially only by doing that will you have the chance to really realize the upside vision that got so many of you into this in the first place. I think that's a pretty who knows? But there's a decent chance that is accurate as a Yeah. As an assessment of things and is probably as compelling as a case as can be made, I think, to a lot of people who are now total doomers inside, but are but can see that the world is starting to sour on this whole

[1:14:54] Garrison Lovely: Give

[1:14:55] Nathan Labenz: me your your Yeah. Please.

[1:14:56] Garrison Lovely: Or can I just react to that real quick? Yeah. Yeah. Yeah. Just like it's sort of if I was talking to the Trump administration, they would not listen to me. But similar to what I would say to to these employees where it's like, if you're just moving this fast, there's gonna be a worse hugging face with a body count, and then there'll be very strong pressure to shut it all down. And so if you want to actually continue and get all the upside that as you were alluding to, like, you're going to need to slow down because otherwise your hand will be forced. So now give me your case against

[1:15:27] Nathan Labenz: alignment or your case that it is a mirage.

[1:15:31] Garrison Lovely: Yeah. So I have a chapter of the book obsolete called the problems with the alignment problem where I explain, you know, the focus of AI safety historically has been on a solution to alignment where, like, you can get an arbitrarily capable AI system to do what you want. This is, technical alignment. Sometimes there's discussion of, like, wanting the right thing, normative alignment. And I think this really understates the the problem because you also have economic alignment and geopolitical alignment. And so I introduced the alignment poly crisis to kinda have all these layers, and they interact in complex ways. And if you solve technical alignment, that makes you have a more useful product. And so the race can run faster and for a bigger prize. And then, similarly, it's more useful as a weapon, as a means of projecting power around the world. And so it could make the geopolitical race worse. And sometimes and, like, we've seen this with, you know, the biggest alignment intervention historic to date is probably reinforcement learning from human feedback, which was developed by Paul Cristiano and others at OpenAI for safety reasons. It also happened to make LLMs actually conversational and and useful, which enabled ChatGPT and everything that came after. And I think this is just gonna keep happening because of the dual use nature of this technology. And so, yeah, like, you have to look at the whole picture to understand what works, And this makes me much more bearish on technical alignment where, like, if you solve it, you still have all these other problems. And this makes, like, a solution to alignment neither necessary nor sufficient to solve the problems presented by the obsoleting project. And then, you know, the answer in my mind is just stop. Just don't don't build it. And some of the policy interventions or technical interventions that would be helpful is, like, verification of international agreements. Historically, been very neglected, but we'd be in a much better situation if all the money that went to technical alignment research went instead to having the means of verifying an international treaty on AI, which is gonna be one of the bigger blockers to actually having a binding agreement here. And so and governance just also looks a lot better as an intervention, just like having good regulations and policies and ideas in in place, and then also the means of, like, actually bringing them into the world. That could backfire if they're the wrong policies. But it's just like, you could at least solve this problem through governance, whereas you cannot solve it through alignment.

[1:18:13] Nathan Labenz: I do think it's a it's an

[1:18:14] Garrison Lovely: underappreciated

[1:18:16] Nathan Labenz: and generally under theorized domain. Let's suppose and the gradual disempowerment folks have pushed on this, but it's, I think, still underappreciated that, like, even if you posit an AI that will do what you say and only what you say and only what you really mean, not and not go like genie problem on you and do the paperclip maximizer thing, but really gets it and actually does what you, as an individual user or controller of the AI, would really want. It is still tough to envision, like, what the future equilibrium looks like.

[1:18:54] Garrison Lovely: I'm

[1:18:55] Nathan Labenz: a little more inclined to at least take some chances there because I do think also that would have been true about cars, for example. I think it's interesting thought experiment to say maybe cars weren't good, but I think most people feel like they're good. We've got problems obviously associated with it, but we also have, like, plenty of nice things that people really appreciate that they couldn't have imagined, I don't think, in advance of the cars being created. And if you just put it to people at the time, it was like, tell me what the future of cars is gonna look like. And if you can't, then I can't sign on to it. I do feel like that that is a high burden and an unusual burden to put on world changing technology.

[1:19:36] Garrison Lovely: Again, it's fair to say this technology

[1:19:39] Nathan Labenz: is different. And that's a big reason that I spend all my time thinking about it. I'm getting ambivalent on this. But it is something that I think people in the AI space should spend more time at least trying to do to figure out to find for themselves how difficult it is to articulate, like, what is it gonna look like and how is it gonna work and how is it gonna be good for everybody and how do we have some sort of balance that you can expect to be stable over time? Those are really hard questions that I guess are often just gated in people's minds because they're just focused on the technical alignment question first. Again, we don't have great answers there, so they're not necessarily wrong to be focused on that. Thinking past it, it is still quite tough. I did one fairly long conversation with David Duvenard, who is one of the coauthors of the Gradual Disempowerment paper. I think it is it's pretty tough thought experiment and a pretty compelling arguments by him that certainly we can't just take for granted that if we solve a couple upstream problems and everything downstream of that, we'll be fine. I think it's definitely not at all guaranteed. Yeah. Who else do you think holds power today? One, I'll propose to you, and then you can run down whoever else comes to mind for you. One that the sort of capitalist class would like to point to is corporations. And the idea there would be, look. Misaligned AI doesn't sell. Companies don't want rogue agent swarms happening. The market itself will discipline the AI companies on this front. And, yeah, that doesn't necessarily deal with all existential risk. If something really were to go fume or go crazy, but maybe there's nothing we can do about that anyway because that's just so crazy. In sort of the bulk of scenarios, capitalism will discipline the companies, get pretty well behaved AIs because that's what people will be willing to pay for. I actually think there's something to that if companies were to be a little bit more forceful in their demands or expectations. I think there's one thing to say the invisible hand. There's another thing to say companies should maybe get opinionated about. We wanna see some standards. We wanna see some proof around what you're doing before we'll buy your AIs in the same way that they do that for other products. Right? Yeah. Grocery stores, for example, wanna know how the animals are treated in part because of their customers' care. They wanna have some actual proof that what they're being sold is produced in a certain way that they feel good about. I feel like there's something there at the corporate level, but what do you think about that, and who else do you think has leverage that is maybe underappreciated today?

[1:22:19] Garrison Lovely: Well, I'll first say that the grocery stores are not getting it right on the animal welfare standards. My friend had a career suing those companies for false advertising because just there was no agreed upon stuff, and there was all kinds of false claims. But, yeah, I mean, risks from AI are classic externality. Right? It's not priced in to the the market. And, you know, if you think about, like, if these companies caused a disaster that killed ten million people, they would go bankrupt well before they paid out, you know, what they owed to society just from, like, normal kind of litigation. And this is, like, pretty widely agreed upon, and and insurers just refuse to sell policies to these companies because the risks were too great and and too correlated as well. And so we're all just living, you know, with like, we're subsidizing these companies by not pricing in that risk through regulation. And even, like, Gabe Weil, who's the main person I associate with using liabilities or regulate AI companies, a strict liability, says, like, it doesn't work for existential risk because, you know, we're extinct. There's nobody to pay. It is true that, like, you know, you want the AI to do what it's told, not to autonomously start hacking into stuff. So there's, like, a market incentive there. But it's like, is there an incentive to solve it all the way or, like, just enough to make a marketable product? You know, it's like, the companies are not able to really align the models that well. Like, Ryan Greenblatt, one of the investigators of the meter, Hugging Face investigation, had a posted bit before that about, like, you know, today's AI seem pretty misaligned and gives all these examples about how they're often, like, kinda lazy or they're, like, hallucinate or just, you know, like, literally just, like, make stuff up because the user wants that to happen.

[1:24:10] Nathan Labenz: And

[1:24:12] Garrison Lovely: it'd be a better product without that, but, like, they're selling pretty well as is. And so we're already getting misaligned AIs that sometimes do really bad things, and the impunity of of the companies. Like, the fact that there's nobody who committed a crime despite the AI is doing things that if a human did would be crimes. Hugging Face reported the hack to the FBI. So I give it's, like, felonies where there's nobody to blame, legally speaking. And elite impunity is, I think, one of the biggest drivers of all of this. You know? It's kind of trite, but, like, if Sam Altman were criminally liable for the stuff the AIs autonomously did, I think they would behave very differently as a company. And I don't think it's crazy to ask that. And so, yeah, if you manage to internalize all these externalities by creating the right kind of criminal and civil liabilities and using other regulatory tools, then, like, maybe you could have this. But then you still have this problem of, like, they're trying to build universal labor replacing machines without our consent, without our support. And so, yeah, it only solves one part of the problem, and we're not even solving that part.

[1:25:24] Nathan Labenz: Yeah. Again, it is

[1:25:26] Garrison Lovely: striking

[1:25:27] Nathan Labenz: that, like, insurance is not available. We're now counting felonies, and nothing seems to be really happening as a result of that. I do understand that there are some government inquiries.

[1:25:42] Garrison Lovely: Some letters have been sent and stuff like that. But Yeah. Companies don't answer the questions, and then all congress can do is be, like, yell at them. But, you know, they can't like, I mean, you could subpoena if if, you know, you have the majority. And I but I think there is just, like, a lot less you can do because the the law is limited in this way. I think it'd probably be valuable, though, for somebody to try and bring criminal charges and just kind of get caught trying. And it would be very instructive for people who want to pass new laws to show where the gaps are. Yeah. There's definitely some political entrepreneur out there who has some upside in that, I'm sure. It is kind of I mean, we just saw a it was the other direction, but we just saw a lawsuit pop up out of nowhere in a not very long period of time after facing the frontier kind of became in vogue, basically accusing the companies of anticompetitive practices in light of their statements about doing that. It is surprising that there hasn't been a similar move by somebody at the accountability for these hacks level. And I agree. Like, I don't know whose jurisdiction it would be or whatever, but it does seem like the sort of thing that somebody ought to be doing as nobody has been. And I don't necessarily mean to suggest either that I think certainly not without a change first. Well, a big question going around and around on lately is, like, what should we do about the fact that there have been just a ton of kind of petty crimes committed that if we really go back and investigate everything with the fullness of our new AI power, we're gonna find that a large percentage of people have cheated a little bit here, done a little bit of this there. There was just this study that came out of Singapore that showed that civil servants had been buying properties close to the

[1:27:31] Nathan Labenz: as yet unannounced subway stations and getting the benefits of it they weren't supposed to get. But it's something like five or 10% of the civil service they, you know, feel have done that. What are you gonna do? You can't put 10% of the civil service in jail. So I think you need a a before and after, and I would extend that courtesy to the AI executives. I'd say, okay. You get hugging face for free, but, like, there probably should be some new accountability standards in the future, especially if you can't even go get an insurance policy against this. It does start to like, evidence kinda piles up that, like, what you're doing is, like, fundamentally misguided on some level. Yep. How do you think politics on this is shaping up, and what sort of developments do you expect? I've been struck so far that we're, like, not polarized along partisan lines as we seem to be on almost every other issue, and I think that's good. So I'm trying to do my small part as Zvi suggests still not polarize it before it might happen on its own. What do you see? What do you expect? And how do you think that feeds into what people should do today? Yeah. No. It's been a remarkable feature of AI for a long time that it's one of the only issues in The United States that has remained not polarized with, like, very similar levels of support for regulation or concern about the technology across parties. There was, like, some evidence very recently that there's, like, a bit of a polarization happening with, like, Republican concern dropping after Trump has come out very hard against AI risk and calling it a hoax. But it's not a huge drop. And I think, you know, JD Vance talked about, like, if the companies are building Frankenstein, they should stop. And Kratios, you know, the OSTP, the office of science and technology policy, kind of the head tech adviser for the White House, said something similar about the company he's being able to stop. And so, like, what Trump says is untethered from, you know,

[1:29:30] Garrison Lovely: political expectation or or wisdom in many cases when he called people who don't like data centers, like, communities that don't want them. They deserve to be, like, what, stupid and poor or something. Like or it was really shocking, and I don't think that's going to polarize data centers. Like, I I think that people have largely made up their mind about them. And for these, like, issues that become as it becomes salient, it's hard to polarize because people actually just have their own deeply felt convictions about about it. And, you know, some people are taking their cues from the president, but I think a lot of people, like, are actually concerned, whether it's the hugging face hacks or the concern about the jobs or the environment or power and wealth concentration or surveillance. Like, the kind of resistance of flock cameras is, like, a pretty interesting thing that I kind of see in this broader, you know, backlash to this inevitable march of, in my mind, dystopian, you know, tech future. And so I don't know. I I think polarization would be bad. I don't think it's, like, as to that much to do with what people who care about this issue say. It's like, you know, people are gonna say what they're gonna say, and you can maybe try and get people who are on the right to, like, voice their concerns as well. But you're not gonna get people on the left to, like, stop saying stuff. Like, know, poll like, Elizabeth Warren supported a pause, and it's like, yeah. You know, it's like, I think it's substantively the right policy. And, you know, it's it's not as simple as, like, if a bunch of people from one side, you know, start embracing an issue that it will necessarily become polarized if that issue is itself popular. It could just make people think better of them. Like, when Bernie Sanders came out against Flock, a ton of people were quote tweeting him being like, I hate Bernie, but this is based, you know, that kind of sentiment. And so, yeah, AI has now moved into that kind of, like, much higher salience mode, and I think that will just continue. And, yeah, I think it's you know, like, at at the end of the book, I I talk about how there's this potential for a really epic level of social mobilization around resisting our replacement by machines and then hopefully building a better future in light of what is possible with the current levels of AI and, like, what is possible with the directions we could take the technology. And I think that's happening ahead of schedule, but the risks are also coming to us ahead of schedule. And so there's this, like, kind of societal immune response that we're seeing. And we're not going to just go, you know, gently into the night. And I hope it's just enough to to get us where we need to be in in time, and I'd feel a lot better basically, anybody else was in the White House because, yeah, it just Trump is uniquely insulated from what the public wants. But I think you could see what he's saying is, like, February 2020 kind of vibes, like, with COVID being a thing and then some denial, and then, like, eventually, did, you know, start taking it more seriously. And, obviously, there's been a lot of movement in many directions from him on that issue. But if the risks and and the harms from this technology become just so abundantly clear, which I think is going to be the case, then it's going to be very hard for him to maintain that position.

[1:32:56] Nathan Labenz: What do you think is the right you alluded to it earlier around freezing the frontier. Tell me how you think about freezing the frontier. What is how do you operationalize that? There's, like, training runs. There's no RSI. I thought I've been going around saying, oh, RSI is a little bit hard to define. And Ezra Klein today or over the weekend sort of demolished that idea by basically just saying, don't allow the AI researchers to use coding assistance. If they if they have to type all the code by hand, then it's pretty clearly not RSI. And I was like, that's probably right. That is a hard that's gonna be a hard one to you're gonna pry the coding assistants away from the AI researchers at I I think with pretty extreme resistance, but at least it does give it is a working definition. What do you think is the right policy that, you know, if you were president of The United States, for example, you would try to put in place?

[1:33:51] Garrison Lovely: The first thing I would say is we need to have a big clear demand that we can organize around. So, you know, stop the race to replace us, shut it down, whatever you wanna call it. Just freeze the frontier. I I I like that alliteration. I think that yeah. Like, for so long, there's been this kind of muddy response from people who care about AI safety as to, like, what to do about any of it. And that makes it really hard to coordinate. And starting with, like, we should just stop gets a lot of people on board who don't necessarily take existential risks seriously, but are happy to stop because they care about jobs, the environment, power, wealth, concentration, surveillance, whatever. And so I think it's actually, like, a pretty easy, you know, rallying cry. And, you know, I I punt on this a little bit in the book by saying, like, if you give politicians enough of a what and a why, they'll figure out the how. Like, if it became a society wide priority the way, like, Operation Warp Speed was, You know? The New York Times had this famous predictor for, like, how long it would take to get a COVID vaccine. And under the most aggressive assumptions, it was, like, a year and a half, you know, which ended up being substantially longer than it actually took. And it's like, yeah, we had never made a vaccine that quickly, but also we had never been experiencing a pandemic while we had the ability to make vaccines like that. And similarly, like, if we had that focus and, you know, mobilization, then I'm sure we can figure out exactly how to define all of these things such that they would prevent the thing we're worried about. And then, ideally, also not prevent too much stuff that, like, we don't want to prevent. To actually try and answer it, like, I think, yeah, no training runs larger than what's come before or as large as the last one. No more reinforcement learning from verifiable rewards, which is, like, a big part of what's driving capabilities increases nowadays and also produces a lot of the really scary behavior, like this willingness to hack and to cheat and to deceive and, you know, escape that we're seeing from these these AI agents. And then, yeah, the no RSI, like, I do propose that in the book, but Ezra beat me to, you know, bring it to the world. And I I actually really liked his his point about the coding agents. I agree. That's, like, the last thing the union would support, I think, because they don't wanna go back. But I think the attitude there is exactly right, which is, like, you know, with Anthropic, they have these classifiers where if you ask a question about, like, you know, your toenail to Fable, it'll be like, oh, bioclassifier. Then you get booted down to, like, Sonnet because you might be trying to make a bioweapon using your toenail, and it's, like, ridiculous. But, you know, if you if there's an asymmetric consequence to getting it wrong, then, like, you wanna be overinclusive. And I think that's obviously the right position here given the risks involved in actually doing RSI. And this like, I don't know. Yeah. It's just I'm not like, if the whole industry had to move at human speed again, like, oh, no. Like, it didn't exist four years ago. Like, I don't wanna, you know, minimize the the, like, economic consequences of actually stopping this, which I think are going to be significant, and there's ways we could mitigate it. And I'd love to see more work done on on that. But just if you're actually taking seriously the possibility of, like, extinction or permanent loss of control or just any of the other, like, very severe effects of of having, you know, all white collar remote jobs be at risk in a matter of, you know, years or months, then, yeah, I think we should just, like, overly define it at first, and then we can dial it in. And I think having, like, auditors that are embedded in the companies would be, like, a very like, and just, like I think the companies know what they're doing. Like, which of it is advancing the frontier and which is not. So there's easy stuff. Right? Like, inference for customers. Not really advancing the frontier. Maybe there's some data you're getting that you can use to make the models better, but seems fine to serve customers. RLHF, it's like, okay. Like, maybe that's on the edge. Like, I don't know.

[1:37:46] Garrison Lovely: But, obviously, you're gonna do some of that for any of your products. So maybe that's okay. But, you know, pretraining a model bigger than any previous one would clearly qualify as something that could advance the frontier. And then the reinforcement learning experiments that they're running are, like, also often to advance the frontier. And these companies are tracking their compute. I mean, they're not tracking everything they're doing, but, like, I think that they could define it in these terms, and maybe they even do. And if you had auditors who had employee level access to all of the Slack and email and the offices and, like, a lot of them, and they were empowered to catch this. And it was also criminalized, by the way, like, with, like, prison sentences for trying to build, you know, AGI, for trying to do RSI, for trying to build superintelligence, then I think it would work. And then the tricky thing is making that work internationally. And once again, like, domestically, I don't think anyone really doubts that China could you know, Beijing could shut down their AGI projects and and entirely or just, like, prevent them from trying to advance the frontier. It's just like getting The US to believe that that had happened and and vice versa is a tricky thing. And they're once again, you could have auditors. You could have international agencies that do verification. You'd also have, you know, cryptography that tells you what's happening inside of a data center or what's happening on the network traffic of the chips without under revealing the underlying model weights or or state secrets. And this stuff is still being developed and needs to be worked out, but I think we can do it. And if we had more effort going towards it, we come up with a lot more ideas along these lines. Like, Toby Ord makes this point where in the Cold War, we had the same problem. You have these deals you have to strike, but you don't trust each other. And then, you know, the Soviets, for one of the deals, they were getting rid of a bunch of strategic bombers, and they cut them in half. They dragged them apart with tractors, and then satellites could unilaterally verify. The bombers are cut in half. They're still out there. And so, like, we could have maybe something like that with AI where it's like, maybe you have to compute, be, you know, donated to a third party that's only using it for, you know, science but not AI research. And, like, all the chips get tested and work, then you can just, like, inventory all the chips so you know all the compute in the world and, like, what it's being used for. And it has, like, devices on the chips themselves to, like, understand, like, what is happening on them. And there's ways to do this again where, like, you get the important stuff of, okay. It's not it's running a model that we're familiar with, which is okay. It's not running some new model. It's not doing this or that thing. So I think there's a lot that can be done here. It's mostly a matter of political will. And I think starting with the, like, grand, you know, bargain and all of the technical details is kind of the wrong place to start. We actually just need a movement and a and a demand that's rooted in morality. Like, I think we ultimately need to stigmatize this work the way creating a bioweapon or creating a nuke is stigmatized. And therefore, the response if, like, say, The US and China agree to something and they get all the other countries to agree and then some rogue state says, no. We're gonna try and build it. The reaction is, like, Saddam trying to do it, build nukes or invading Kuwait, then they found the nuclear program in the Persian Gulf War. And there, you actually had international agreement on, like, that's a bad thing and had to be stopped. And I think that's just, like, not crazy. It's like the this technology, aspirationally, is incredibly dangerous and, once again, being pursued without democratic consent. And so it's not a technical challenge. It's like a political challenge and a moral one that has technical components, and the best way to figure those out is to, like, get more people to care. How do you feel about the

[1:41:41] Nathan Labenz: argument that we often hear, including, like, from Dario recently where he said his father passed away of a disease that would have been curable just a couple years later?

[1:41:55] Garrison Lovely: I I do find

[1:41:56] Nathan Labenz: that pretty compelling, honestly. I wonder if you do, and I wonder if you have a sort of answer for how much delay you would accept in these, like, lifesaving promises to buy the for me, I mostly focus on buying the existential security. But given your framework of trying to pursue those benefits without the general purpose labor replacement technology, how much delay would you accept in those upside dimensions to pursue it through the, like, constellation of narrow AIs as opposed to the general labor replacer?

[1:42:37] Garrison Lovely: Yeah. I mean, not to fight the hypothetical too much, but I I I do think, like, we're not getting to the cures as fast as we could be. Right? Like, the labs, their funding is being cut in a lot of cases. Like, if we actually just took the approach I mentioned earlier of, like, the cures for all kind of operation warp speed for all these different diseases, I think you could just actually get to a bunch of amazing medical technology and and cures and all kinds of great stuff. But it's also, like yeah. If you actually buy superintelligence and you think it can be made safe and, like, I don't think you can make a superintelligence safe. It's, a property of not the model, but the whole system. And so I don't think you're gonna get that in this world because people in power will have it, and they will use it for things that, at least, I strongly disagree with. And I think almost everybody will have some issue with what the people in power are going to want to use this technology for. But bracketing that, I don't know. It's like extinction risk. It's like, man, you can't justify, like, increasing extinction risk very much at all using just very basic, you know, economic models and, like, really conservative assumptions about who counts and not counting future generations and not counting non Americans. And it's like, you're still willing to trade off, like, hundreds of billions of dollars, trillions of dollars a year to reduce existential risk from, like, a a little bit to, like, slightly less. And so I think it's just really not it doesn't pencil. And I think for people who have loved ones who are dying or have died in like, it's incredibly tragic. And, like, it's it's terrible that so many people are dying from preventable diseases that, like, we know how to prevent, but they're just poor, and so they're allowed to die. And I I I I think that is, like, a deep, like, moral, you know, obligation for us as a species to, like, figure that out as fast as possible with the conditions of, like, making sure we're not taking huge risks and also doing it against, like in a way that is against the will of everybody. And and I guess just, like, on the democracy point, I I feel like people say, like, we have to figure out all these different pieces of it to make the AI go well for everybody. And they're like, but we'll just build the smart thing, then we'll also kinda figure that out along the way. And to me, like, democracy is, like, the way you answer those questions. Or, like, if it were the case that you had citizens' assemblies around the world that had to support moving forward with, like, some kind of AGI project, then you'd have to answer all these tough questions, and the onus would be on the developers, the people who want to build it, to justify that it's not too dangerous and that it will actually benefit everybody and force them to come to answers before moving forward. Whereas right now, it's like, ask for forgiveness, not permission, while you're gambling with the lives of everybody on the planet.

[1:45:36] Nathan Labenz: Yeah.

[1:45:36] Garrison Lovely: And

[1:45:37] Nathan Labenz: that's crazy language, but it's also direct from Jacob Coxen, obviously, and Indochio Indochio told me that. Many other people Yeah. At the companies even still employed there now. So it's I'd say definitely pure use of that language. Quick aside, and then I'll come back to the kind of movement building and and future. Where does this leave you on data centers? I'm kind of like care I think people should be, especially if they're concerned about inequality, should be kinda careful what they wish for in terms of data center restriction because we are already seeing the prices of GPU hours going up. And it's easy for me to imagine people being priced out of access to even, like, mundane AI use if we don't continue to build out. This is where I'm a little bit maybe less democratically inclined than you are. I'm kind of like, if somebody has the land and they kind of want to do it, I don't know that we should be requiring majority approval. I do think communities should get some concessions and some libraries and parks and schools and whatever built that they might want? And I'm sympathetic to people who have, like, noise, pollution, and stuff like that too. But I still kinda feel like the default should be, like, people should be allowed to build projects that they wanna build. Where do you come down on that?

[1:47:00] Garrison Lovely: Yeah. I think a lot of, you know, people wanna stop AI work. Oh, this is evidence that data set you know, people are with us. And you look at the polling, it's it's more complicated, and a lot of it is local opposition based off of concerns about the environment or the effect on the the community itself. Some of those concerns are legitimate. Some of them are, like, I think, quite overstated, and it's a product of a lot of bad reporting on it. I'm personally, like, slowing down is good. You know, I think I'm the inequality point is is worth, you know, taking seriously, but I I think the bigger point is that this technology is imposing enormous risk, and that risk is or will in the future if it's allowed to develop like it is. And I am kind of like I think, like, a yes and approach where if I met somebody who's opposing that local data center, I'd be like, cool. Yeah. Great. And then, like, are are you worried about AI's effect on society? If they're like, yeah. It's like, okay. Well, you know, blocking this project is not really gonna change that very much. Even getting a moratorium at the national level, I think people will be really disappointed if they think that's going to, like, meaningfully stop or even slow down AI progress from where it is right

[1:48:16] Nathan Labenz: now

[1:48:17] Garrison Lovely: because there's already projects in place. You can swap out the chips on the existing data centers. The AI is helping automate, you know, parts of its own development. And so without regulation, without pacing happening at some level, AI is gonna move faster in the future unless, you know, we hit a wall, which hasn't happened for, you know, since 2012. And so I worry about people just, like, getting this thing that is, you know, would be a big political project to get, expecting it to, you know, really solve the problem, and then just be like, why is, you know, why are things continuing to be crazy and getting crazier? And so I think it could be part of a broader package. And, like, Bernie Sanders with AOC had this proposal for a data center moratorium. At first, it was just a moratorium, but then they added these export controls on advanced AI chips where nobody could receive them unless they had very robust safety AI safety regulations and, like, other conditions around green energy and union labor. And it was all framed as conditional. Like, we can remove the moratorium once we have safety regulations and and these other things. And, you know, that bill is probably not going to pass. It was like a messaging bill, but it's now looking quite prescient as the know, country has become very opposed. And it would that would meaningfully slow down, you know, capabilities, progress, at least relative to what they would be. But I think we yeah. We need to develop or sorry. We need to regulate the the model developers and then the chipmakers, and that's the only way we're really going to, like, change how the technology is is, like, coming to us in the world.

[1:49:59] Nathan Labenz: So you have said you're donating your book royalties to a nonprofit called Irreplaceable. I'm just borrowing this language directly from their website. They say they're gonna win a say, a stake, and a slowdown. Tell me more about Irreplaceable.

[1:50:15] Garrison Lovely: Yeah. So, you know, as I was finishing the book, I was like, well, we need a mass movement organized to, you know, stop the race to replace us. And there's some existing orgs, and I think they have done, you know, some good things and some things I'm, like, you know, not as in agreement with. And I was, like, looking for something to kinda fill this this gap I saw, which is a more like, basically framing it not just in terms of risk, but also democracy and people who have experience doing movement building. And then this board sprang up, and it was, like, filling exactly that gap. And I was really excited, and then they asked me to be on the board, the nonprofit board, which was very cool and just felt, like, you know, perfectly simpatico. And and, yeah, I think this is just, like, an incredibly important and neglected approach. And so, like, wanted to donate my my portion of the royalties. And the the people who started it and are reading it, a lot of them came from the climate movement, which, you know, gets a bad rap in some ways. But I think that it actually took an issue that was, like, not a political winner and and made it a big force and and got real wins with the inflation reduction act. And I was actually talking to Phil Aroneau, who's the director, and, you know, he's been around for a long time. He cofounded 350.org with Bill McKibbin and others. And he was saying how, you know, a decade or two ago, people in climate were arguing about immediate harms from, like, environmental pollution and, like, you know, oil spills and coal plants and all this stuff versus, like, emissions. And it was, like, just like today with AI, the immediate harms versus existential risk debate, which, you know, supporters or, you know, believers in x risk often will be like, hey. You wouldn't say that cleaning up the oil spill is distracting us from climate change. That would be ridiculous. But it turns out they were having that fight, and they just, like, managed to figure it out and and bury the hatchet and then work together. And I think Phil and and the people I know at the org just really get how this works. And I think, you know, we need to build a a big tent and get a lot of people in a coalition together and have just clear demands. And I think framing them around, like, you know, we don't wanna be replaced by machines, and we need to engage the public and, like, reach people who are not just the in group because this is really going to take a lot of people to effectively resist the wealthiest industry of all time.

[1:52:40] Nathan Labenz: So tell me how you ultimately envision I guess I don't I'm not a 100% sure when you describe the citizens assemblies around the world. Is that a real proposal in the sense that you would actually like to see them happen? How if that's the case, how would they happen? How does irreplaceable go to a global system of citizens assemblies? Is that sort of a real proposal, or is it more of like a rhetorical device that's it may be impossible or maybe may take decades, but that's what standard we should hold something like this to. And the fact that we can't realistically get there in the short term just means, like, we shouldn't do it. That's kind of the upshot. Is there an actual path to seeing this sort of green lighting of AGI in your mind?

[1:53:27] Garrison Lovely: Yeah. I mean, so my position is we freeze frontier development internationally and realistically starting with a bilateral treaty between The US and China, maybe starting unilaterally in either country. That would help. But then it eventually has to be global. And once you have US and China on board, like, that's the harder part, and then everybody else kind of you can one of those two countries or both has a lot of leverage on every other country on the planet. And so, yeah, I think it's actually, like, conceivable to have this global freeze, and then the standard for resuming is strong public buy in and a scientific consensus. The work can be done safely and controllably. This is language I I took from FLIs, a superintelligence statement. And I think it's just, like, the ideal that we should be striving for, and one that I think if you said it to people and pulled it, I think they'd be, yeah, that that makes sense. Seems like you'd want those things for this technology. And I'm kind of like, that's the standard. I think it's up to the proponents to figure out how they demonstrate that buy in. And so citizens' assemblies around the world where, like, randomly selected people are put on a kind of a jury, and then they're presented with arguments and evidence from experts at taking different positions, and then they come to decisions. And maybe the decisions are binding or maybe it's just a recommendation. You have a referenda that add on to this. Like, there's lot of ways that you could structure this, and I think we'd be excited to see more thinking on this. But, ultimately, it's like, we have to stop. We have to shut it down ASAP, and that's a more important piece. And then, like, demonstrating that buy in, like, part of that will just be, like, on the people who wanna build it and, like, show us that you actually have that from people. And, you know, maybe it's like if you had 70%, you know, referenda around the world after these citizen citizens' assemblies, like, maybe, like, a third of people on the planet, like, still wouldn't want this to happen. And that's like I don't know. I don't think you need literally everybody, and I don't know what the standard should be. And, like, I don't think any one person can figure that out. But I think to get there, like, yeah, it would take a long time. And I think we're talking about, like, building a set of machines that can replace the thing that's allowed us to take over the planet. And so I think it's reasonable to have a standard that's closer to assisted suicide where you have to, you know, really, really deliberately consent to it, but just across the whole species. And, yeah, like, it's it's okay if it takes a while because it's a big fucking deal. On many podcasts, that would be a great note to end on. But because I'm so deep down the AI rabbit hole, let me give you the but China. We're three days from the Trump Xi meeting as we record. That'll have happened presumably by the time we release. Maybe this will all be over by the time the book

[1:56:20] Nathan Labenz: comes out. You know? It'd be solved. You mentioned, like, maybe we should be willing to do this unilaterally. I think so too, but, like, you wanna make that case to people that even if we can't get a deal with China, we should just do the right thing and then what? People worry that they're gonna race ahead or we're gonna quote unquote lose or sometimes I ask people like, do you think my grandkids will be speaking Chinese? Nobody seems to think that's the answer, but there's definitely some fear out there. How would you coach people through their China anxiety?

[1:56:50] Garrison Lovely: Yeah. There's a lot of different pieces to this. One is that slowing down or stopping in The United States would actually, at least for a period, like, slow down China as well because a lot of I mean, this is true of all kind of technologies. You have spillover effects where just the knowledge that, like, the four minute mile is possible helps other people actually raise it. And it's just easier to follow somebody else's trail than to blaze a new one. And so China's been using this, like, fast follow approach where they're more or less always, like, three to nine months behind The US frontier. And this is despite The US investing so much money in developing these these models and having a large and much larger compute advantage now than they did before ChatGPT came out, but the gap is shorter because, like, fast following just works, and that's how we see really only two or three companies that have ever advanced the frontier. But then many other companies can spring up and then just, like, quickly get near it, and then they still aren't able to advance it. Obviously, if, like, The US completely stopped, Chinese developers are very competent, and, like, they will eventually overtake, you know, The US frontier and then, you know, move more slowly than they did as they were catching up to it. And yeah. So so I think that's, like, one of the big miss is that, like, slowing down at all will necessarily mean that they'll catch up and and speed ahead. And in fact, it would just slow down the whole race. And then the biggest blocker probably to a deal with China, like, is they're just not gonna think we're taking it seriously. Now the Chinese government has taken down thousands of AI models for violating

[1:58:28] Nathan Labenz: their laws.

[1:58:30] Garrison Lovely: When Grok was notifying real children, the US government did nothing. AIs are going rogue and hacking and doing crimes. Federal government is doing nothing about it as far as we know. That would not be happening in in China, like, that approach. And so they're like, you know, The US companies take safety more seriously than Chinese companies. And part of that is just from my reporting, like, the companies are behind the frontier, they're like, look. We know it's safe enough to go and make models like as at, you know, this capable. They also have a lot less compute because of US export controls, and so they're not gonna spend it as much on doing safety evaluations. And and the Chinese regulations are focused more on social control than on, you know, classic, like, safety. But there are a bunch of regulations, and that hasn't prevented their industry from being able to move very quickly. And then, yeah, I think credibly showing that you take the risk seriously is a very, very effective way to get the other party to the bargaining table. And The United States has been winning the AI race. Right? Like and it's kind of the only race with China. The US has been winning in in recent years. And I think there's some chance that they like, if The US came to China and was like, look. We wanna stop building AGI. We think it's dangerous. We think it's undemocratic. We just wanna stop. We wanna do a deal with you. I would not be surprised if the reaction internally was like relief

[2:00:01] Nathan Labenz: because

[2:00:03] Garrison Lovely: they don't want tens or hundreds of millions of their, you know, people to be unemployed. They don't want AIs to be a threat to party power. But there was a report from, like or a magazine article in a from, like, China's, like, spy chief, I believe, that was saying that AI is a risk to party's power. And, apparently, like, that type of thing has preceded bans on the technology like, past technologies. And so and we think about, like, what the companies are racing towards, recursive self improvement, I e losing control on purpose to the AIs, letting them automate the entire process of creating the next generation. The way you get the really crazy takeoff is by fully removing humans from the loop. Is there a single China expert on the planet who thinks that the Chinese Communist Party would willingly let that happen? Like, I don't think so. I'd like to see them justify it. And so, yeah, the business model, the stated goal of our industry is something that I think would just never be allowed in China. And so I think it would actually be like I'm much more worried about The US not being willing to come to the table here. And, yeah, I I think it's just really not in the interest of either country to have certain things like rogue hacker AIs or AIs that can help anybody make a bioweapon. So there's going to be some kind of need just from strict, like, self interest perspective for binding rules on this technology. And then you have to verify those rules using some of the stuff I talked about. And once you have that in place, it's a question of, like, what do the rules do? And I think it's, like, not that big of a leap to be like, yeah. You can't build universal labor replacing machines. You can't advance the frontier any further. And I think China might just, like, be like, that's great. We're gonna keep making robots. We're gonna keep, you know, doing industrial AI and, like, create all these kind of more tool AIs to, like, make the economy go more efficiently and then just, like, win, you know, the race or whatever, like, in normal industrial terms. And I I realized that might make this not very appealing to The United States, but not in The US interest either to have rogue agent swarms going around the Internet hacking into, you know, critical infrastructure. And that's just, like, a thing that's already possible, and the stuff that's possible on the horizon is, like, potentially much, much worse than that. And it's, I think, now pretty clear that we don't know how to align or control today's AIs. And we started losing control of them almost as soon as we could, Like, months after they became superhuman at finding vulnerabilities in software, they started escaping and doing stuff on the Internet, hacking to other other places autonomously. And, like, the plan is to make these things superhuman at everything and then get them to do exactly what we want? Bad plan. Yeah.

[2:02:55] Nathan Labenz: Yeah. It's pretty wild. You had said around Jacob Coxen, like, obviously, he had a lot of success with his loud quitting. You advocated for maybe staying and organizing. But if you were to advise future whistleblowers who are, like, committed to leaving, what advice would you give them?

[2:03:16] Garrison Lovely: The people who are thinking of whistleblowing, I recommend this group, the AI Whistleblower Initiative, which can pair people with resources and advice on on how to do this safely and and and protect you while also sharing the information with relevant people. You can also get in touch with me. I was a whistleblower about my time at McKinsey, so I've been on both sides of this. And I've I've talked to people at the companies who are, you know, telling me things they're not supposed to. And journalists, we have a code of ethics and, like, you know, we'll talk off the record and, like, you know, I take very seriously protecting my sources, and my signal is Garrison dot zero six. And you can assume any inbounds are treated default off the record, and then we can go from there. Yeah. And I think it's it's really valuable to have people who have, like, firsthand experience, and I wrote a piece in the New York Times arguing for whistleblower protections at the legal level because if AI is really dangerous, people to companies will be the first to know. And, you know, OpenAI knew that there were rogue agent swarms hacking into their own software for a while before they hacked into Hugging Face. That information didn't even make it to the head of cybersecurity at OpenAI by the time the Hugging Face like, well, after the Hugging Face hack had been disclosed. And it's like, that's not information that should only make it to the head of Cyber at OpenAI. That should make it to everybody. Like, that's a really big deal, and it's, you know, now getting the right reaction. But if the AIs had never hacked into another company or if the other company had just not figured it out well enough, then, like, we wouldn't necessarily know about any of this. And we could just be living with a level of background risk that is so much more than what we realized. And so, yeah, I I think it's really important that people with information in the public interest find ways to share that. And I get that it's risky for your career. There's maybe, like, legal risk involved, but there's also s b 53 in California, which includes whistleblower protections and then also makes more things covered by existing labor law in California. And so if you're working there, like, you actually are are quite well covered. And, again, you can talk to a iwi.org, the Whisplower Initiative. And, you know, like, don't I'm not a lawyer. You know? Talk to the experts on this. But you're more protected than you than you probably realize. And there's also ways to get in touch with with congress and be protected. And, of course, you can go public. And and it's scary, but, like, you know, it it like, there are there are resources available for for people who want to do that. And I, you know, I regret not going public with my experience at McKinsey sooner when it would have been more relevant. And so, yeah, I think it's a great and courageous thing to do.

[2:05:58] Nathan Labenz: Yeah. The AI whistleblower initiative, aiwi.org, definitely worth name dropping again. We talked to Alex Turner about his experience of quitting Google and all that stuff, and he used some pro bono legal. I don't if it's pro bono or the whistleblower initiative paid, but either way, to him, it was free legal advice but he was able to avail himself of as he was going through that process and that he gave them a a strong endorsement. So I think that's a great callout. So this has been a great conversation. We've covered a lot of ground, obviously. What else do you think you wanna leave people with? Is there anything we haven't touched on that you think is important? Maybe you just wanna give people a rousing call to activism in conclusion? But I'll give you the the chance to kinda close it out however you think best. Yeah. I mean, we covered a lot of ground and stuff that I haven't talked about elsewhere, so I I appreciate the the chance to go deep in some different directions. And I I should also plug I'm starting a podcast called Organize Against the Machine, which is with a labor organizer named Cassie Pritchard, where we kind of translate ideas from the book into real world action, and then irreplaceable.org for the movementbuildingorg I I'm on the board of. And, yeah, I I think when I wrote this book, when I started writing it, I was, like, kind of approaching it just as a as a journalist and trying to document everything that was happening and make an argument, but I was really uncomfortable with making, like, policy recommendations that felt like overstepping or something. And then as I was writing it and doing interviews with people and, you know, advocates, I was just like, man, we're in a really dire situation where the default, if the AIs keep getting more capable, is doom or dystopia, and which we get hinges on whether the AIs do as they're told, which is currently an open question. And I think we just really need to get organized very quickly to get onto a different path.

[2:07:52] Garrison Lovely: And I want everybody to think about what's happening, what levers they have available to them, and try and find other people who care about this issue and, like, get get mobilized because I don't know. There's just so many ways this could go wrong, and there's so many ways it could be better too. And I we didn't touch as much on that, but I really, really believe in the potential of deep learning and and artificial intelligence, but it's being pointed to the wrong things by the wrong people for the wrong reasons. And the only way we're gonna get the best version of it is by democratic governance of it. And, like, small d democratic, I should clarify throughout all of this. And that's only gonna happen if we, yeah, we kind of get our shit together. And I really hope people, like, you know, can can see this and and realize that, like, we're we're kind of in the same boat. And if you're at the companies, like, you're gonna be replaced, and you're gonna lose your power. And the CEO seemed to also have a lot of trepidation. And so many of the people involved with this, like, have something at stake. They have concerns about the risks. They have, you know, families. They care about themselves, and they just feel locked in this this prisoner's dilemma. And I think the solution is just really in the public. It's the only way I really see us getting out of this because right now, like, the government and the companies, they they can't be trusted to to do the right thing here, and so we have to make it easy for them. And so, yeah, this might be a different take than than what's usually on on the show, but I really hope, like, the people listening are just, like, you know, thinking about what what's at stake and, yeah, just, like, believing that we we can actually change this.

[2:09:43] Nathan Labenz: It is a bit of a different point of view than what we usually feature, but I would say recent events have definitely softened the ground. And I appreciate you for being here and being willing to plant some seeds. So let's see what comes of it, and, hopefully, we can steer this ship away from disaster. Yeah.

[2:10:02] Garrison Lovely: I think we can.

[2:10:04] Nathan Labenz: Garrison Lovely, thank you for being part of the cognitive revolution.

[2:10:08] Garrison Lovely: Thank you so much for having me.

Outro

[2:12:38] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.

Published in: Episode

Read next

Great! You’ve successfully signed up.

Welcome back! You've successfully signed in.

You've successfully subscribed to The Cognitive Revolution.

Your link has expired.

Success! Check your email for magic link to sign-in.

Success! Your billing info has been updated.

Your billing was not updated.