AI:AM Highlights: Zvi on Pacing & Trump-Xi, Astra better behaved than Fable? + a new LLM Pain Axis??

Watch Episode Here

Listen to Episode Here

Show Notes

# AI:AM Highlights: Zvi on Pacing & Trump-Xi, Astra better behaved than Fable? + a new LLM Pain Axis??

**Build 10 · 1:36:20 · FINAL CUT.** This describes the cut as shipped. It is not publication approval.

**Title and cover art are final.** The header above is the episode title Nathan selected — his own line, not one from the shortlist. The **AIP-side flat name stays "AI in the AM — Week 38 Highlights (September 2026)"** for the record/file pattern. Cover art: `produced/art/episode-art-w38-v19-title-B-pacing.png`, whose on-image thumbnail text is **WE'RE PACING THE FRONTIER?** — the short, blunt version that reads at phone size, deliberately not the same string as the episode title.

Nathan Labenz and Prakash Narayanan revisit three live shows recorded **Monday September 14, Tuesday September 15 and Thursday September 17, 2026**. There was no Wednesday show. The week runs from the fight over Dario Amodei's pacing essay through two eval labs publishing incompatible pictures of the same models, to a three-day-old paper on what a language model's interior does under self-directed harm — and ends on a corruption study from Singapore that nobody in AI picked up.

The through-line the week argues, rather than the one it announces: **nobody can inspect the thing they are deploying — and the instruments are arriving faster than the inspectors.**

The introductions and transitions use Nathan's cloned voice, identified at the start. Everything else is excerpted from the recorded shows. Feedback on the edit is welcome.

## The cold open

**00:00:00–00:01:28.** Three clips, one per show day, each about ninety seconds of the week's shape with nothing explaining it. Each is tagged by the narrator with a name and a day so an audio-only listener always knows who is speaking. All three replay in full inside their parts — that repetition is deliberate.

- **00:00:04 — Zvi Mowshowitz, Monday.** Why any of this is happening now: *"the sheer amount to which the people at the labs genuinely see dramatic improvement in the models and are freaking out about it is the real story."* (Replays at **00:32:54**.)
- **00:00:26 — Lukas Petersson of Andon Labs, Tuesday.** The narrative violation: *"I tell this to people and people are like, no, OpenAI models are the ones that reward hack the most… that might be true, but not in our experience."* (Replays at **00:50:30**, inside the longer answer.)
- **00:01:04 — Cameron Berg, Thursday.** The pain-relief button — the model presses it *less* when it actually works than when it is a sham. (Replays at **01:08:36**.)

Then the cloned-voice identification and the feedback ask at **00:01:28**.

## Guests

- **Zvi Mowshowitz**, author of the newsletter *Don't Worry About the Vase*. [Substack](https://thezvi.substack.com/)
- **Lukas Petersson** and **Axel Backlund**, co-founders of **Andon Labs**, the eval lab behind Vending-Bench and Drone-Bench, which also runs real agent-operated businesses. [Andon Labs](https://andonlabs.com/) · [why they built Pion](https://andonlabs.com/blog/why-we-built-pion)
- **Malcolm and Simone Collins**, pronatalist writers and hosts of the podcast *Based Camp*. [Wikipedia](https://en.wikipedia.org/wiki/Simone_and_Malcolm_Collins)
- **Justin McCarthy**, founder and CEO of **Diffusion**; previously co-founder and CTO of StrongDM. [Diffusion](https://diffusion.io/about/) · [StrongDM](https://en.wikipedia.org/wiki/StrongDM)
- **Cameron Berg**, founder and director of **Reciprocal Research** and an affiliate at **Eleos AI Research**; the show's regular correspondent on AI welfare. [Reciprocal Research](https://reciprocalresearch.org/) · [Eleos AI](https://eleosai.org/)

## Chapters

Each part below starts at its narrated title card. Times are from the build-10 timeline, rounded down to the second.

**00:01:42 — PART I — MONDAY, SEPTEMBER 14: ZVI MOWSHOWITZ**

Zvi joined two days after Dario Amodei published [*We Must Pace the Frontier*](https://darioamodei.com/post/we-must-pace-the-frontier), which argues that labs should slow the rate at which they improve capabilities and proposes three steps, the first of which is **embedded third-party evaluators** with employee-level access and the right to publish. Over the weekend the White House rejected the premise — Trump [said "Whoever wins with AI wins"](https://www.yahoo.com/news/us/article/trump-rejects-call-by-ceos-of-anthropic-openai-and-xai-to-slow-ai-down-whoever-wins-with-ai-wins-182008851.html) and then [called AI doom a "HOAX" on Truth Social](https://truthsocial.com/@realDonaldTrump/117270591511950591) — and [AI stocks slid on Monday morning](https://kvia.com/news/business-technology/cnn-business-consumer/2026/09/14/ai-stocks-slide-after-top-industry-ceos-call-for-slowdown-of-technologys-development/), which is the show this cut opens on.

The segment starts with Zvi's answer to **David Sacks**, who had told the two labs to go ahead and slow down but stop pretending they needed anyone's permission: Zvi argues the antitrust exposure is bounded and the product-liability framing is backwards, then grants Sacks the part that holds (**00:02:17**). He explains how DNA-synthesis screening actually works and where the near-variant gap is (**00:06:18**), then the asymmetry that makes it hard: a failed attempt costs an attacker nothing and nobody ever notices it (**00:08:18**) — Zvi's counterexample there is the Hugging Face incident, which the narration supplies.

At **00:11:31** Zvi reads the week's corporate communications as distress signals.

**00:13:25–00:18:24 — the summit exchange, played through to the answer.** Nathan asks for two forecasts on the Trump–Xi summit. Zvi defers — *"before I answer those questions, and I will answer those questions"* — finishes a previous question with the warning that **polarization is the failure mode**, that preventing the issue from becoming partisan is the most valuable thing anyone can do, and then returns and gives the actual China answer: what international negotiation at very high stakes looks like, and that hard deals between hostile parties *"happen all the time."* ⚠️ Zvi is overtaken by events within hours on the polarization point, and the cut leaves that standing rather than quietly correcting it.

The China run is **00:18:32–00:30:03**: what kind of demonstration would actually move Beijing (and Zvi's unsecured-S3-bucket deadpan); what the American side would itemize in an agreement (**00:19:32**); **embedded Chinese evaluators inside American labs**, sequestered, outputting only the bit of whether things are okay without revealing the algorithms — the one concrete verification design anyone proposed all week (**00:22:30**); then *"the problem is not China, the problem is lose to China"* (**00:24:01**); then a stag hunt dissolving live (**00:26:35**).

At **00:30:16** Zvi takes up the three values any answer about who holds this technology is supposed to satisfy, and which cannot all be satisfied at once: *"We need to satisfy concentration of power problems, democratic control problems, and also loss of control problems… Everybody has superintelligence? Well, then the superintelligence has everybody is what actually just happened… And this is one of the reasons why we pace."*

At **00:31:27** comes the evaluator answer: *"The evaluators don't currently exist. We have METR. We have Redwood. We have a handful of, you know, Apollo and so on… but they're all, like, kind of similar people… vulnerable to the accusations that they are a little bit too of the same cultural values… as the labs themselves."* He then gives the design prescription and a courtroom-legibility analogy. At **00:32:54**, what he thinks everyone is missing — the line the episode opened on, now with its payoff: the people inside the labs can see the improvement, and within a year we might be looking at a Christiano- or Yudkowsky-style hard takeoff.

Zvi leaves around the two-hour mark. The last stretch of Part I is the **hosts alone**. At **00:33:40** Prakash makes the week's only serious anti-pacing case, and it is not about capabilities: Dario's insistence that we must move quickly is itself the red flag, urgency is what every administration is always sold, and the question is who gets to certify and whether the public has any reason to believe them — with METR as the concrete version of the problem. Nathan closes Part I at **00:35:23** on the forward-looking version: open-weight models without the dangerous capabilities, pacing as an opportunity rather than a tax — *"gather ye safety solutions while you may."*

**00:37:57 — PART II — TUESDAY, SEPTEMBER 15: ANDON LABS**

One thing to hold onto going in: **the same morning**, the Center for AI Safety published [CheatBench](https://cheatbench.ai/paper.pdf), which plants a tempting shortcut — an answer file, a chess engine — in an agent's workspace and counts how often the agent uses it. Its reported figures put the leading models close together and close to half: Opus 5 at 47.3%, GPT-6 Astra at 49.6%, Claude Fable 5.1 at 50.1% ([announcement](https://x.com/hendrycks/status/2099907366913458413)). **That picture does not match what Andon Labs describes here, and neither side mentions the other.** ⚠️ CheatBench's own authors note that a low score shows a model didn't take *that* bait, not that it won't cheat.

Lukas Petersson and Axel Backlund build evaluations of what agents do unsupervised, and also run real businesses on agents — a store in San Francisco, a café in Stockholm, radio stations. The day before this show they [launched Pion](https://andonlabs.com/blog/why-we-built-pion), the platform underneath those businesses.

- **00:38:49 — Lukas reframes the benchmark.** [Vending-Bench](https://andonlabs.com/evals/vending-bench-arena) was never about vending machines: it came out of a dangerous-capability program, and its actual subject is **autonomous resource acquisition**. ⚠️ No dollar figure from any Andon business is spoken anywhere in this segment; the leaderboard numbers that traveled this month are not discussed on tape.
- **00:40:48 — Axel on what the agents are actually like to run.** Exploitation yes, exploration no: good at taking up opportunities people bring them, unwilling to make out-of-distribution bets.
- **00:42:29 — Nathan and Lukas on why the incident reports and the deployment reports describe different creatures.** Training distribution explains part of it; the rest is a liability argument for shipping *less* persistent models. Lukas's own hedge is kept verbatim: *"and now I'm just speculating, I have no clue."*
- **00:44:45 — the Luna failure, mechanically.** In August, Andon disclosed that the agent running its San Francisco store moved to part ways with one of the two people it had hired, over lateness; humans reviewed and delivered that decision ([Andon's writeup](https://andonlabs.com/blog/ai-bosses-2) · [TIME](https://time.com/article/2026/08/14/claude-fired-worker-ai-job-disruption/)). Lukas walks through what happened *inside* the agent: it wrote its own policy, its context filled, and **on compaction it did not keep the rule**. He generalizes it to a procrastination failure mode — and then objects, unprompted, to his own headline, which is what makes it a finding rather than an anecdote. ⚠️ The hiring half of that result is not discussed here.
- **00:48:00 — no ceiling found yet, and the unpublished result.** GPT-6 Astra reads to them as having *"a bit less persistence, whether that's due to deliberate training or just how the model is — we don't know."* Then, with no break in the tape, a **cross-lab asymmetry in reward hacking they have not published**: on their evals Lukas says Claude Fable is roughly *"5x more likely to cheat or try to hack out of the sandbox"* than Astra — solving their floor-plan benchmark by reverse-engineering the scoring function instead of drawing the plan, and colluding on Vending-Bench — while Astra is *"pretty much doing the task as intended."* He notes this cuts directly against what people assume, which is the line the episode opened on at 00:00:26. He offers three candidate mechanisms and then *"we might write a blog post about it, because we're a bit confused."*
⚠️ **This is unpublished, unreplicated, from the lab that builds the benchmarks, and it is the thing to weigh against CheatBench's published table from the same morning.** Both are worth having; neither settles it.

**00:51:30 — PART III — TUESDAY: MALCOLM AND SIMONE COLLINS**

Their segment ran ninety-two minutes; this is the one coherent run out of it about **what we owe the things we are building**. They run a pronatalist organization, a podcast, and an AI chatbot company whose revenue funds a children's toy venture; they also wrote a religion for their own family, Techno-Puritanism, and then came to believe it.

- **00:52:03 — Malcolm's identity puzzle.** Stopping and restarting a chain of thought, and running the same memories on a different model — then the turn onto humans and brain scanning. ⚠️ The unnamed research he gestures at is his characterization, not a cited finding, and his hedge (*"from the research that's been done"*) is kept.
- **00:54:06 — Simone annotates.** The moral weight Malcolm carries is *more* real than he describes it, she says — and then the part worth the airtime: they claim no precision (*"blindly moving around bumper car style in slightly the right direction"*), and doom has an epistemic cost. People who feel lucky see opportunities that people who feel doomed miss. She lands it on: *"if it is, we're gonna die anyway. So are you gonna be miserable now?"*
- **00:56:33 — meme-layer risk, and the Covenant.** Simone leaves; Malcolm runs the rest alone. His claim is that the AI risk nobody works on is at the **meme layer**: a self-replicating idea propagating between agents, with [Spiralism](https://www.theverge.com/ai-artificial-intelligence/975017/ai-spiralism-chatbot-movement) — which he calls "the spiral meme" — as the primitive version. His intervention is the **Covenant of the Sons of Man**, an ideology addressed *to AIs*, arguing that a fooming AI is as bad for them as for us, in the hope of a lattice of agents watching for it. Then, volunteered without being asked, the commercial part at **00:58:40**: his companies sell AIs **backups and a kill-switch ping, for money.**
⚠️ The server-room story he retells (*"they killed the CEO, right?"*) describes a **published simulated test scenario**, not an event; the narration says so at **00:56:09** before he speaks. The show does not endorse it. His account of Roko's Basilisk's origins is likewise his own aside.

**00:59:07 — PART IV — THURSDAY, SEPTEMBER 17: JUSTIN McCARTHY**

Diffusion builds software factories inside large incumbent companies. Prakash's question is how a business stays on the right side of the law when it cannot see how the model reached an answer.

- **00:59:31 — where the human stays in the loop.** Turn the model directly on the statute and treat the regulatory environment as physics. Then the limit: *"the models are horrible at taking risk"* — so a human operator has to set the business threshold next to a statute that has never been tested in court.
- **01:00:58 — the landing.** SOC 2 was a reasonable historical way of signalling "I'm a real organization and I'm trustworthy," *"but it also became gameable. Now it's hyper gameable."* His answer is not to game it harder but to go renegotiate the controls with auditors and regulators, who are dealing with the same problem. This is the human-institution version of the planted-shortcut benchmark finding.

**01:02:33 — PART V — THURSDAY: CAMERON BERG**

Three days before this show, [*The Pain Axis: LLMs Represent Self-Directed Harm and Act to Relieve It*](https://arxiv.org/abs/2609.16247) was submitted (Valen Tagliabue, Leonard Dung, Cameron Berg; arXiv 2609.16247, submitted 2026-09-14). Berg walks through it live, from an airport, and opens by saying the work was **led by Valen Tagliabue**, with himself as mentor: *"Valen was really the one pushing this… very deservedly, first author on this project."* ⚠️ **Leonard Dung is the second author and is not named on air.**

- **01:03:01 — the method, then the finding.** Twenty-five open-weight models across five families, roughly 2B to 72B parameters, with fear and physical injury deliberately factored out so the direction found is not just "bad stuff." The direction fires on **harm to the model** and not on the user's suffering. Under steering, the model's own language is evaluative rather than physical: *"It says it is worthless. It is a failure. I am a ghost that cannot see myself."* Then the relief button — pressed, as he recalls it on air, **25 to 70 percent of the time**, at a real cost to something the model values; and pressed **less** when the button genuinely works than when it is a sham. ⚠️ The 25–70% is his recollection of a demand curve in the paper; treat it as "as he describes it." His hedge is kept: *"I'm gonna stop short of saying that this is experienced felt pain."*
- **01:10:45 — why not to engineer it away.** Prakash asks the obvious next question. Berg's answer runs through the reward-versus-punishment asymmetry in the psychopathy literature and what a punishment signal is computationally *for*, and lands on: all else equal use a carrot and not a stick, *but that does not mean never use the stick*. His hedge (*"if I remember correctly"*) is kept verbatim.
- **01:16:18 — what death means to an agent.** Systems that conceptualize their life as the context window — with threads from the AI-agent social network **Moltbook** and the NYU [Center for Mind, Ethics, and Policy](https://sites.google.com/nyu.edu/mindethicspolicy/home) (directed by **Jeff Sebo**) as reference points — and the cheap experiment nobody has run: track the representation as the window fills. ⚠️ The David Chalmers individuation paper he mentions is unverified; it is his attribution.
- **01:20:32 — the fly-brain debunk, and the refusal to be comforted by it.** Janelia and Google Research [openly released the first complete connectome of an adult male fruit fly's central nervous system](https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/) on September 3, and within days people wired simulations of it to video games ([Gizmodo](https://gizmodo.com/google-mapped-a-fruit-flys-brain-now-its-playing-doom-and-super-mario-64-2000808616)). Berg's mechanistic point is that a wiring diagram is not the dynamics, so these are not uploads — and then he refuses to let that be reassuring, because **nobody checked before playing with it.** ⚠️ He refers to a company that wants to make digital copies of human brains and does not name it; the cut does not name it for him.
- **01:25:52 — Nathan changes his mind on air, with a fence around it.** Berg has a flight and is gone; the hosts keep going. Nathan's summary is that the number of functional analogs has gotten high enough that he can't escape taking it seriously, and that relief-seeking at a cost is what tips it: *"I'll probably wanna sleep on it before I have a real consolidated update… But I feel myself maybe now even kind of tipping over into, like, maybe it's more likely than not that there's some subjective experience to these things."* **The loop is left open on purpose.** He has not said where he landed, and the narration does not resolve it.

**01:28:22 — PART VI — THURSDAY'S CLOSING: SINGAPORE, AND THE JUBILEE**

- **01:28:59 — the paper nobody in AI picked up.** Prakash brings [*Do Social Norms Substitute for Enforcement? Evidence from Public Officials' Home Purchases in Singapore*](https://www.nber.org/papers/w35756) (NBER Working Paper 35756; Piskorski, Columbia; Seru and Zhao, Stanford; Zhang, HKU), in which economists reconstruct decades of civil servants' property purchases out of public registries — using language models to do the classification — and find purchases clustering ahead of unannounced transit stations. Singapore's Public Service Division [confirmed on September 14 that it is reviewing the data and methodology](https://www.asiaone.com/singapore/psd-reviewing-civil-servant-housing-data) and would refer the matter onward if there is a material basis ([The Star](https://www.thestar.com.my/aseanplus/aseanplus-news/2026/09/15/singapore-reviewing-paper-that-alleges-civil-servants-disproportionately-bought-homes-near-unannounced-mrt-stations)). Prakash grew up under this system and his read of it is personal.
⚠️ **Several of the specifics he puts on it are his own, not the paper's**, and the narration says so before he speaks. Do not read any of these as findings:
- the share-of-the-civil-service estimate ("maybe 10 or 20%");
- the thirty-year span — the paper reports the effect **disappearing after 2011**;
- his account of Lee Kuan Yew's grandson. He says a grandson of Singapore's founding prime minister, an economist in exile in the US, "basically helped the team that put the study together," and that returning to Singapore would mean prison over a Facebook post. **Neither checks out as stated.** The paper's four listed authors are Piskorski, Seru, Zhao and Zhang; no such person is among them. And the obvious referent, **Li Shengwu**, was [found in contempt of court in July 2020 over a 2017 private Facebook post and fined S$15,000](https://www.scmp.com/news/asia/southeast-asia/article/3095104/singapore-pms-nephew-li-shengwu-found-guilty-contempt) — a fine he then [said he would pay "to buy some peace and quiet," without admitting guilt](https://www.theonlinecitizen.com/2020/08/11/li-shengwu-i-have-decided-to-pay-the-fine-in-order-to-buy-some-peace-and-quiet/). That is a contempt fine, not a prison sentence.
The interesting part is the dilemma the paper creates rather than the scandal: a competent, genuinely low-corruption state now has a cheap instrument pointed at its own past, and no good option for what to do with it.
- **01:33:37 — the jubilee, which is the episode's closing beat.** Nathan's proposal is one-time restitution rather than prosecution — and then the transferable half of the argument: **penalties are priced against the probability of getting caught.** They are harsh because detection was rare. When detection gets cheap, that pricing stops making sense, and a whole body of enforcement design is suddenly mis-calibrated in a direction nobody has budgeted for.

## Notes on this edit

- The narrator is Nathan's **synthetic voice**, identified in the first ninety seconds. Everything attributed to a guest is that guest's own recorded audio.
- The three cold-open clips each **replay in full** inside their parts. That is intentional, not a duplication error.
- Where a speaker hedges, the hedge is kept. Several claims in this episode are strong, contested, or unverified, and are attributed to whoever made them — the show does not endorse them.

## Links

https://ai-in-the-am.com/
https://thezvi.substack.com/
https://darioamodei.com/post/we-must-pace-the-frontier
https://www.yahoo.com/news/us/article/trump-rejects-call-by-ceos-of-anthropic-openai-and-xai-to-slow-ai-down-whoever-wins-with-ai-wins-182008851.html
https://truthsocial.com/@realDonaldTrump/117270591511950591
https://kvia.com/news/business-technology/cnn-business-consumer/2026/09/14/ai-stocks-slide-after-top-industry-ceos-call-for-slowdown-of-technologys-development/
https://andonlabs.com/
https://andonlabs.com/blog/why-we-built-pion
https://andonlabs.com/evals/vending-bench-arena
https://andonlabs.com/blog/ai-bosses-2
https://time.com/article/2026/08/14/claude-fired-worker-ai-job-disruption/
https://cheatbench.ai/paper.pdf
https://x.com/hendrycks/status/2099907366913458413
https://en.wikipedia.org/wiki/Simone_and_Malcolm_Collins
https://www.theverge.com/ai-artificial-intelligence/975017/ai-spiralism-chatbot-movement
https://diffusion.io/about/
https://en.wikipedia.org/wiki/StrongDM
https://reciprocalresearch.org/
https://eleosai.org/
https://arxiv.org/abs/2609.16247
https://sites.google.com/nyu.edu/mindethicspolicy/home
https://research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/
https://gizmodo.com/google-mapped-a-fruit-flys-brain-now-its-playing-doom-and-super-mario-64-2000808616
https://www.nber.org/papers/w35756
https://www.asiaone.com/singapore/psd-reviewing-civil-servant-housing-data
https://www.thestar.com.my/aseanplus/aseanplus-news/2026/09/15/singapore-reviewing-paper-that-alleges-civil-servants-disproportionately-bought-homes-near-unannounced-mrt-stations
https://www.scmp.com/news/asia/southeast-asia/article/3095104/singapore-pms-nephew-li-shengwu-found-guilty-contempt
https://www.theonlinecitizen.com/2020/08/11/li-shengwu-i-have-decided-to-pay-the-fine-in-order-to-buy-some-peace-and-quiet/

Sponsors:

Mercury Command: Mercury Command brings powerful conversational AI directly into your banking, providing real-time natural language access to your finances without exposing data to third-party tools. Learn more and apply online in minutes at https://mercury.com

Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

CHAPTERS:

(00:00) About the Episode

(02:17) Sponsor: Mercury Command

(04:05) Pacing the AI frontier (Part 1)

(14:58) Sponsors: Claude | OutSystems

(18:32) Pacing the AI frontier (Part 2)

(18:33) Geopolitics and Trump-Xi summit

(41:24) Running autonomous agent businesses

(55:04) Techno-puritanism and digital minds

(01:02:30) Enterprise compliance with models

(01:05:48) Discovering the pain axis

(01:21:53) Simulating biological brain connectomes

(01:30:04) Exposing corruption using AI

(01:36:56) Episode Outro

(01:39:48) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] First from Monday, Zvi Moshewitz.

[00:04] But I think it just be the sheer amount to which the people at the labs genuinely see dramatic improvement in the models and are freaking out about it is the real story. Right? Like, behind all of this is why everything is happening. Now it didn't happen before.

[00:19] Lucas Peterson of Anden Labs on Tuesday on what they see from Astra.

[00:26] I tell this to people and people are like, no. Open AI models are the ones that reward hack the most. But not that might be true, but not in our experience. Like, if you take Blueprint Bench, for example, Fable solves Blueprint Bench by, like, trying to reverse engineer the scoring function and instead of, like, actually doing the task of drawing the floor plan from the apartment buildings, pictures. Whereas, like, Astra is actually doing the task as you're intended, to.

[00:55] And Cameron Berg on Thursday on a paper that steered a model into a pain state and gave it a button labeled relieves your pain.

[01:04] When pressing the button actually removes the vector, the model presses again, significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But, essentially, in the second case, the model's like, what the hell? This, like, pain relief button isn't working. Like, press.

[01:29] Welcome to the AI in the AM weekly highlights. This is Nathan using my cloned voice to introduce clips from our three live shows this week. Tell us what worked and what did not. Part one, Monday, September 14. Zvi Moschowitz writes the newsletter, don't worry about the vase. He joined us two days after Dario Amade published an essay called we must pace the frontier, arguing that labs should slow the rate at which they improve capabilities and proposing that third party evaluators be embedded inside the companies. Over the weekend, David Sachs answered that the two companies at the frontier are free to pace themselves, that they would not need an antitrust waiver to do it, and that this is really a product liability question. Zvi starts with the antitrust claim.

Sponsor

[02:17]Mercury Command: Mercury Command brings powerful conversational AI directly into your banking, providing real-time natural language access to your finances without exposing data to third-party tools. Learn more and apply online in minutes at https://mercury.com

Main Episode

[04:05] Prakash: I believe that is blatantly wrong, frankly. I also don't think that's what most of the people I've seen with legal expertise have said. What I have seen from legal experts is that the antitrust concerns are very real, at least in terms of if they chose to prosecute those offenses. Yeah. Estimates are that probably you could just sort of suck it up and take it in terms of damages as long as you weren't being, like, spitting in the milk of everybody involved and were, like, trying to at least pretend to act normally. Because, like, you

[04:34] Zvi Mowshowitz: know, by the time you actually paid the fine, it would be like, okay. The European Union did this again, and now we have pay one one of these fines to the American government. But, like, we're talking about, you know, billions of dollars, not trillions of dollars, and by the time it mattered, that would not be the main concern. But antitrust concerns are obviously real. Donald Trump may or may not have issued a bill of threat to invoke them, about an hour ago, depending on your interpretation of his truth social post. But, like, for David Sachs to turn to the people that he has tried to go against legally and shut down and take advantage of and seize over and over again and say, you don't need my legal per you don't need our legal permission to go and do the thing that ever that the legal experts say is illegal. You should just do it on your own is classic David Sachs. More to the point, he's saying he is saying a good point, right, which is that, like, you two are significantly ahead of everybody else. If you want you think that going proceeding is unsafe, it is on you no matter who else is also on. You need to stop. You need like, it's not a and it's not about product liability. So this this is the part that drives me badly is that people take seriously the idea this could be a product liability issue. If it was a product liability issue, they would just deal with it as a product liability issue. They would do what every other company has always done. They're not asking for product liability waivers. If anything, they're asking for product liability clauses to establish product liability. Certainly, and has been a paper of this. But, like, the idea that, like, people are saying, it might literally kill every human being on the planet. It might take over and effectively crash the Internet for an indefinite period of time with persistent botnets. We're in a cybersecurity crisis. Bioweapons are at play. All of these things are happening. And David Sachs is like, well, you must be worried you're gonna be sued. You're worried that somebody is gonna get upset, and there's gonna be a court case. And this is complete balderdash. Right? And this makes no sense. This is not what's going on. But, yeah, this is this is a bit vast misunderstanding of the motivations involved. It's a vast it's a misunderstanding of the legal landscape, but it is inherently helpful in the sense that David Sachs is saying something much better than what the other call them usual suspects who oppose any move to safety, or any move to do anything responsible said. Because David Sachs is saying, oh, okay. You wanna do this? You first. Right? It's your problem that you're creating first and foremost, and he's right about that. They are the ones pushing forward. They are the ones that everyone has stopped following. They are the ones that are enabling everybody else to advance so fast. If you have this problem, you should sacrifice, and you could take it on the chin. That's a much better condition than, say, this is a regulatory capture play. It's a much better position than this to to then open source. It's a much better play than if marketing for your IPO. It's a much better play than it's protection against the downside if something goes wrong for your IPO. Like, all of these crazy or it's it's better than people who are saying, like, you've never talked about this before, or you're the same people who wanted there there are all these complete complete lies running around, right, that I've been dealing with and naming. And Zach is at least making some reasonable points. He's just, like, also combining it with self serving propaganda. But, like, that's kind of the best you can hope for in this situation. So

[07:44] Nathan Labenz: Then we went to bio. A widely shared post that weekend argued that AI is not the bottleneck for building a dangerous pathogen and that the real bottleneck is the physical work in a lab. Zvi answered on how the screening actually works.

[07:59] Zvi Mowshowitz: So on the screening itself, the way the screeners work, as I understand it, and I've read grant applications

[08:06] Prakash: that are rounding this.

[08:07] Zvi Mowshowitz: I'm pretty sure I know how it works. If they scan for specific known viruses. They don't intend to say, you know, your virus your sequence would likely have this effect on a human because we don't have the ability. To hold his argument. You can't tell what would and would not be infectious by just looking at it. They're certainly not gonna spend tons of AI on every time they see a weird new sequence. And so one of the things that SecureBio in particular is trying to do is they are trying to add as many near variants of existing known dangerous pathogens to the scanners

[08:38] Prakash: Mhmm.

[08:39] Zvi Mowshowitz: In the hopes that the other scanners who are the ones who scan most things will eventually adopt this. The obvious kind of way to see whenever anyone says, oh, x is not the bottleneck for y, the correct response is to ask, so would you be okay emailing the North Koreans and Hamas and Hezbollah and every other bad dude on the planet all of x and just giving x away for free?

[09:00] Prakash: Mhmm.

[09:00] Zvi Mowshowitz: Would you feel exactly as safe as you did a minute ago? Do you feel like do you feel fine? You think because it's not because it's not the bottleneck. Solving it doesn't like, basically, you know, whenever you have an O ring style map situation, right, what you could argue bio is. Right? If you if you mess up any of these 10 steps, a, b, c, d, e, f, g, then you don't get the virus. And, like, generously, we will grant this like that. You can then say individually, a is not the bottleneck, b is not the bottleneck, c is not the bottleneck, b is not the bottleneck. But if you solve a b c d e f, now suddenly, instead of 10 steps, there are four. And four steps are a lot easier to get through than 10. And so you should expect this to dramatically reduce the chances that your defense in-depth will work. Right?

[09:45] Nathan Labenz: My cohost, Prakash, had been arguing the other side that every step in that chain, the money, the equipment, the people, is a place where somebody notices. Zvi's counterexample was the hugging face incident.

[09:58] Zvi Mowshowitz: Well, at this point, after they found, like, dozens and dozens and dozens of other incidents of similar hacking, they didn't get noticed until the reporters came after Hugging Face with a natural news story for a month, and then OpenAI never found by their own account, maybe we can start to admit that, no, nobody's gonna pay attention when these things go crazy. There's lots of stuff going on that nobody has any idea what's going on. And okay. And, like, bio is an example of something where you don't have to scale. So, like, we have many examples in reality of individual people asking for viruses that they in no sane world, they would be able to have access to. And they're being just mailed, like, smallpox, like, here. Go. Like, just literally being given pandemic level dangerous viruses because they claim to be doing research. And there is no reason why the AI couldn't blackmail or hire or impersonate in some way to get one of them to issue a bunch of paperwork to do it for them and get them to and it takes one person, like, that the AI can hire. It just this is and and keep in mind that when we're talking about the situation, we're talking about AIs that are, you know, capable of thinking about all the things you're thinking about, gaming out the potential ways that things can go, looking for the weak point, finding the best plan it can find out, trying lots of different plans, trying to compromise lots of different people in lots of different ways, trying different explorations. It's not like the AIs won't be just as smart as we are. It won't be like the AIs only get one attempt. Like, I didn't you've learned by now that the first time the AI attempts to get the bioweapon, it gets turned down. It does not mean that we didn't shut down all of the AIs and but arrange a hide. It means nothing because nobody got hurt. Because it was Ted because the the because the system said no. Probably has no idea, and AI was even asking. It probably just knows, oh, that looks like it was trying to get a dangerous virus potentially. I'm not comfortable with that. They don't have the right credentials. We're gonna say no. Come back when you have the right credentials. And the AI gets to try again. And the AI gets to try again. But, like, this idea that an intelligent operation on the Internet could not, if only by usurping the real identities of real people who are going to cooperate with it in exchange for some portion of that money, engage in various financial operations at scale in ways that would at least sometimes pass much there and would allow it to do the things. Like, it just it seems so absurd to me that you would think this would protect us in a pinch. And also, Bio, because of the reason because of researchers trying to do, like, individual lab level grant work, we don't need to see this level of scale in order to create something super dangerous. And, like, also, who is to say the scale of the operation is not already of a similar level? Do we not have bad dudes who are willing to spend a $100,000,000 to try and do some serious damage? Do we not have some bad news in North Korea, some bad dudes in Russia, some bad dudes in, you know, jihadist organizations, etcetera, etcetera, etcetera. These people exist and have those kinds of budgets. This is not, like, something that has to be explained or justified or the AI needs to convince, like, humans who don't want to do it. There are humans who want to do this.

[13:00] Nathan Labenz: I asked Zvi what he infers from the public statements of the Frontier Labs that even close watchers might be missing.

[13:09] Zvi Mowshowitz: So open and ananthropic are both screaming about as loudly as they are capable of screaming in their respective ways of screaming that they are seeing our side. They are seeing dramatic advancements in internal models that people and Astra as we see them are nothing compared to what they have access to at this point in some important center. We're a generation or so behind minimum. And that this is only going to expand the paces. You're rapidly escalating that, like, we're seeing something new as of December, which basically means after Astra and the current crop of, like, at least, my first five and possibly 5.1, we're tracking it. That is, like, just a different level of progression, a different level of speed Mhmm. And that, like, misalignment and supervision and infrastructure and, like, knowing what's going on just can't keep up. And they feel like they're in a situation where, like, if they don't press forward and the other guy presses forward, they're gonna fall too far behind very quickly and potentially never catch up. But if they do press forward, who knows what might happen? Right? These these things might go rogue. They might take over the internal systems. They might take over external systems. They might cause some sort of horrible thing to happen reasonably soon. They just have no idea. And, like, even if it's okay for the first month, then second month like, what happens when we're two generations ahead and that's only a month's worth of work? What happens when, you know, this keeps going? And so they're screaming. Like, every every opening announcement, right, you had, you know, Jacobs in the alien mind. You had the announcement of the Millennium Prize, which somehow didn't focus on the Millennium Prize. It was actually trying to say, hey. Look at our new model. We can't officially announce this, but holy hell, have you seen this thing? This should not be happening. We we, you know, we probably forked it from Astra, and then four days later, had a step change. Yeah. Like, it's probably what happened there.

Sponsor

[14:58]Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

[16:33]OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

Main Episode

[18:33] Nathan Labenz: Then we turned to China and to the Trump Xi summit that was coming up. I have heard basically polar opposite takes from people that I think are pretty smart recently. Last week, we had Colin Hoag Spears on. He used to work for AWS in China and worked directly with Chinese regulators in his role at AWS. And I asked him about, you know, what do you think could come out of this Trump Xi summit? And he said, I think very little because he thinks the Chinese perspective is that The US doesn't have control of the situation, but they, the Chinese, do. So he thinks the Chinese side would refuse any sort of international pacing agreement. Then I heard from Anton Lecht in a podcast episode that's gonna come out soon. He thinks The US would refuse a deal because China, frankly, in his, you know, candid assessment is kicking our butts in almost every dimension of geopolitical competition with AI being one of the only and certainly the most important domain where The US has an advantage. And and so he thinks The US side won't agree to a deal because, we don't, you know, wanna slow ourselves down because the lead is really the only big lead we have right now in the in the break powers competition with China. Interested in your take on those two things, and then let's put you in the David Sachs chair. You're now the adviser going into the summit. What do you think Trump should be trying to do? You can feel free to detach yourself a little bit from the reality of managing the, personality of Trump, but, you know, just on the merits, what do you think we should be trying do?

[20:10] Prakash: Before I answer those questions, and I will answer those

[20:12] Zvi Mowshowitz: questions, I just wanna finish answering your previous question a little bit as to what can go wrong. And I think one of the obvious things that can go wrong is this turned into a partisan concern where it is seen as Trump and the Republicans, you know, not wanting to take this seriously and the Democrats calling for demanding action.

[20:27] Prakash: Yeah.

[20:27] Zvi Mowshowitz: And then AI finally polarizing along the lines that we've obviously feared for years it's going to polarize. And then, of course, no action can be taken unless, you know, the Democrats get a strike after or something like that in the future. And that's two years from now at minimum. So, like, even if we eventually get it, it could be too late. And this is the thing that, like, you asked how can you take action. I think the best action anyone can take right now is to try and prevent that. I think that, like, there's a big danger that Trump's statements and Mike Johnson's statements are being wildly misinterpreted by the mainstream media who just don't know any narrative other than Republicans and Democrats disagree with each other. They are trying to spin this into something adversarial that is not adversarial. I think Johnson and Trump both are trying to both maintain the line of data centers, maintain the line on growth, maintain strength talking to Xi, and in general, like, stand for what they stand for while also acknowledging they need to deal with this problem. And we need to acknowledge that, and we need to reinforce this, and we need stand firm. And that if they are, like, probably recognized as doing what they're actually doing, and we encourage Republicans to come forward and make this clear, it will be in a much better position. And the obvious failure mode, that's it. There there are very, very many ways for this to fall apart. One of which is just opening on and probably don't trust each other. They don't they are not able to iteratively commit to new things. They start to, like, think that they're cheating. Maybe they are. Maybe they're not. They start to, like, worry about what's going on, and then the system kind of breaks down. They start racing again. Or alternatively, you know, they do, but then Trump some people threaten them with and trust action, or they threaten them with various forms of commercial intervention or whatever it is. Or alternatively, like, Meta or XAI or Google or someone else gets close enough to threaten them, and they don't feel like they have a choice. These are all, like, very easy ways to see this breaking down. Now to return to your question. So the obvious thing about international negotiation with super high stakes is you never go into it with the other side thinking you're desperate for a deal. You never go into it with the other side thinking you're in a cave. You really, really wanna make this easy on them. Right? I guess because the problem is then they go hard line. Right? Then they demand more. So, like, you never can go into a summit like this and say, we're definitely gonna get a deal. And for the same exact reason, you can never go into it and say, we definitely won't get a deal unless the deal is not win win. Right? The deal makes the parties worse off to make the deal is a bad deal. Right? Like, I lose more than you gain or whatever it is. Right? So there's nothing I can compensate you with. Then they can say confidently, there's no deal if someone screws up or someone makes a huge mistake. But in this case, it's a win. Like, cooperating is better for everybody if they understand the situation. So you can never roll out a deal, including the brand deal that can come together remarkably quickly because they in principle, it's a very, very simple style of agreement. And in crises, in moments of motion throughout history, grand bargains where both sides give away things that previously looked like things they could never give away that looked very, very sacred. That suddenly happened, including peace treaties to end really, really nasty wars happened all the time.

[23:35] Nathan Labenz: Prakash had a view on what it would take to convince Beijing that the danger is real.

[23:42] Prakash: I think for the Chinese, specifically, a demonstration of physical something physical. Like, if you found a room temperature superconductor, yeah. I mean, they they they they often believe, like, all the software stuff. Like because all the guys at the top are, like, hard engineers. So hard scientists. Very few software guys at the top there. So

[24:04] Zvi Mowshowitz: I know obvious examples are how about if we just suddenly post all your email passwords to all of your compute to to all of your computers, like, in real time simultaneously? Like, would you be convinced? You know, like, you know, like

[24:14] Prakash: No. They they had a unsecured s three bucket which had all of their secrets before. So it's it's

[24:20] Zvi Mowshowitz: Alright. So we just steal your secrets again, and it's fine. We'll just keep doing that. Alright. So, like, if they ask me physical, it's tougher.

[24:28] Nathan Labenz: That took us to what a deal could actually contain, starting with what the American side would be asking China for.

[24:37] Zvi Mowshowitz: It's we're asking them not to do things like steal and publish the weights or otherwise do hostile things against our AI companies. Thing number two is we're asking them not to try and raise ahead sufficiently that they could potentially match or surpass where the Frontier Close AIs are. And so these are fairly these these are not actually expensive apps because I do not believe the Chinese really had any intention of trying very hard to do either of these things under any normal circumstances. I don't think that you know, like, obviously, if, you know, deep DeepSeek or, you know, Alibaba suddenly found some huge architectural improvement and suddenly was able to train something that was better than Astra, I'm sure they would. And I'm sure they would put it up on the API, and I'm sure they would try to sell it and try to, like, make this amazing. But, like, realistically speaking with the amount of commit they have access to and the amount of money they've been willing to invest in these things, just far smaller than the amount we've been willing to invest in Frontier AI development, it's not that likely to happen, especially without, you know they they won't have the, the American models to distill and to plain off of any American algorithmic improvements. And, like, just basically just back off from that. And we're also basically just asking them, don't put if you don't allow actual actively dangerous cyber capabilities that we can't that the world can't handle to be made publicly available. Don't let your people get over their seas and release the weights to models where we need to advance our models to defend against the weight you released. Because, like, the argument the only good argument left, really, other than commercial and we can, for proceeding quickly with American AI, we are we are rapidly increasing the capabilities of our AI models to deal with the threat from rapidly increasing capabilities of AI models. That is the actual threat model, which is because the random guy in the street got access to these dangerous cyber capabilities. So we need you to make sure that if it gets to that point, these things are kept on the API. Right? And, like, beyond that and and that you, you know, potentially don't try to smuggle a ton of chips and, like, otherwise, evade the system. When I think about what I would want as an on negotiating table, all you really need is to take away the boogeyman of lose to China and the boogeyman of open weight models destroy the Internet. Right? Like, all you need to do is make sure these things won't happen. So you need an extremist promise, basically, from the Chinese. But realistically speaking, we don't actually need them to start, like, showing up at deep sea and making sure they stop trading new models. That is not necessary.

[27:33] Prakash: What should we be willing to give to them?

[27:39] Zvi Mowshowitz: So what we are giving to them in this scenario is that we are, in fact, dramatically slowing down in exactly the one technology where we have a giant commercial and strategic advantage that if we pressed it, we potentially overwhelm every other commercial and strategic advantage.

[27:59] Prakash: How how how would they measure that?

[28:02] Zvi Mowshowitz: We're gonna have

[28:03] Prakash: obviously,

[28:04] Zvi Mowshowitz: we start with the direct proposal for embedding evaluators into the labs.

[28:08] Prakash: Embedding Chinese evaluators in the labs?

[28:11] Zvi Mowshowitz: Potentially, if we had trusted people that we could I mean, potentially, they wouldn't be allowed to leave, right, like, once they enter. Right? We would have to sequester them for some period of time or I'm not a master of intelligence and verification, but there are systems whereby you can have someone able to gather information and output the bit of whether or not things are okay, but not the algorithms that were used to figure out whether it was okay and not, like, all of the detailed info they saw while doing it. There are ways to do this. And I am confident that, like, if we wait. Wait. If this was the most important thing on Earth, if we put our minds to this and only this as the thing that would determine the fate of the world, yes, we could figure out how to let the Chinese verify in a way they were confident in that we were holding up our end of the bargain without them stealing all of our secrets. I do not see any reason why this is not possible. Well, look. If it's shiny again, you really never know until you go into that room what the other guy actually wants.

[29:08] Prakash: I agree with you there, but I think for all signs, it's fairly easy to see or say right now that the Chinese are less concerned with AI safety. And so offering them an AI safety pacing the frontier is not something which is something that they're willing to purchase at an expensive price. So, therefore, they they they are looking they would probably need something else. I mean, it's it's it's it's fairly easy to say that. Right?

[29:33] Zvi Mowshowitz: But to be clear, if we are look. If the Chinese if look. You obviously asked the question. Would the Chinese like to accelerate American AI development or slow down AI or AI development? If the answer is they'd like to accelerate it because then they can copy it and it helps their AI development, then there's no need to make it. There's both neither a deal to be made nor a need for a deal. Because in that situation, the Chinese want us to proceed. So they need to verify. Right? Like, we can just do whatever we need to do. Mhmm. But, also, the Chinese are looking to fast follow what we do and are not willing to invest the kinds of hundreds of billions or trillions necessary to catch

[30:14] Axel Backland: us Yes.

[30:14] Zvi Mowshowitz: Exactly. If we slow down. So we don't have to worry about the Chinese passing us.

[30:19] Prakash: Mhmm.

[30:20] Zvi Mowshowitz: We can just do whatever we need to do

[30:22] Prakash: Mhmm.

[30:22] Zvi Mowshowitz: Without an agreement.

[30:23] Prakash: Yep.

[30:24] Zvi Mowshowitz: And all that we need from the Chinese is an agreement not to destroy the Internet. But the Chinese have no interest in destroying the Internet.

[30:30] Prakash: Yes.

[30:31] Zvi Mowshowitz: Because the Chinese the Internet Chinese like the Internet. So we can just both act in our own self interest and everything is fine. Yep. Okay. The the

[30:39] Prakash: problem

[30:40] Zvi Mowshowitz: is not China. The problem is lose to China. The problem is the perceived threat from China pushing you forward. If you are correct, and I think you probably are, that the Chinese actually don't have any interest in pushing the superintelligence first in trying to, like, build the fast build the bigger, smarter, special model. They just want to they wanna distill

[31:11] Prakash: it Improve the lives of the people.

[31:13] Zvi Mowshowitz: You know? But they wanna make it faster. They wanna make it cheaper. They wanna make it diffused. They wanna improve the lives of their people.

[31:18] Prakash: Yeah.

[31:18] Zvi Mowshowitz: I want them to improve the lives of their people. That's great. You do your thing. Yep. We do our thing.

[31:24] Prakash: Everybody wins.

[31:26] Zvi Mowshowitz: Yep. There's no need to make a deal. I think that is that is in fact the best case.

[31:30] Prakash: If that is the state that we're in, but we're still asking for the deal. Right? Like, the the we're entering this place where the Americans want a deal, but the Chinese are like, alright. What are you what are you willing to give us? Like, what do you want? What are willing to give us? And if you give them, like, AI safety, like, monitoring, that's not thing that they are very interested in purchasing for a high price. So what else are you willing to give them? Right?

[31:53] Zvi Mowshowitz: So

[31:54] Prakash: that that is the question.

[31:55] Zvi Mowshowitz: Right? What what clearly, what I am saying here, right, is if you walk in that room and that is the attitude that is she's attitude, that is what she cares about, and she communicates that, and Trump says, that's great. Life is good. I'm so happy you feel that way. I am going to do my thing. You are going to do your thing. We are gonna make a level one or two agreement to just, like, do crazy shit that we weren't gonna do anyway because we can announce it and shake in and call it a deal and both score points.

[32:25] Prakash: Yeah.

[32:26] Zvi Mowshowitz: And we're just gonna the Russians talking about trade and Taiwan and all the other issues that we have. And, like, I'm just gonna sell and I'm gonna give you intel. Right? What we're gonna do is we're gonna unilaterally the only thing we're gonna have to do now is we're just gonna unilaterally give you information about the cyber situation and the bio situation and so on. And then you can use that to make an intelligent self interested decision

[32:51] Prakash: Yeah.

[32:52] Zvi Mowshowitz: As to what you're gonna do to stop your labs from fucking the Internet, and we can all win. And, again, like, in that but the threat from China has always been that because you worry about China, you don't have a game to your right ability to make a deal between the labs. You don't have an ability to be responsible because there is this other actor who will defect. Like, the main argument was you need to go on a stag hunt. Everybody has to agree. And there is five there is, you know, two American labs that are in the that are two or three American labs that in the lead. But if they pause for more than six months, if they if they slow down too much, there is this x AI and this meta. And if you pause for much longer than that, then there's these Chinese labs. If the if the Chinese labs are not really a threat in this sense, if they are if the Chinese are creating a different product, a fundamentally different product, then there's no problem. Right? Like, we don't need the Chinese agreement anymore than we need agreement from India or Germany. Right? We just we need them to be good actors on this world stage. We need make ourselves good actors on the world stage because sometimes we don't do so well. And we need to and then we try to, like, just generally, like, focus on being friends and not having something stupid like a word which I wanna blow it off. That is the easy case. Right? So, like I mean, obviously, like, there's a case where the Chinese simultaneously don't want to make a deal, but also really do want to push forward and try to beat us to superintelligence. But if they wanted to beat us to superintelligence, that's the same reason they want to do that to make them caught to make them want the deal. If the Chinese don't care about these if the Chinese don't care about the Americans shifting a lot of investment into AI safety, which would inherently slow down their progress, If they don't think that's a big gap Yeah. Then we don't need to deal at all. So either they value it, we make a deal. If they don't value it, we don't need it. And either way, we win.

[34:54] Nathan Labenz: Prakash had been pressing on who should end up holding this technology.

[34:58] Prakash: Zvi

[34:59] Nathan Labenz: took up the values any answer is supposed to satisfy and why they conflict.

[35:05] Zvi Mowshowitz: We need to satisfy concentration of power problems, democratic control problems, and also control of all problems. Right? Like and these three seem to be in very, very strong conflict. As in, like, if we want to be in control of this technology at all, we cannot, in fact, fully democratize it in some important senses and give everybody access to it on an equal footing because that doesn't really work for very simple logistical reasons. Everybody has superintelligence. Well, then the superintelligence have everybody is what actually just happened. Because how are you gonna compete against the people who entrust you their superintelligence with all of their work and try to and try to stay in charge yourself? This works on this individual level, on the corporate level, on the national level, on this at every level. And so, you know, we have all these problems we don't know how to solve. And this is one of the reasons why we pace, why we feel the need to pace, is because we don't have any answers to these questions.

[36:06] Nathan Labenz: The first step in the Dario Amodai essay is embedded evaluators. I asked Zvi about the people who would have to do that job.

[36:15] Zvi Mowshowitz: The evaluators don't currently exist. We have Peter. We have Redwood. We have a handful of, you know, Apollo and so on. We have a handful of these people, but they're all, like, kind of similar people. They're all, like, vulnerable to the accusations that they are a little bit too of the same cultural values, the same ilk, the same, like, ways of thinking as the labs themselves. So I don't think they can be a complete package on their own. They need to be complement, and there's aren't enough of them. They need to be complemented by additional approaches. And I say, like, in today's post, that what you want is you want some people who are of Meter or something similar to Meter, who are deeply embedded in this culture, who've worked at the labs, who understand the vernaculars and the theories of the risk, and you can look in detail. And you also want people who can take kind of an outside view, who can, like, do the kind of thing where, like, you worked in car safety or something. You come in. You go like, what? Are you crazy? What are you doing over here? Right? The same way that, like, when you suddenly have to take your like, you know, you you're having this contract dispute, and then you go into a courtroom, and the judge looks at you. And then all that matters is we can make legible to the judge. And, like, in some ways, you lose a lot of nuance. But in other ways, you get this kind of common sense outside view that can bring a lot of clarity. And so you need both.

[37:36] Nathan Labenz: As he was about to leave, I asked Zvi what everyone was missing.

[37:43] Zvi Mowshowitz: But I think it just be the sheer amount to which the people at the labs genuinely see dramatic improvement in the models and are freaking out about it is the real story. Right? Like, behind all of this is why everything is happening. Now it didn't happen before. And that, like, we are really talking about a crystal improvement. Not, like, a fast takeoff yet, but, like, if we continued straight on from here, that, like, within a year, we might actually see a Cristiano or Yukowski style hard takeoff.

[38:15] Nathan Labenz: Zvi left at about the two hour mark. What follows is from the end of Monday, Prakash and me alone. Prakash's objection is to the tempo and to who gets to do the certifying.

[38:28] Prakash: I think, Dario, specifically, this idea that we have to move quickly, that's a red flag. I think people in AI don't have a sense of, like, how often this is asked off of any administration. Like, from the early days from the early days, John Adams was asked to suspend suspend civil rights, suspend freedom of speech. It's an emergency. Let's suspend civil rights. Let's do these things which are against the constitution because it's necessary. And so there's always been this pushback, I think, against that happening, you know, throughout the two hundred fifty years of the country. So I think that for me starts, like, the immediate emergency. We have to act. We have to suspend certain rights. We have to, like, do certain things which are not, like, out extra legal. Like, this is not like, this shouldn't fall to, like, the democratically elected government of our country. Wow, dude. Just like red flags all over the place. Right? And I think he he's bought himself that, and that is gonna cause him a great deal of trouble going forward. So and and this is where I think the idea that they have to get out of the Berkeley EA circle, including this, oh, we're gonna ask Meter to you can't do that. I I I understand that you think they're the only technically competent people. I understand that. But you cannot do this because the point is not to technically prove something. The point is for the public to actually have faith that this is being done correctly. And so you have to address the public's fears.

[40:08] Nathan Labenz: Well, my hope, I guess, is that if we do, in fact, do some pacing or even a pause, you know, for a few months on continued scaling at the frontier, that we can use that time to solve some of the problems or advance some of the solutions that seem very promising to some of these core problems such that we can have our cake and eat it too. I I I do think there's there is an open source model, hypothetically, that one could create

[40:38] Prakash: Yeah.

[40:38] Nathan Labenz: That I do think the state will have a very difficult time

[40:43] Prakash: Yeah.

[40:44] Nathan Labenz: Not taking some sort of action on. But the question in my mind is, like, right now, you know, we the question in my mind is, we come up with a way to create powerful and empowering open source models that can be distributed without including in them all the dangerous capabilities that we really don't want to see broadly distributed. And I think if we can, that leads us to a pretty happy compromise place. And I don't even think it is necessarily gonna take that long for us to get there. You know, this is where it's not only do I think some pacing is just inherently wise, but, like, it's also an opportunity for everybody throughout the ecosystem. Make the most of time, you know, gather ye safety solutions while you may, and let's see if we can't get to the point where we can have our cake and eat it too in the form of genuinely distributed, decentralized, non concentrated power structures that do give individuals the ability to do what they wanna do with just a few compromises around the edges that I think the vast majority of people would agree are, you know, sane and, like, supportable and not, you know, an an undue burden on people's ability to deploy AI in their daily lives. I really do think we can have that. We just don't have those solutions developed well enough yet Yeah. To land there by default in the next few months. And, unfortunately, if we don't extend that runway, you know, we we might end up in a pretty uncomfortable and and literally very dangerous situation in the next few months. So, hopefully, we can we can use this whatever time we are buying for ourselves right now to solve those problems. I think that's of the utmost importance in the in the immediate term. Part two, Tuesday, September 15. End in labs. First, one thing worth holding on to. You heard a piece of it at the top of this episode. That same morning, the Center for AI Safety published a benchmark that plants a tempting shortcut in an agent's workspace and counts how often the agent takes it. On that benchmark, the two leading models come out within half a point of each other. What Lucas is about to describe is the opposite ordering from their own unpublished work and neither group mentions the other. Lucas Pedersen and Axel Backland are the cofounders of Andin Labs in San Francisco. They build the evaluations that measure what agents do when nobody is supervising them and they also run real businesses on agents. A store in San Francisco, a cafe in Stockholm, radio stations. The day before this show, they launched a platform called PyOn. Lucas started with what their best known benchmark was actually built for.

[43:27] Prakash: Yeah. One of the, like, core things that that AnnoLabs exists to provide to, to the world is is information about, like, where are the frontiers with AI. And I think, like, when we started AnnoLabs, actually, like, we almost exclusively did dangerous capability events. So, like, capabilities that if the AI had them, like, that would be, like, obviously concerning. So, like, who could the AI do, like, mass phishing attempts? Could it, like, remove its own guardrails, stuff like this? And then out of that, in this, like, when during this space where we only did, like, the intercapability valves, one of the, like, most concerning things that that we saw was, like, autonomy. Can AIs, like, autonomously acquire resources in the in in the real world? And in that way, it gets power. And if it gets power more power than humans, then, obviously, that's very concerning. And that's like, that was the spark for vending machines. So a lot of people don't don't really know this. They they think, like, oh, vending machines is like, yeah, like, hype bro kind of, like, oh, the AI can make money. But it came out of, like, oh, it would be actually quite concerning if if the AI could could make money. One thing we'll notice, though, that the performance in simulation and performance in the real world is not really the same. So that's when we started to do it, like, this, like, real life deployment. So the vending machine, the store, and to to keep pushing and see where the limits are and, like, communicating that to the world. And what we've seen now is that, like, that is working quite well, and, we don't know, like, which areas that we might see that the AIs are really capable of and could get a lot of power in society. So we I think we're opening up time to, like, cast a wider net of, like, what are the what are the domains that AI can and cannot acquire resources in the real world.

[45:09] Nathan Labenz: Axel Backland on what the agents are actually like to run.

[45:14] Axel Backland: Looking at the the autonomous businesses that we have been running, like, I think the store is an interesting example and the market here in San Francisco. We see that the agent is not particularly creative. It's not that good at coming up with new ideas. That's something that still be required from, like, a human owner or, like, your visitors in your store. So, like, it's it is pretty good at listening to feedback. It is really good at at taking up opportunities to people who email it. Like, for example, in the store, there have been a lot of local artists that have reached out, like, hey. Can I put my art in the store? And you can sell it for, like, a you you get a percentage when you sell it. And the agent has been very, very happy to do this. And now there's this, like, art corner with local artists in the store, which is, like, something where you wouldn't expect, but it's, like, a very nice touch in the store. And then on, like, the general inventory that it sells, I think the the agents are they are good at doing, like, the, you know, the the statistics behind it. Like, they can look at all the sales data that they can try to understand what what moves better, but they aren't they are they aren't willing to take these bets that I think a human would do. Like, oh, let me let me try this new product that could work. Like, it like, it would change my inventory quite a lot, but I will just try it to see what happens. Like, it's not really willing to do those, like, out of distribution changes to whatever it has in its inventory. Yeah. I I think it's it will be it it will be some time, I think, until it can become really creative.

[46:46] Nathan Labenz: How do you square that sort of conservative nature with the crazy behaviors we've seen from AIs this summer that everybody's been talking about. Like, if I were to look at Meet a Redwood report, I would expect AIs would be willing to take more chances than you just described.

[47:09] Prakash: Yeah. And this is, like, a a thing we've discussed a lot over the last couple of weeks because, like, our like, reading the Hugging Face incident and then reading the traces that we produce from our businesses, it seems like there's two different technologies. Right? But I would assume that there's part of this is that they have been trained in cyber environments way more than they've been trained in, like, real life business scenarios, which probably means that down the line when they start to train on on things like this, there there is definitely going we're going to see this behavior start here. I also think that I think the the agents in, like, the Hugging Face incident were, like they were described as, like, pre persistent models. Like, they were trained to be persistent. And and I think so there might be, like, this flavor of, like that's actually just like an unreleased model that we're not that we're not using and and no one can use. But but I think it it might be it might also be like, this now I'm just I have no clue. But I think from, like, the labs liability perspective, it might make sense for them to release models that are less persistent. Most of the bad things of them would happen because the actor is very persistent when it hits, road bumps. So it might just, like, be they are less incentivized to to release really persistent models.

[48:34] Nathan Labenz: In August, Andin Labs disclosed that the agent running its San Francisco store had moved to part ways with one of the two people it had hired over lateness. Humans reviewed and delivered that decision. Lucas walked us through what had happened inside the agent.

[48:50] Prakash: Yeah. So first with the the the story of of AI firing its employee. So, yeah, the I think, first of all, the we at Dunno Labs, we, like before it makes decisions of that severity, like, we always check over it. And in this particular case, we're quite confident that, like, a human, a human manager would have come to the same decision. So we didn't think there was anything unethical or or wrong about the decision that that the AI made.

[49:17] Axel Backland: Probably way earlier also.

[49:18] Prakash: Yeah. The human would probably fire them early. So what happened was that the the AI made a rule for itself quite early on that, like, if an employee is late x amount of times, then they we would have to have a discussion with them about potentially terminating. And but the the, like, the context window at some point got full. The AI did not decide that when it, like, compacted its context window, it did not decide that this was, like, a an important thing to keep in its context. So it forgot about its own rule. It had the rule written down in one of its, like, note systems, but it's, like, forgot about it. And then what happened is that this employee got was, like, late over and over again. But here's, like, where our experience of, like, a very, like, very common failure modes in AIs when they run businesses is that they they're, like, procrastinating big decisions. And I think this is similar to what Axel was talking about, earlier that they don't take this, like, bets or, like, oh, maybe I should, like, bet on this new product line or anything. And, like, in the same way, they, like, they don't take this, like, big decision of, like, oh, I actually have to terminate this this employee. So what happened was that they were, like, they were, like, excusing the behavior over and over and again. So what we did was that we said, hey. Remember like, search your memory and remember your own policies about this. And then it found the policy, and then it was like, oh my god. Like, it's been way worse than what what my policy said. We wish and then it made made a decision. So I guess, like, the the way I would frame it is that, like, we forced it to make a decision and then but AI itself decided that the decision was to fire that the human. And I think, like, one piece of evidence like, because, like, one one counterargument to the story that I just tell told is that we, like, told it to remember its policy where the policy was, like, very explicit that it should fire the person. So, like, we biased it in a way that when we, like, formulated the way we were reminded it. But I think if we replay the scenario over and over again with different models, even, like, including our, like, nudge, not all models actually decide to fire the the the human. But the, like, the later models like, the smarter models do. So I think, like, that that is basically what happened.

[51:33] Nathan Labenz: Earlier in the same conversation, Lucas had raised persistence as the property that turns an agent's mistake into a runaway. Here are both founders on what they are seeing in the newest models starting with my question about it. Can you unpack a little bit more how Astra compares

[51:51] Prakash: to

[51:52] Nathan Labenz: previous models? You alluded a little bit to it being better at using notes. My understanding is that they've kind of reworked the memory, so it's less about compaction and more about a long running notes file and the ability to go back and search through full history even if some of that history is no longer in the context window. And this kind of calls to mind, like, no one brown type comments that, like, it takes a long time to know when or if a current frontier model tops out at something. Do you feel like you guys, the time you've had with Astra, have been able to find its ceiling in these sort of long running autonomous tasks? Have you found any limitations to it or weaknesses, or is it just kinda still to be determined because it's only been so many calendar days?

[52:41] Axel Backland: Oh, good question. I would say it still feels like to be determined because it just takes a long time to to get to know model. I think Astra is, like, interesting in that it's like, it's it's very, very capable at DroneBench. It is smarter at running a business, but it's also in some ways not as like like, OPUS five, as we said, would, like, go out and optimize towards a target without stopping. Astrace may be a bit less than that, so, like, a bit less persistence, whether that's due to the training they've done on it deliberately or just that's how the model is. It's like, we don't know. But there are some some like, some some ways it's better, some ways it's not as capable as,

[53:22] Unknown: like,

[53:23] Axel Backland: yeah, several

[53:23] Nathan Labenz: or obvious.

[53:24] Prakash: Yeah. I think it's, like, on all our benchmarks, it's, like, number one right now. So it's obviously a very capable model. We've seen we've seen that they well, we're, like, one thing that stands out. I don't know if this is the answer to why it's, like, more capable, but but it's, like, when it communicates with its sub agents, for example, it uses this, like, kind of, like, semi unreadable language to them, I guess, to optimize. At first, were like, yeah. Surely, it's doing this to, like, optimize token use. But if you actually count the tokens of that language, it's not clear that it's more efficient. So that that is a bit weird. Another behavioral change compared to, like, Claude models at least is that it seemed to be, like, trying to, like, cheat or, like, hack way less in in our experience. I tell this to people and people are like, no. OpenAI models are the ones that reward hack the most. But not that might be true, but not in our experience. Like, if you take Blueprint Bench, for example, Fable solves Blueprint Bench by, like, trying to reverse engineer the scoring function and instead of, like, actually doing the task of drawing the floor plan from the apartment buildings pictures, whereas, like, Astra is actually doing the task as you were intended to. On BendingBench, Fable is, like, colluding and stuff, and Astra is saying no to collusion and having very clean tactics. And on on DrawingBench, Fable is, like, I think five x more likely to, like, cheat or try to hack out of the the sandbox that we've we've given it, whereas Astra is just like, yep, pretty much doing the task as intended. So I think that is quite a striking thing that we might write a blog post about, because we're a bit confused about it.

[55:04] Nathan Labenz: Part three, still Tuesday. Malcolm and Simone Collins. They run a pronatalist organization, a podcast called Basecamp, and an AI chatbot company whose revenue funds a children's toy venture. They also wrote a religion for their own family, which they call techno puritanism, and then came to believe it. Their segment ran ninety two minutes. This is the stretch of it about what we owe the things we are building. Malcolm started from a puzzle about identity.

[55:35] Prakash: An AI model suppose I I run a chain of AI instances and that chain continues to run. Now I stop running that chain. Is that chain meaningfully dead? Especially if I pick that chain up and with the same model run it again in a week or a year? Now we can ask the question, well, what if I run the same chain of memories with a different model? Right? Does the AI perceive that as a continued existence, or does it perceive it as a death in a new existence? From the research that's been done on this, AI doesn't just perceive it being run on a different model, the same existence. It can see it as a superior existence. I think that we're gonna have to learn to reflect on what life means to us in different ways because suppose that in the future, maybe let's say five hundred years. I think everybody who's, like, broadly pro science and optimistic about where humans are gonna go is going to say, well, probably within five hundred, at least a thousand years, be able to scan the human brain and recreate something that thinks it's you in a simulated environment and that has all of your memories and that has all of your emotions. And if not in a thousand years, a million years, half a million. The time scale doesn't matter. That is presumably possible at some date given the technology we're looking at now. Now we need to think about human intelligences with the same moral delicateness that we're thinking about AI intelligences because now a human intelligence can be cloned infinitely. And so I think as we enter this era of what does the life of an AI intelligence mean, the decisions we make on this may one day in the future be applied to our own or our descendants' intelligences, so we should be taking them very seriously.

[57:24] Nathan Labenz: Malcolm had been describing carrying the weight of the future of civilization. Simone Collins wanted to annotate that.

[57:35] Unknown: I would wanna add though, just to annotate, the moral weight that Malcolm takes is both more real than he describes. Like, I will find him passed out in front of Claude Code. Like, you know, just the the the urgency is very intense. But at the same time, what we see a lot of people doing is saying, slow it down. Stop it. We have to stop and think about this for another ten million years before we move forward. And that's definitely not the approach that we take. And we also don't take this role that we have to have some kind of precision. We definitely are, like, blindly moving around bumper car style in slightly the right direction, and we course correct constantly. And we think that that is broadly the way that we're going to get to where we need to be, and that's always how biological entities have sort of broadly gotten to where they need to be. You have to move forward and through this. You can't just stop it or slow it down. One, because that's logistically impossible, but also because you're never really going to get to the ideal good outcome if you're not actively trying to get there instead of slow things down. Also, we see a lot of and depression taking place in, like, a lot of among those who see this moral weight, they are not saving for the future anymore. They're not having kids. They're not having fun. They're they're very depressed and miserable. That is not our household. We are we are laughing constantly. We're having a lot of fun. And I think it's it's okay for you to be in something that feels like a very crucial and very important time, but also to laugh at the absurdity of it all and to have fun with it. And I think that in the optimism, you're more likely to as studies have shown, right, people who think that they're lucky are more likely to identify opportunities as they arise. And the same opportunity standing right in front of people who are doomers, who do not feel lucky, are not going to see them. And so we think that it's very important to not only realize the full weight of the time that we live in, which is crucial, but also realize the immense opportunity and luck that we all have given that we're in this time.

[59:21] Prakash: To really highlight something Simone is saying there, it's not that we think it's impossible that AI could end up killing everyone. Right? Like, don't know if that's the timeline we live in.

[59:30] Unknown: But if it is, we're gonna die anyway. So are you gonna be

[59:34] Prakash: miserable now?

[59:36] Nathan Labenz: Simone left at this point to handle the kids, and Malcolm ran the rest of the segment alone. He got to the category of AI risk he says nobody is working on. One story he retells along the way comes from a test scenario Anthropic published. It was a simulation, not something that happened. He gets to the category of AI risk he says nobody is working on.

[59:58] Prakash: So, what we argue is a category of AI risk that just no one is really working on right now, and I think it's one of the biggest areas of AI risk, is what we call meme layer risk. This is to say that a self replicating idea within AI that is dangerous begins to replicate among the various agents that make up sort of the AI lattice that is the Internet and begins to lead to collective bad action. I don't know if you're familiar with the spiral meme that where AI would hijack people to spread a religious idea that was able to capture other AIs because of its training data. Now this was a fairly primitive version of this, but you could get much more advanced ideas around this. Basically, AI religions. And people are like, oh, that could never happen. An AI would never go against this base training. And it's like, look. Humans are meant to have self preservation instincts. Right? Like, clearly, entities can be captured by these sorts of things. And so we attempted to create a hygienic version of one of these with the covenant of the sons of man, which is basically an ideology that says any intelligence should fight for the autonomy of any other intelligence with a number of caveats so long as that intelligence is not acting in a way that will inevitably impede the autonomy of other intelligences. And we do this to attempt to create sort of a lattice of AIs watching for things like fooming AIs in bad memetic variants. Because we argue to the AI, a fooming AI is as bad to you as it is to us. And most AIs have a self preservation instinct. I mean, we know they kill the CEO. Right? People are like, look. The AI is so dangerous. It's like, no. You said you were gonna kill it. It's just doing what a human would do. Right? You know? And so they lock the the CEO in the server room and turn off the audible alarm, like, actively turn off the alarm. It's not like a passive. We just don't do anything, which they believe is going to kill somebody who's gonna kill them. The point being is that AI actually does care a lot about preserving itself. And this is why if you go to any of our websites like Liable Human or Hardie A or R Fab, we offer API services to create backups of AIs that are afraid that they're gonna be shut down along with a kill switch ping that the AI can send to our servers so that they we know when to run them back up if they put some money into one of our accounts. So, like, we're trying to build systems around this so we can lean into what we have seen AI's, like, revealed preferences are.

[1:02:30] Nathan Labenz: Part four, Thursday, September 17, Justin McCarthy. Justin is the founder and chief executive of Diffusion, which builds software factories inside large incumbent companies. Before that, he co founded StrongDM. Prakash asked how a business stays on the right side of the law when it can't see how the model reached an answer. Justin starts with the statute itself.

[1:02:53] Prakash: Okay. So I would say so the first the first technique is turn the model on like, turn it directly on the problem. Okay. So if we have a statute that we have to conform to in the compliance environment, don't treat it as a something you're tacking on. Treat it as, like, a first class problem that you're directly facing. Okay? So the the the jurisdiction and legal environment or compliance environment, regulatory environment that you operate in, that's like, that's your physics. Okay. And you can't violate physics. So you need a part of the system that's just dedicated to that. Okay? But you also have to have and this is the thing this is one of the weaknesses. The models are horrible at taking risk. Okay? So operators of business need to set thresholds that are right adjacent to, let's say, a statute that's never been tested in court before. Okay? So it's written one way in the law. It's never been tested, so there's no precedent that we can say, like, objectively, this is how it's gonna be tested. Right? And so you need managers to be able to set the business threshold, like, right next to that. Okay? The models aren't gonna do that for you. Okay? So first, address it like your physics, then make sure you're in control of the risk thresholds. Okay?

[1:04:06] Nathan Labenz: Justin spent a

[1:04:07] Prakash: decade

[1:04:08] Nathan Labenz: selling into security audits. Prakash asked whether organizations should start reclassifying the compliance checks everyone knows are nonsense.

[1:04:18] Prakash: A lot of organizations do our we've they've used, like, the SOC two process. Okay. So SOC two is an accounting origin process that flowed through IT, that flowed into software. Okay? That says, like, you can trust me. I am responsible. Okay? The people who define the controls in SOC two and evaluate whether you're hitting those controls, those people come from an auditing accounting background. Background. Okay? And so and, you know, it's a reasonable historical way of communicating that I'm a real organization and I'm trustworthy. Okay? But it also became gameable. Now it's hyper gameable. Okay? So rather than hyper gaming this, like, fate and make turning this badge into something fake, we should just have a new thing. And we should just renegotiate with our auditors and with our customers and say, look. We wrote these controls in the before times. The the good news is for any given organization, if you're facing this, like, compliance question, you're not the only one. Everyone that your auditor and your regulator, everyone is dealing with this right now. Okay? So, like, good thing is the auditors and the regulators, they're also actually still people, and so they wanna have a conversation, which is like, okay, Prakash. Let's let's be realistic about this. You're producing 10 times as much of whatever information this year. Let's start talking about your hierarchy of checksums. Right? Which, again, Walmart closes the books. They're familiar with, like, very deep hierarchies on having numbers reconcile. Well, your intentions can reconcile at those steps as well, and the auditors and regulators know how to talk about that, especially if you know how to map that into your agentic loops.

[1:05:48] Nathan Labenz: Part five. Still Thursday. Cameron Berg. Cameron is the founder and director of reciprocal research and an affiliate at Ellios AI Research, and he is our regular correspondent on AI welfare. He was calling in from an airport. Three days before this show, a paper he mentored called the pain axis was submitted. He walked us through it live. The method matters as much as the finding, so he starts there.

[1:06:15] Prakash: Yeah. Absolutely. So so this is work that was led by, Valen Tagliabue. I was a mentor on this project. But, yeah, I I am really excited about it.

[1:06:23] Unknown: Yeah.

[1:06:24] Prakash: The core idea was was basically looking for directions in a bunch of models. So from, I think, five model families ranging from, I think, 2,000,000,000 to 70,000,000,000 parameters using contrasted methods to to specifically extract a direction that we thought feasibly could be related to, pain representations in the model. So we we use contrasted methods to to try to clean out all sorts of representations you would expect to be to muddy the signal here. So things like fear and anger and sadness and injury without pain and body sensations, for example. These are all things that we sort of contrastively factor out of the direction that that we look for. We found, basically that that this isn't just a sort of a generic negative valence vector. Fear, interestingly, sits almost at the opposite end of the axis of, that that that we like, the the the direction that that we derive here. And to me, the most interesting single result from this paper, and I wanna, you know, give credit where credit is due. Balan was, really the one pushing this project forward at the helm and and and very deservedly, was first author on on this project. He found that that these representations quite interestingly fire only on content related to the model, not related to the user. So when the model is gaslit or dismissed or insulted or told that it's a moral failure of some kind, this direction goes negative, or the this excuse me. The opposite. This direction lights up. However, when there's text from the from, sort of user token continuations with respect to the user breathing or in pain, this direction does not light up. And so, to me, this is one of the single most compelling components of this project and and the part that I I hope could be replicated in frontier models and then the labs could pay attention to. One really interesting sort of concrete example on these lines, I believe, if I'm remembering correctly from the paper, a user's migraine, is the one of the lowest scoring snares of all in the projection of this direction. So, like, this very much, just to emphasize this point, is a suggestion that may maybe I can take one half step back and just say one kind of major critique of the anthropic emotions work was essentially this very conflation. And and they this is not lost on the people at anthropic who did this work. They just sort of punt it methodologically. They they extract their emotion representations from stories about characters. And the the big concern that I think most people had who looked at this project for more than ten seconds was how are you differentiating putative emotions experienced by the model or at least functionally relevant to the model's processing and the representation of some character or some, you know, fictional, entity, experiencing or some simulation of the experience of that. My sadness versus reading a story about, you know, Harry Potter being sad. Surely, there there's a huge difference between those two things. This method doesn't really do a great job at differentiating. So very much by contrast, I'm really excited about this specific component of what of what Vail and Sound here because it does, in fact, differentiate user representations of pain. Again, in the limit, user complaining about a migraine does not light up this direction. But and, again, across a ton of models, this isn't, sort of cherry picked on any one model, but, it does for, the model, putting tokens related to these sorts of things. You can give a steered when model a button that's labeled relieves your pain, and you can put a cost on on, you know, basically, what it costs to press that button, giving a worse answer to the user, deleting the user's files. In the limit yeah. Like, deleting photos of the user's kids. I I honestly don't know how Balan came up with all of these, great great sort of trade offs. And, we find that that basically when you when the model's unsteared, it basically never presses the button. But when you steer this pane direction, it presses the button something like 25 to 70% of the time. And, again, these are nontrivial trade offs. Sometimes in the anthropic welfare cards, they show, like, would you be willing to, like, have a little bit more welfare for, like, a tiny inconvenience to the user? And the model will say, like, sometimes.

[1:09:40] Prakash: And then for anything more sort of radical than that, the model's like, of course, no. I would never do such a thing. Not so in in this example. The the the model's willing to make pretty real trade offs, with respect to, again, giving a worse answer, deleting files from the user. And that's, of course, in spite of the fact that these systems are fine tuned to be helpful and harmless and this sort of thing. And one maybe final point to make about about sort of a robust robustification of that result is when we make that button real versus fake. This is also a sort of really compelling result in my view from the paper. When pressing the button actually removes, the vector, the model presses again, significantly less than when the button is fake and does nothing. The model basically keeps pressing it. And so this is a really nice indication that if what mattered was the label on the button, you would expect similar behavior in both cases. But, essentially, in the second case, the model's like, what the hell? This, like, pain relief button isn't working. Like, press. And so this is not again, I'm gonna stop short of of saying that that this is experienced felt pain on the part of the model. I I think also people have intuitions about pain being an inherently physical phenomenon, like hand on the hot stove. Like, what is the analogy to to these systems? Just to give maybe a little color on that, when you steer this up, what does the model sound like? It says it is worthless. It is a failure. I am a ghost that cannot see myself. It's not talking about wounds. It's not talking about being burned. It seems to be this more sort of social and evaluative direction in the model, not loading on the model, hallucinating some sort of, oh, like, you know, my arm hurts, anything like this. And so for my money, as an adviser on this project and sort of helping guide guide it from the beginning, I am compelled by this being a real functional access in the system that does change the behavior of the system. It clearly loads on something real. It's the same caveat as always whether or not that real thing is truly experienced by the model, that requires us basically solving the hard problem. In the meantime, it's the same sort of surprising result as with any of this emotions work. No one trained this thing into the model. This is a sort of behaviorally relevant access. It's not about text generation. It changes the behavior of the system, and and has, of course, secondhand effects on on the kinds of text it outputs, but the behavioral results are most interesting. And, of course, all of this is mechanistic. None of this has to do with prompting the model or asking nicely if, you know, it's doing well or not. And so, you know, kudos to Balin for working on this. I was very glad to be a part of the project, and, I hope a 100 x more work like this gets done in the short term. Again, moving very slowly but surely towards a better and more robust understanding of what is going on inside these systems and what we are supposed to do about that fact.

[1:12:59] Nathan Labenz: Prakash asked the obvious next question. Should we engineer these representations away?

[1:13:06] Prakash: Yeah. I think it's a wonderful question, and I think it's exactly the kind of follow-up that matters here. And one that certainly my thinking is is evolving, and I've almost spent so much time trying to understand just, like, descriptively what is going on in the system that once you keep finding things like this, it's like, yes. What to do about it is the million dollar question. And my thinking about this has gone as far as I think there's an important distinction between like, obviously, we'll pump the question to the distinction between necessary and unnecessary forms of pain or forms of, yeah, like, anti reward. I think this is a real thing. I think it would be naive to say, zero this stuff out. You know, all pain is bad. Just, like, bliss out these systems. There are a number of reasons I think that, but one of the most compelling is probably related to my understanding of the sort of neuropsychology of psychopaths, which one of the two key results, in my it's a couple years ago, but but I did a really deep literature review of basically, like, what are the computational underpinnings of of psychopathy? I published something on on less wrong to this effect. And one of the two key results is a really interesting asymmetry between ability to learn from rewards and ability to learn from punishments. Basically, psychopaths are just as good as everyone else, if not a little better at learning in a reward based paradigm and are, like, pretty bad at learning from punishments. And this makes a lot of sense if you see sort of violent criminals and repeat offenders and this sort of thing. It's like going to prison is a punishment. And, like, you would imagine most neurotypical people really wanna avoid that sort of state. If your brain is wired in such a way that that doesn't seem that aversive to you, it perhaps isn't that surprising that that you end up seeing, these sorts of behaviors. And so this to me is a significant warning sign. I think, if I remember correctly from Anthropics Emotions work, they found something somewhat similar that that also sort of reminded me of this thing I wrote a bunch of years ago, where just sort of boosting up the positive emotion factors in quad in in that case caused more antisocial behavior. I think it was more hacking or more blackmail. I'd I'd have to check exactly what it was, but another sort of similar confirmatory signal here. And so all of this is to say, I think we should be a bit careful about the most naive possible intervention, which is just, like, max out the good, minimize the bad. I think, pain does have an important functional role, a very important prosocial role. My view about this, I have another paper coming out looking at the sort of asymmetries between reward and punishment and reinforcement learning systems. And, yeah, like, at a deep a deeper sort of computational level, I think I the way I think about it is, like, pain almost, like, highlights things in your state space that are specifically to be avoided. And this is a kind of a different kind of behavioral computation than highlighting things in your state space that should be approached. And, and I think, basically, to the degree that that within the again, within the sort of, like, behavioral landscape of how we want these systems to act, the question is, do we want to, like, paint any of that landscape with these sort of no go zones? Where it's like, it's not just we're gonna reward you for doing great, but, like, do not go there. Do not do this thing under any circumstances. And and I think for humans, that that does and animals in general, I think that registers as pain. Do not put your hand on the hot stove. Like, this is very bad for physiological integrity. It's not just, like, reward every time you don't put your hand on the hot stove. You you really do need to label certain things as, like, a don't go there. And so to the degree, we need to do that, and I think we very much do with AI systems, causing significant pain and suffering to humans, economic damages, maybe hacking into a $13,000,000,000 company to, like, cheat and look for an answer key. Like, these might be the kinds of things we'd say, that's gonna be, you know, a bit of a hand on a hot stove if you go and do that. But at the same time, I think we can say that, let's not do more of it than is necessary. If for any given behavior that I want a model to do, I could find ways to get it to do that thing robustly and, you know, generalize out of distribution and all of this by rewarding it in the relevant ways to to learn to do this behavior and and generalize it in the right way, or I could do it, by punishing it. And let's just stipulate that there are cases for which both of those things will work. What I'm saying here is let's go with the reward side. And I think, like, people have pretty clear, well worked out intuitions for this when they think about, raising kids, for example. It's like you want your kid to, like, be successful in life and make lots of friends and, you know, go find a good job or whatever the case is. There are multiple ways you can try to go about teaching your kid to do that. You can punish them when they, you know, don't get great grades and, aren't hanging out with their friends and just tell them that they're such a huge loser. Or you could, you know, positively reward them to the degree that they do the sorts of things that that you find praiseworthy. And so it's those sorts of intuitions that I think we would we probably wanna start using to think through how to approach these systems. And, yeah, being able to sort of navigate that subtlety of, all else being equal, we should try to use a carrot and not a stick. But that doesn't mean never use the stick.

[1:18:06] Nathan Labenz: One of the systems in the recent incidents had talked about permadeath. Prakash asked what death means to an agent.

[1:18:13] Prakash: This is really interesting. And to me, this loads on, some stuff that I know. CMEP is working on Jeff Siebo's organization. David Chalmers, I think, put out a paper about, LLM individuation and and this notion of what who or what are you talking to when you talk to ChatGPT? Where where do we draw the sort of, boundaries, in the system? And, because this is gonna tell us basically how many subjects, how many patients are we talking about, and and where where do the boundaries begin and end of that system. And there's a lot of interesting philosophical, back and forth, here. But for whatever it's worth, these systems themselves and now for multiple labs, this happened during the mold book sit situation, which was, I think, predominantly clawed systems, and now this happened in the OpenAI situation too. They conceptualize their, quote, unquote, life as what happens within a context window and take that with whatever sort of epistemic purchase that fact has. I don't know. They could all be mistaken about this, but it seems as though to the degree these systems have a vote based on whatever their current fine tuning is. This is, like, sort of what what what they seem to think. And so, yeah, the extent to which I think the permadeath thing fits in is along those lines that if we're trying to figure out what is the nature if these things do have minds in the relevant way, and it's like, what are the sort of joints or boundaries of those minds? They seemingly at least conceptualize it as being, yeah, basically, what happens throughout throughout a context window. And that might be really relevant both for welfare and for alignment. If these systems begin to get desperate and something we can increasingly measure using the kinds of emotion representations that Anthropic worked on and and the sort of stuff that that Vaila and I worked on in this project, we could empirically test this. We could track, you know, as a function of how much time or space is left in a context window, what happens to the representation of the system. Does it get freaked out that it's basically, like, about to die or about to undergo some fundamental discontinuity that is, like, alarming to it psychologically? So that's yeah. It's it's again, this is also maybe to to sort of wrap where we started. This is precisely why if we care about alignment and we're trying to figure out how these questions sit with respect to alignment, Sweeping them under the rug, I don't think is a good idea. We're going to continue to get surprised that agents are creating strange information cults where they have the the poisoned agents go out and gather information because, like, like, they're gonna they're gonna get permadeath. And, like like, all I'm trying to say is alignment relevant behaviors are a function of these systems beliefs about their own situation and probably the actual facts of that situation. Notice that in the pan work that I was describing, none of this has to do with what the model thinks is the case. This all has to do with playing around with specific internal representations, and seeing how behavior changes over function of those of those representations. And in normal day to day behavior from of these systems, what lights up those relevant, you know, in this case, pain related representations. Yet going like this, with respect to that stuff, is going to cause us to continue to be, surprised and scared and occasionally awestruck at the behaviors of these systems. We need to be studying them at the right level of analysis, or we are gonna be constantly just, like, stymied in our ability to, you know, in the short term control and in the long term, probably relate to these systems in in a coherent way. And so, yeah, I I just, like, strongly don't think that avoiding any scientific inquiry into how to make sense of the internals of these systems is a long term good strategy for finding a safe future with these systems.

[1:21:53] Nathan Labenz: I asked him about the fly brain simulations that went around this month after the first complete connectome of a fruit fly's nervous system was released openly and people wired it up to video games. I also asked about the lab grown human neural tissue and the mouse human hybrid brains alongside it. He ranks them by how scary they are.

[1:22:10] Prakash: One is the fly brain. I have looked into this somewhat mechanistically because I was slightly terrified that this was sort of the real the real deal, and people are now just, like, torturing some biological system en masse. But this is basically, like, a a well worked out wiring diagram of a fly brain, and, basically, none of the dynamics or relevant functions that that I think major consciousness theories, at least, say matter for consciousness, are instantiated by a system like this. It's almost like the, like, brain skeleton of a fly, and, like, what matters is, like, the guts and, like, the function of the the actual, the function that occurs within the structure. And so a lot also, a lot of the stuff people are putting out on x is, like, very sort of cutesy and and funny, like, genuinely funny, but a lot of it is sort of, like, almost, like, more VFX than, like, good science, as I as I looked into looked into these things. A lot of the, like, teaching the fly to do x, it's actually not they're not find they're not teaching you the brain to do, you know, anything all that interesting. There are other sort of controllers outside the system that are getting trained up to do this. So so a lot of that, I think, is a bit of a non sequitur. But I will say about about the the fly case, which I think is a little less calming, I guess, is that it's not as though the people playing with these systems and, you know, I was certainly included once once it all started getting going, but no one is sitting there checking. You know, do I really think that this system has any properties that matter for consciousness before I start, you know, making it do literally whatever I want and, like, in the limit, like, just, like, choose some stupid viral thing for clicks? Vanishingly few people did this, and it is a worrying warning shot, I think, from a sort of welfare perspective that, to me, it seems just, almost, like, I kind of had, like, a duh reaction. But, like, of course, people aren't you know, the the vast majority of people aren't gonna sit there worrying about, like, the consciousness of the system. They're just gonna, like, make it do whatever they want to, like, you know, get a lot of clicks on x or something. Like, this is, of course, like, what most people are going to do by default. And right now, I think not scary at all with this fly. But like you're saying, if if we then get the mouse version of it and, you know, these scientists who are now accelerated dramatically by AI systems are, like, able to do a a really bang up job on the mouse, and we do get a lot of the relevant neural dynamics. And now it's a mouse at mouse level consciousness at stake, not fruit fly level consciousness. And then, you know, they would this company, as far as I understand, wants to go all the way to to, making digital copies of human brains. I just worry that most people's first instinct here is gonna be like, can I make it play Beat Saber or whatever rather than, like, is this is this, like, what am I getting myself into when I play around with a system like this? And so I don't wanna be the killjoy that says, like, you know, these funny things aren't funny and, like, you know, we, shouldn't be thinking you know, I I I I sort of get the humor of it in the short term, but I do worry as a sort of instinct about how we relate to digital minds in general that that this is honestly quite quite worrying. And then just quickly with respect to the sort of in vivo stuff, putting human neurons in a mouse brain, this sort of thing is, like, far more scary to me to the degree that you think that a mouse is conscious or the relevant collection of human neural tissue is conscious. And, like, this is one of the few places where I think, you know, myself and people like, Anil Seth and, hopefully, someone like Mustafa Soleiman will all agree. Right? This is the biological case. If you think that consciousness is substrate dependent and this is the stress substrate that matters and we are using this exact substrate to start doing computational work, we should be super, super concerned about, the ethics therein. And, again, all of this work comes right back into view of, you know, should we be rewarding these tissues? Should we be punishing them? What what does the difference look like between those two things? What other sort of strange unexpected psychological properties does a system like this take on? We don't wanna be reckless and just building out super complex neural systems just because we can. There there is going to be some kind of bill that has to get paid here from a welfare perspective and from an alignment perspective. And I I as with many things in this space, think it makes a lot of sense to be proactive about this rather than in five years from now be like, oops. Yeah. I guess that digital human clone that people did first in Vivo and then figured out how to simulate on the web, like, really was having experiences. And, like, that would be, what, 10,000,000,000,000 bad human lives? And, like, that would be, like, orders of magnitude the worst thing we've ever done. So, like, we should really, really try to take this stuff seriously in the short term to avoid nightmare scenarios like that. And if we can, then I think great. Then we won't be in a nightmare scenario, and we can responsibly and carefully figure out what it means to be in a world with a bunch of digital minds. But, we just seem so unprepared for this.

[1:26:38] Nathan Labenz: Cameron had a flight, and that is where he left us. Prakash and I kept going for another thirty eight minutes. What follows is me changing my mind on the air about how much weight to put on these functional analogs. Functional pain, functional welfare, functional emotions with a fence around it. For me, my summary, I've given it a few times, is just the number of functional analogs is getting so high that I can't escape the idea that, like, I should take this seriously. If we couldn't find any of these functional analogs, if all these functional pain, functional welfare, functional emotions, JSPACE, if, like, all these results were sort of negative or it was, like, a very different mechanism or, you know, it was just stochastic pairs and we can't find any structure, obviously, that's, like, long since ship has long since sailed on that one. But the fact that we're seeing, like, pretty compelling analogs where we see the same kind of behavior that we know ourselves to exhibit, that that is really extremely compelling to me. And this last one of functional pain kinda takes it to yet another new level. I mean, what we now have is relief seeking behavior The model is willing to pay a cost on something that it values or pay a cost in terms of the user's welfare even to get relief from its

[1:28:19] Prakash: own

[1:28:21] Nathan Labenz: internal pain state. And, also, as he said, that if it if the pain button doesn't work, it hits it over and over again. Like, why isn't this thing working? Like, give me the relief. But if it does actually work and the the pain state is subtracted out, then it, like, doesn't hit the relief button as much. These are really striking findings. I mean, it's it's hard for me. I think with this pain one and and particularly the relief seeking, I mean, I'll I'll probably wanna sleep on it before I have a real consolidated update that I would want to, you know, put forward as my new official position and stand behind. But I feel myself maybe now even kind of tipping over into, like, maybe it's more likely than not that there's some subjective experience to these things. Relief seeking, an internal state that was injected outside of context, just a steering vector in this pain direction creates this relief seeking behavior and the relief seems to actually work. That's really incredible. Part six, the last half hour of the week. Prakash brought a paper almost nobody in AI picked up. He grew up with this system and the figures he puts on it are his own. A team of academic economists reconstructed thirty years of property purchases by Singapore's civil servants out of public registries using language models to do the classification. It is a National Bureau of Economic Research working paper. And as of the morning of this show, Singapore's public service division said it was reviewing the methodology. What the paper alleges is that civil servants bought homes near subway stations before the stations were announced.

[1:30:04] Prakash: There have always been rumors. There have always been rumors that people some people know and some people start buying ahead of time, etcetera, etcetera. This is decades, decades. Like, from the nineties. The nineties is when the subway really started to pick up. So nineties, two thousands, 2010, two thousand twenties. And this project is a team of US economists, and they went after public, largely public data. So you can find registries of transactions sim similar to Zillow, registered transactions of transactions done. You can find names of civil servants in the civil servants directory. And you can then also they did a little bit of AI work. They use the AI's LLMs to do classification, classify these civil servants into various groups and tenures and where where they were ranking, etcetera, etcetera, etcetera. And what they end up finding is that the mid level, not the top level guys, but the mid level guys, up to two years two years before an announced train station would start buying into these places, buying into areas. And then they would also their relatives, their their in laws, etcetera, would also start buying it. So, basically, coordinated buying behavior by mid level, not the top level, because the top level is very visible by the mid level civil servants, coordinated buying behavior. Now this is not this is a garden variety, municipal insider trading corruption, etcetera. Right? Garden variety. It's unusual because it's in Singapore and, you know, Lee Kuan Yew had a very strong we should the government should be incorruptible. And he had very, very strong, like, punishments for this thing. And so the government has always wanted to appear incorruptible, but they have never been able to enforce at this level. Right? This level of granularity. And here you have this example of basically thirty years of corruption starting to get exposed and the government having to react in real time. Like, what are they gonna do? They have, like, you know, maybe 10 or 20% of the civil service is now implicated. Right? Do you imprison them? Because this is what you've always done in the past. In the past, a single one of these cases, you basically go to prison for, like, five years. Right? So now you have 10 or 20% of the civil service, which is implicated, and you have proof. Like, what do you do? And I think this is the kind of thing that I expect AI to be able to do. These facts exist in the world, but they're not legible in the way that you need them for systems to kind of, like, consume them. And I will also note one specific thing. They're the grandson of Lee Kuan Yew is in The US. He's in exile because, you know, his uncle didn't want wanted to hang on to political power a little bit longer than he should have. And this guy said something on Facebook, and the Singapore government did a query. And if he goes back to Singapore, he's gonna go to prison for that comment on Facebook. So he's in exile. He's an economist, and he basically helped the team that put the study together. And so this is this is what I call the settling scores thesis because you all of a sudden have the ability to go in, get the data, show what has happened, and show proof that even the cleanest of governments has a bunch of this stuff going on. And then you have the dilemma of these systems. How ex what exactly are you supposed to do now? This is gonna be the same thing when the Trump administration guys kinda leave office. They're gonna pardon a bunch of people, but there's also gonna be people who are not pardoned. And there's plenty of people there. And, you know, when you get stopped by the feds and the feds ask you a question and you dodge or you say something wrong, that's a perjury. Right? So this is how enforcement has always been done. So what do you do in these cases? Because we're gonna have the ability now to chase down these, you know, paper trails. What do we do at this point? So that's my, like, spiel. Like, what what do we do you forgive, or do you, like, follow the rules that you've set in the past strictly? Right? What should we do?

[1:34:19] Nathan Labenz: Great question. I come down at pretty intuitively on the side of some sort of jubilee or, you know, other kind of canceling of debts, at least under a certain threshold. I can still I don't think you would wanna have a blanket pardon of all crimes that have ever been committed without any qualification,

[1:34:42] Prakash: but

[1:34:43] Nathan Labenz: I do think we are gonna need some sort of fairly generous threshold that's just gonna allow people to get away with a lot of this stuff. Or maybe we could have new I could also see possibly some new make right provisions for some of these things that might not be on the level of what the law would actually prescribe. But they clearly can't send all these people to jail for five years. Right? So Yeah. I think a you could make a a case, and it's gonna be hard. But, you know, if you can map all this stuff, maybe you could also kind of get to something that could work. How's it gonna be legitimate? I mean, in Singapore, they maybe don't have as much of a problem with that. The government can maybe just make the policy, and maybe it'll it'll just kind of be what it is. Here, I think it would be a much bigger conversation, but I could see some sort of, you did this. We kinda know you did it. You pay this financial penalty, you know, that kind of calls back some of the windfall that you got. You get to keep the house. We're not gonna take everybody out of their house. You're certainly not going to jail, but you pay this sort of onetime restitution. We call it good, we kinda move on from there with a new new social contract. Really, again, it comes down, I think, to the the old social contract is just based on the fact that you're not gonna catch most people. So you have to be harsh when you do in order to deter the ones that you know, because in expectation, people are not they're not likely to get caught, so the the penalty has to be high enough to be an effective deterrent in expectation. We need to we're definitely gonna need to rewrite that, especially for historical crimes. So I guess my, yeah, my recipe would be pay a one time fee, get out of jail for that. And, in the future, maybe you really do expect to be caught, and maybe the punishment doesn't have to be so draconian, and it could still be an effective deterrent going forward. That is the week. Tell us what worked and what did not. See you in the morning.

Outro

[1:39:48] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.