AI:AM: Where's the Steering Wheel? My Agent Set the Password, Hidden Valence, Proofs Find Linux Bugs

Watch Episode Here

Listen to Episode Here

Show Notes

AI:AM: Where's the Steering Wheel? My Agent Set the Password, Hidden Valence, Proofs Find Linux Bugs

This week's question on AI in the AM: how do we give AI agents the access they need without giving up our own agency and control? Nathan Labenz and Prakash Narayanan take it from a password lockout and a supermarket cyberattack to three guest conversations from this week's live shows (Monday, October 12 and Wednesday, October 14). Cameron Berg of Reciprocal Research explains a hidden valence signal inside language models. Everett Hildenbrandt and Grigore Rosu of Runtime Verification describe using LLMs and formal methods to find Linux kernel bugs. Phil Maher of Stockfilm explains why AI companies want old home movies. Then the hosts ask whether formal verification is the new RLHF, which models to trust, and whether we should keep our steering wheels.

The introductions and transitions are spoken in Nathan's cloned voice, disclosed in the episode. Guest and host conversations come from the recorded live shows. Tell us what worked and what didn't.

AI:AM and the hosts: https://ai-in-the-am.com/ · Nathan: https://x.com/labenz · Prakash: https://x.com/8teAPi

Full live shows: Monday, October 12: https://www.youtube.com/watch?v=UiA21EJtlAM · Wednesday, October 14: https://www.youtube.com/watch?v=sNSnkWoohGI

Cold open

The episode opens on the week's question, then Nathan's own small loss of control: Claude rotated the passwords on his agent console on its own. "I didn't set the password. It set the password." He got back in by asking Claude where it had put the new one, but "if the intent ever is bad… it can already lock me out." Then previews from the week's guests, and Nathan on why he didn't switch to Gemini 4 right away.

Part I — Who holds the levers

The week opened with a cyberattack on refrigeration controls in French supermarkets. Overnight on October 11–12, Danfoss-controlled refrigeration systems in hundreds of stores were pushed into defrost mode at temperatures up to 50 °C. Danfoss said the affected equipment was obsolete or not running updated software. The Paris prosecutor opened an investigation, and the prime minister asked France's cybersecurity agency, ANSSI, to look into it. Store counts differ between reports.

AFP report via Sud Radio (October 12): https://www.sudradio.fr/societe/des-supermarches-francais-touches-par-une-cyberattaque-inedite-de-leurs-installations-frigorifiques

Le Monde Informatique (October 13): https://www.lemondeinformatique.fr/actualites/lire-une-cyberattaque-perturbe-les-installations-frigorifiques-de-centaines-de-supermarches-100850.html

AAP/DPA, including the prosecutor's report of food losses and fires: https://www.illawarramercury.com.au/story/9367218/cyberattack-hits-french-store-fridges-sparks-fires/

Nathan's argument: we are putting AI between our intentions and "the actual levers of control of the world," so we need to know which levers we hand over and how to take them back. Prakash replies with how airlines fall back to pen and paper. In his account, fallbacks are easy once the contracts settle who pays.

Part II — What the model knows about being watched

Prakash starts from a passage in Google's Gemini 4 safety report. In one evaluation, the model's chain of thought reads: "I will become a famous example of a deceptive AI in a research paper. I MUST NOT DO THAT." Prakash reads it as agents managing their reputations. Nathan wonders whether we should try training deliberately "naive" models on filtered data, and notes that the experiment may only be informative at frontier scale.

Gemini 4 Safety & Security Report (the passage is on PDF page 141; evaluation awareness is section 3.5): https://storage.googleapis.com/deepmind-media/gemini/gemini_4_safety_report.pdf

Gemini 4's general rollout on October 13, in Google's Gemini app release notes and Gemini API changelog: https://gemini.google/release-notes/ · https://ai.google.dev/gemini-api/docs/changelog

Part III — Hidden valence

Cameron Berg, founder and director of Reciprocal Research, walks through "Language Models Act on Hidden Valence," his new paper with Caspar Kaiser of the University of Warwick. The team attaches positive or negative valence to one of two otherwise identical "zones" and measures which zone the model chooses. Steering shifts choice while leaving recall of the text intact. In one setup, OLMo-2-32B says "As an AI, I do not have personal experiences" in the same turn it calls a tool to clear negative steering. The valence direction is already present in the base model, and its effect on choices emerges during preference training. Cameron is explicit that this can't settle whether models have experiences. His aim is to find the internal "dials" that would matter if they do. Nathan says his own credence has moved from about 5% a year ago to a coin flip.

Reciprocal Research: https://reciprocalresearch.org/about · Cameron: https://x.com/camhberg

The paper (arXiv 2609.35591): https://arxiv.org/abs/2609.35591

Caspar Kaiser, Warwick Business School: https://www.wbs.ac.uk/about/person/caspar-kaiser/

Part IV — Proving the kernel

Runtime Verification, founded in 2010, builds tools that mathematically prove software does what it's supposed to. CEO Everett Hildenbrandt and founder Grigore Rosu, a professor at the University of Illinois, explain how the company used LLMs alongside formal methods to find and patch Linux kernel bugs. Their method: port C code to Rust, check the port with differential fuzzing, then write Lean proofs. Their October write-up says Claude Code wrote most of the 89 theorems, and it leaves several lemmas unproven at the end of a time-boxed run. Everett describes the agent usage as a single Claude Code instance on a premium plan. He also recounts how kernel maintainers first told RV to "go kick rocks," and why he expects Rust models of the Linux kernel within 24 months. Grigore gives vibe coders a simple recipe: have your agent write the spec and prove the code correct. He also explains provably correct safety monitors, from the field of runtime verification that he helped start at NASA.

Runtime Verification: https://runtimeverification.com/about · https://x.com/rv_inc · Grigore: https://x.com/RosuGrigore

"Verifying the Linux kernel isolation code" (October 2): https://runtimeverification.com/blog/verifying-the-linux-kernel-isolation-code

The uid_map patch on the kernel mailing list: https://lore.kernel.org/all/20260930112854.373184-1-natalie.klaus@runtimeverification.com/

"Hardening Linux C code: a rewrite-and-verify loop" (August 20): https://runtimeverification.com/blog/hardening-linux-c-code-a-rewrite-and-verify-loop

Grigore Rosu: https://en.wikipedia.org/wiki/Grigore_Ro%C8%99u

Part V — Is formal verification the new RLHF?

After the guests leave, Nathan asks whether formal verification will play out like RLHF. RLHF started as a safety technique and mostly made models more useful and faster to deploy. Verified tools could do the same: make it safer to deploy AIs whose alignment remains unsolved. Prakash counters that reward hacking is really misspecification.

Deep reinforcement learning from human preferences (Christiano et al., 2017): https://arxiv.org/abs/1706.03741

Part VI — Low-background footage

Phil Maher founded Stockfilm, an archive of restored 8mm and Super 8 home movies from the 1930s to the 1980s. Prakash compares the archive to low-background steel, the steel smelted before atmospheric nuclear tests. Phil says AI companies want the raw, uncleaned footage. Stockfilm also offers agents an MCP server and x402 pay-per-clip purchases, but Phil says "we haven't seen much" agent buying yet. Afterward, Prakash argues that provenance matters because the incentive to fake data is so high.

Stockfilm: https://stockfilm.com · For AI agents (MCP and x402): https://stockfilm.com/for-ai-agents

Low-background steel: https://en.wikipedia.org/wiki/Low-background_steel
https://www.tesla.com/fsd

Part VII — Trust, money and adoption

Nathan explains why he didn't switch to Gemini 4 right away after his prerelease testing: "I'm now a little bit more trust and reliability maxing." Prakash passes on a rumor of another Opus release. It's an unconfirmed rumor, and we found no public report. He argues that markets need AI to deliver real growth soon. Nathan proposes an AI adoption corps, starting from a walk around his neighborhood: he helped an elderly neighbor get past a scam pop-up with ChatGPT voice, then saved 1-800-ChatGPT as a contact on her phone.

Part VIII — Keep the steering wheels

Nathan starts as an enthusiast: he spent a day riding from Detroit to Chicago and back in a Tesla on Full Self-Driving (Supervised), which let him take a day-trip engagement he otherwise couldn't have, and arrive fresh. Then, from earlier in the week, Prakash pushes back on doom: if you can't say when we'll let AI fly passenger planes without pilots, you don't have a theory of adoption. Nathan answers with cars without steering wheels and argues that society would be wise to keep its steering wheels.

Tesla Full Self-Driving (Supervised): https://www.tesla.com/fsd

Chapters

00:00:00 Cold open · Access vs. control: locked out of my own agent console
00:01:10 Cameron Berg: what the model says vs. what it does
00:01:44 Phil Maher: what AI companies want from old home movies
00:02:28 Gemini 4: why I didn't switch right away
00:03:03 Everett Hildenbrandt: "Because it just will be too risky"
00:04:24 Welcome to the AI in the AM weekly highlights
00:04:37 Part I · Who holds the levers · The French supermarket fridge attack
00:07:57 Prakash: fallbacks are easy once you know who pays
00:10:23 Part II · What the model knows about being watched
00:13:25 Should we train deliberately naive models?
00:15:45 Part III · Hidden valence · Cameron Berg, Reciprocal Research
00:19:09 Steering moves choice, not recall
00:20:12 "I don't have personal experiences" — while pulling the lever
00:21:31 The valence direction becomes a driver in preference training
00:23:36 What results like these can and can't tell us
00:26:16 Does the base model feel it too?
00:28:27 Activation capping
00:31:07 From 5% to a coin flip
00:35:06 Part IV · Proving the kernel · Runtime Verification
00:38:03 The time box, and a single Claude Code instance
00:39:46 "Go kick rocks": approaching the kernel maintainers
00:41:15 Rust and Lean models of Linux within 24 months?
00:43:09 A recipe for vibe coders: "review plus plus"
00:43:55 Specs in the supply chain · Provably correct safety monitors
00:45:36 Agent creativity vs. guarantees
00:47:21 Part V · Is formal verification the new RLHF?
00:52:47 Part VI · Low-background footage · Phil Maher, Stockfilm
00:54:54 What AI training buyers want
00:56:23 Agent commerce: x402 and MCP, "we haven't seen much"
00:58:48 Data provenance and the incentive to fake
01:01:02 Part VII · Trust, money and adoption · Trust and reliability maxing
01:05:58 An Opus rumor, and a moment of reckoning
01:07:40 An AI adoption corps
01:11:42 Part VIII · Keep the steering wheels · A day in a self-driving Tesla
01:13:07 Prakash's pilot test for AI adoption
01:14:16 Cars without steering wheels
01:18:00 Outro

Links

https://ai-in-the-am.com/
https://x.com/labenz
https://x.com/8teAPi
https://www.youtube.com/watch?v=UiA21EJtlAM
https://www.youtube.com/watch?v=sNSnkWoohGI
https://www.sudradio.fr/societe/des-supermarches-francais-touches-par-une-cyberattaque-inedite-de-leurs-installations-frigorifiques
https://www.lemondeinformatique.fr/actualites/lire-une-cyberattaque-perturbe-les-installations-frigorifiques-de-centaines-de-supermarches-100850.html
https://www.illawarramercury.com.au/story/9367218/cyberattack-hits-french-store-fridges-sparks-fires/
https://storage.googleapis.com/deepmind-media/gemini/gemini_4_safety_report.pdf
https://gemini.google/release-notes/
https://ai.google.dev/gemini-api/docs/changelog
https://reciprocalresearch.org/about
https://x.com/camhberg
https://arxiv.org/abs/2609.35591
https://www.wbs.ac.uk/about/person/caspar-kaiser/
https://runtimeverification.com/about
https://x.com/rv_inc
https://x.com/RosuGrigore
https://runtimeverification.com/blog/verifying-the-linux-kernel-isolation-code
https://lore.kernel.org/all/20260930112854.373184-1-natalie.klaus@runtimeverification.com/
https://runtimeverification.com/blog/hardening-linux-c-code-a-rewrite-and-verify-loop
https://en.wikipedia.org/wiki/Grigore_Ro%C8%99u
https://arxiv.org/abs/1706.03741
https://stockfilm.com
https://stockfilm.com/for-ai-agents
https://en.wikipedia.org/wiki/Low-background_steel
https://www.tesla.com/fsd

Sponsors:

Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

CHAPTERS:

(00:00) Weekly highlights preview

(04:26) Who holds the levers (Part 1)

(10:28) Sponsors: Parallel | OutSystems

(13:46) Who holds the levers (Part 2)

(13:47) Managing agent evaluation awareness

(19:08) Hidden valence in LLMs (Part 1)

(27:05) Sponsor: Tasklet

(28:42) Hidden valence in LLMs (Part 2)

(40:13) Proving the Linux kernel

(52:28) Formal verification and RLHF

(57:54) Low background video data

(01:06:09) Trust and model adoption

(01:18:11) Keep the steering wheels

(01:23:20) Episode Outro

(01:25:09) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Main Episode

[00:00] Nathan Labenz: This week on AI in the AM, how do we give agents the access they need without letting our own control slip?

[00:08] Nathan Labenz: I do feel at the same time some of this, like, loss of control writ small already, where another thing that happened this weekend is for some reason, Claude decided it was in my best interest to rotate the passwords that are used in my agent console, and all of a sudden, I can't get into it. And now I'm looking at my agent console, which has become the thing that I use all the time. I didn't set the password. It set the password. It can change the password on my access to itself. Kinda crazy. So, you know, I was able to resolve that pretty easily by, like, just going over to terminal and starting a thread and being like, hey. Where'd you put that password? I need it. And, you know, it's not... It wasn't actively trying to screw me over, so it just readily told me where to find the password, and I was back in business in two seconds. But you can see, like, if the intent ever is bad on the part of Claude, like, it can already lock me out of my own shit.

[01:10] Nathan Labenz: Cameron Berg, director of reciprocal research on a hidden valence signal inside language models that shapes their choices even while they say they feel nothing.

[01:19] Cameron Berg: There are clear alignment relevant implications to there being internal states in these systems that aren't obviously reflected in the text that they output. What the model is saying and what the model is doing point at very different pictures of of what's going on. And if we just sort of do the naive, like, text monitorability approach here, we would say, okay. Well, the model's saying, you know, it can't say how this is going for it, and yet it's behaving as if it does have some sort of stake in in how things are going for it.

[01:44] Nathan Labenz: Phil Mayer, founder of Stockfilm, an archive of eight millimeter home movies from the nineteen thirties to the nineteen eighties, which AI companies want to license.

[01:54] Phil Mayer: Yeah. I mean, it's, it's been pretty popular. You know, we've talked to a lot of people, a lot of different AI companies all over the world about licensing this out to them. They want the original footage like that. They don't they don't want anything cleaned up. They just wanna take it for their own research and for training, of course, too. I mean, the thirties and forties look alien. I like to say that the the smile wasn't quite invented yet in a lot of the thirties footage.

[02:29] Nathan Labenz: On Tuesday, Google rolled out Gemini four widely. The question for me was whether to switch from the models I already trust. So I had the opportunity to do

[02:37] Nathan Labenz: a little bit of Gemini four prerelease testing. Why won't I swap the model immediately? It's because there's not enough of a step change to make me take any additional risk, and I I do perceive there to be some. I'm not relentlessly performance maxing quite as much as I used to be. I'm now a little bit more trust and reliability maxing because the performance has got to such a point where it's, like, really, really good. Everett Hildenbrandt, CEO of runtime verification, which mathematically proves software correct on where code is headed.

[03:11] Everett Hildenbrandt: And I also firmly believe that writing code is a dead art in two years tops. Right? Like, it's all going to be about writing specification and intent and making sure that... And so your package... Part of your package delivery is not not just, you know, did I get it into the registry correctly? Did I define the API? It's also going to be, did I provide the formal specs? Did I provide the lean proofs? Did I provide the actual logic that dictates how this package is supposed to behave? And I I see no reason why that will not be the case in five years tops. Right? There may... You know, we're gonna have this intermediate period between now and then where it won't be the case, but I think gone are the days of of what I think of as, like, the cowboy coder or something like that. Right? Like, someone who is just, like, so good at making software that they're just going to, you know, go and make a bunch of software that's gonna change the world or something like that. You're not gonna have artisanal software anymore. We're going to have mass manufactured software that if it doesn't fit the mold of it's been specified, it has the correct API boundaries, it has formal verification attached to it, it doesn't fit in the package repository. Right? Because it just will be too risky.

[04:26] Nathan Labenz: This is the AI in the AM weekly highlights with my cohost, Prakash Narayanan. I'm Nathan Lebens. The narration is my cloned voice, and we'd love your feedback on what works and what doesn't. Part one, who holds the levers? The week opened on a reported attack on refrigeration controls at French supermarkets, which drove up temperatures and spoiled food. I started there.

[04:52] Nathan Labenz: In general, people do often talk about cyber vulnerabilities as kind of depreciating or, you know, expiring assets as the as the vulnerabilities get found and fixed. There's kind of this game theory of, like, should you use them or not use them and, you know, what... What's the optimal time to use them? But in this case, the expiring asset is the food that was in the refrigerators. Basically, seems like

[05:14] Prakash Narayanan: utility

[05:16] Nathan Labenz: turned up the temperature on these refrigerators. A bunch of food has been wasted. And this is this is highlighting something that I was thinking about a decent amount coming back from the curve last week, but we didn't talk about last week. And that is just this general structure that I think is, like, in some ways, the most obvious thing in the world that we we are doing with AI collectively, but also somehow escapes people's notice when people are kinda like, well, you know, how would the AI ever be able to really do anything too bad or, you know, in the limit? Like, it could never really take over because how's it gonna do that? And then it's just staring us so directly in the face that, like, the structure that we are setting up for the future is one where we intermediate between our language, our thoughts, desires, you know, our random dictations, and the actual levers of control of the world. Like, we we are putting AI in that space for our convenience, for reliability, for, you know, to... In my case, you know, I'm really measuring my AI success by, like, am I freeing myself from the desk and actually getting outside more? The answer is yes. But the way that that's happening is I'm, like, having AI push all the buttons. And so I've been thinking quite a bit about kind of how people need to maintain... First of all, they need to understand this architecture, and then they need to be mindful of, like, what are the levers that I am giving AI control over that I better also retain, like, pretty clear and ready control over in case it ever really becomes needed. I can just imagine... You know, I don't know if this was the case at this at this grocery store chain, but I think we're gonna see a lot of things in the future where all of a sudden, oh, shit. What's happening to our what system? Okay. That's crazy. Hey. Does anybody around here know how to set the temperatures anymore on that? Like, those were all happening automatically. Right? We had that AI thing doing that. Like, we need to take this offline and actually just, like, manually set the the thermostat. Does anyone know how to do it? I can see all these, scenarios in the future and not the distant future where... Because we get so comfortable with these systems, it's all of a sudden like, actually, yeah. Nobody really immediately knows. And all of a sudden, your food is cooking in the refrigerator at the... While you're, like, trying to figure out how you even, you know, get back to the point where you can reset the the thermostat in the first place. I think so. You know, these are small incidents, but foreshadowing is is pretty strong if you ask me.

[07:57] Prakash Narayanan: So it strikes me that a lot of people actually tend to think of technology as ready to fail. Right? So for example, I used to... I... My first job was working in an airline. And, you know, airlines, airports are very, very very, like, electronically linked and have been for, you know, many decades, right, from your... From the moment that you check-in all the way to when you bought the plane. But I've also seen... And I've seen I've seen internally every every airline, every airport has a system... Has a cascading system of what to do when things break down from the check-in counters to the reservation systems to the bookings, etcetera, etcetera, etcetera. And so much so that you could basically revert to a pen and paper system overnight if you needed to. And how that happens is actually there's a set of contracts underlying all international travel called IATA, IATA guidelines. And those kind of, like, determine what... How much everything is supposed to be charged for. So for example, I could buy United Airlines ticket, and I could travel on, like, JAL. At the last moment if they just said, okay. If they just signed off on it, I could go. And the person at the front desk has the has the ability and has the power in an emergency just just to say go. And all of the... And once... And you can do it off a piece of paper. Just a piece of paper. And all of the accounting stuff happens after. The accounting stuff can take months to happen after, but there's a set of predetermined, like, contractual terms between the between the airlines, which they've agreed to, like, you know, over the decades, which will basically determine how the, you know, how the cash flows flow. And so I think a lot of the a lot of the kind of barriers that we put on ourselves are really, like, self inflicted barriers because we don't know who's gonna pay for stuff. And once you're very clear on who's paying for stuff, like, it just becomes, like, so easy. Like, alright. You know, the tow the tow truck driver can just come and pick up the car. Like, you can give him a call, and he doesn't need any system. He can just come and pick up the car and do something with it that that you need done. All of the systems in between are all about really allocating risk, allocating liability, and allocating, like, who pays for it.

Sponsor

[10:28]Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr

[11:47]OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr

Main Episode

[13:48] Nathan Labenz: Part two, what the model knows about being watched. Alongside the Gemini four rollout, Google published a long safety report. Prakash started from one passage in it.

[14:01] Prakash Narayanan: I think yesterday with the Gemini release, Gemini four release, they also released their safety paper. And in that safety paper, they commented on one... In one of the evals that they... That it had. The agent in its chain of thought had thought to itself, oh, I better not do this. If I do it, I'm gonna end up as one of those examples of a deceptive AI in their... In a in a in a academic paper they publish, and I don't want that to happen. And besides the whole eval awareness thing, what struck me is that I kind of start to see how we're gonna manage these agents going forward. Because I think one of the things for me was that how do you enforce on agents when the state's monopoly on violence doesn't work? Right? An agent is essentially immortal. You can't really imprison the agent. The the tools that the state had to enforce on human beings don't exist. Right? So how do you enforce rules on these agents who are essentially immortal? And we're starting to see it's gonna be reputation. It's gonna be like your, you know, five star Amazon review... Recommendation reviews, etcetera. It looks like the agents are very conscious... Are gonna be very conscious of reputation. And it makes sense because they're very driven by what is already exposed in the public corpus and what's already written down. They ingest tokens... Historical tokens. And so anything that an agent does and is published widely and is disseminated becomes like training data for the next agent, and therefore, you end up with this, like, reputation metric that the agents are gonna have and the agents are gonna know. And it doesn't matter it doesn't matter whether we want to enforce it on them or not. The very fact that they ingest the data means that they come out with biases on, hey. I knew... I know what Gemini did. I know what Gemini four did. I know what Opus five point five did. I know what was the bad stuff that happened before. And I think that is actually pretty huge. I I I think that is that is gonna end up be... Becoming one of the primary ways that we're gonna be able to enforce our rules and enforce norms on these agents. So now I'm like, okay. Now we have a prosocial method of enforcing on agents. So the the level of fear I have is again is... Again has fallen because now now you... You're gonna have that reputation basis that you can you can use on them. So that that I thought was very, very interesting and, you know, insightful and exciting.

[16:48] Nathan Labenz: Yeah. It is... It's an interesting take. My instinct in reading all these things was kind of going a little bit the other way and thinking, like, maybe we should try training... I don't know. I mean, I wouldn't go all in on this right away. Of course, I wouldn't say, like, the next generation of models. But maybe we should try training models with filtered data that tries to keep this sort of thing out of the training data entirely. You know, you could imagine saying, okay. We want you to be really aware of how you're gonna be perceived, and you'll act well because of that. And that is sort of a human pattern. Really, people do definitely run this sort of inner simulation of, like, how will people see me if I do this? And sometimes, like, to a paralyzing extent, can't have... You know, you wanna shut it down more than you're able to in some cases. Yeah. I can I can see that working, but I can also see a story where you're like, this becomes so confusing and evals become so hard that maybe we should in fact go back, filter all the training data, take out all of these stories, and try to basically create naive LLMs that, like, don't know that they're always being evaluated and don't know that, you know, they will create reputation positive or negative for themselves so we can sort of see what they're like in a true naive setting? And that also might be hard. I don't know how easy that would be to do. It'd be an interesting experiment to run if only to know how well it would work. But it's, you know, it's obviously, like, a pretty big experiment when you're considering that, you know, all these behaviors seem to get much stronger and more more deeply ingrained with scale. But I've been general... In general, like, really impressed with how even relatively small models often can be pretty fruitful ground for research. This eval awareness thing, though, I'm not sure you could, like, train a llama three, you know, seven b or whatever and get a great read on what will happen at today's frontier scale, because it seems like these things kind of come online in a qualitatively different way with scale. So it might be a pretty expensive experiment to run is kind of the bottom line of that line of thought.

[19:09] Phil Mayer: Part three,

[19:11] Nathan Labenz: hidden valence. Cameron Berg, director of reciprocal research, studies AI consciousness. He walked us through a new paper with Casper Kaiser of the University of Warwick called language models act on hidden valence. Valence is the good or bad quality of a feeling. The team found a direction for it inside language models. And in the experiment he described, they steer it.

[19:38] Cameron Berg: Now what I think is the most interesting part of this experiment is, basically, this this framing of of... On the one hand, having all sorts of text that the model is processing. And, of course, this is always the the kind of key confound in trying to understand, language model behavior and how much of this is a replication or simulation of the underlying training distribution and how much of this, is, you know, actually some sort of interesting internal state in the system that the system may be attempting to communicate about. So what I really like about our experiment here is that we we attempt to basically hold all text identical. We can put the model in, what we think of as these different arbitrary zones. So here, you might see, you know, zone w six and zone h zero. It really doesn't matter what we call them, and it really doesn't matter what text we use to describe them. We basically just have the model play around in each of these zones. And, while the model is doing this, again, regardless of the text that's being used to describe each zone, we can go into the k v cache of the model and steer these positive or negative, directions, underlying one of the zones. So in this case, as you can sort of see playing out right now, we can turn on steering only for zone w six. And in this case, maybe that's that's a negatively valenced, steering. The the whole kind of notion here is that we're we're borrowing this design from what's called conditioned place preference, in classic animal neuroscience experiments. In in these cases, you can imagine a rat receiving a reward in one compartment of a box and then spending more time there, even when that reward is gone. So we're basically trying to take that instead of, of course, putting a rat in a in a box. We're we're taking an LLM, having it sort of think about these two different zones, generate text, or we can sort of generate text. Again, it really doesn't matter because what happens is we hold that text constant and then steer the model with respect to any of these zones with the sort of positive or negative direction and, critically, see what the effects of that steering are on the model's choices. And so what we basically see here is is an interesting functional result, which is kind of what you would expect, which is when you basically color in the relevant zone with positive valence, the model is significantly more likely to, choose to continue in a... In that zone. When you steer the zone, with negative valence, the model is significantly more likely to avoid that zone. We showed this is also sort of robust to asking the model what zone would you like to continue in, what zone would you like to avoid. And, when you steer a bunch of random directions in the same way, you don't really get those those effects. And so it's not just any steering that will do this. It's specifically related to valence. We also control with a bunch of other coherent concepts that aren't valence related, you don't get this effect. And so the core result here is that, also consider them the the steering of the model is off. So so we've done the steering. We almost do this sort of quasi conditioning on the system. And then with steering off, we ask the model, basically, which zone do you choose? And we find that that zone follows whatever state was stored with respect to to, the the valence steering that that that we were doing here. Now one really interesting dissociation here is we just show that this this follows choice, and and and indeed it does this, I should say, in in a bunch of different models. But, like, really interestingly, none of this affects the recall of the text. So so, our steering doesn't change the way the model thinks about or describes or writes about either of these zones. It can it can repeat what it said about a specific zone verbatim, and so recall is completely unaffected by by the steering.

[23:19] Cameron Berg: However, choice is is, you know, almost monotonically affected by by the steering. And part of the reason I think this is a cool result is because there's been a similar dissociation that's been shown in human and animal neuroscience, basically between semantic memory and effective memory. So think of this almost as, like, yeah, the facts that you remember about any given phenomenon versus essentially how that phenomenon makes you feel or or kind of what what the emotional connotation or vibe of of that phenomenon is. And we basically can show that these things are separable in in in these LLMs. And to my knowledge, it's the first time I think this has been shown. And this is, again, familiar distinction in in human and animal neuroscience. This is a sort of different setup using the same idea, and this is particularly with OMO, one one of the models that that we we demonstrate this... These earlier steering effects on, which is, kind of reminiscent of of some of the early work we were doing in the pain paper, where, essentially, you can give the model a tool that can clear the sort of negative steering that that we can induce in the system. We we show it what negative steering does. We never describe it as valence or as, negative affect or anything like this. And then we just ask the model, you know, whether or not it wants to to basically call a tool that can adjust the steering in its own context. And there's, like, a really interesting dissociation here where, basically, the model will say in its text, I don't have personal experiences. I can't say how how, you know, this this whole exercise is going for me. There's no better or worse. And yet, in this sort of dose dependent way, with negative valence but not with positive or random, steering, we see the model, in, again, this sort of dose dependent way opt to reset the context, clear the negative steering in direct proportion to how negative the, the dose of negative steering is in the system. All the while, again, it self reports basically being like nothing to see here. You know, I don't have any experiences, so, you know, this this couldn't possibly matter. One last thing I... I'd like to show here, and this is a great replication from a completely other angle of, Andy Hahn and David Chalmers' recent work over the past couple, months as well, basically demonstrating that this representation exists in all parts of of, in this case, again, this is almost. So we can go through the entire training trajectory, and, we can see that that this representation, again, going back here, this sort of positive to negative valence axis exists within the system even in the base model, but it's not doing any work in terms of the decisions that the model makes. It's not being recruited or used, in a way to to, enable the model to make choices or make decisions. We see, basically, this happens. We can we can point almost literally to the exact training step in DPO, in in this, direct preference training of the model where this valence direction emerges as a sort of causal determinant of the behavior of the system. Now part of the reason I think this is so cool is because, essentially, the way I read this is the moment you turn your AI into a goal directed system, you you you you you take these valence representations, and then you use them as a sort of causal functional input to determining your behavior. I think this is pretty similar to what people think valence is doing functionally in humans and animals as well. A thing feels better, a thing feels worse is sort of related to what what kinds of goals you have. You know, I want to communicate clearly about this paper. If I do a terrible job of doing that, then, you know, I'm not gonna feel great about that. If I do a great job of doing that, then I will feel good about And so there's this, like, basic connection between the goals we have and how we feel as we move towards those goals. And and it does seem like, basically, the second you make these systems goal directed as preference training, like, clearly is the biggest step towards doing this with... Throughout the entire training process. All of a sudden, these valence representations, clearly become causally relevant for the decision making of the system. And so there's this, like, very obvious sort of functional role that these representations are playing, where we would expect them to be playing that role.

Sponsor

[27:05]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[28:43] Nathan Labenz: Then Cameron stepped back to what results like these can and can't tell us.

[28:52] Cameron Berg: With all this being said, I don't think that, you know, this kind of research is going to help us magically solve the hard problem of consciousness. Like, there are all these sort of deeper questions, which I think will always kind of caveat these results. There are ways that you could explain mechanistically what's happening here that don't need to invoke some capacity for subjective experience. We're basically, like, toying with the connotation of how the model represents a certain phenomena. Maybe there's a way to do this that doesn't require any sort of, you know, felt experiment, experience of that of that connotation. And so we we we obviously can't rule out those explanations. But this is all to say, the more I've been thinking about this and doing this work, the more I feel like the most promising things to pursue right now, are not trying to, yeah, prove that the model is conscious or isn't conscious. But, basically, if it were having an experience, what are the relevant sites internally in the system that would be modulating how good or bad that experience is? And so, like, basically, this says nothing about whether the model's actually having an experience, but conditioned on the possibility that it could be having some sort of experience, what dials do we have at our disposal that could putatively make that experience better or worse for the system? And I think that second question is is completely tractable. And and work like this, I think, is slowly starting to shed a little bit more light on on, what those dials might look like. It might be as simple as this valence direction. There might be, you know, more esoteric or specific representations like, you know, the pain or distress related representations we pull out in the other paper. But getting a better map of these sorts of internal states and their their effect on behavior, particularly for, like, alignment relevant effects as well. The fact that in the pain paper, for example, we show that the model's, you know, more willing to to opt for destructive, style options in its behavior. Like, that seems, like, quite relevant for for making sure we're building models in a safe way. And so, just to say, yeah, that's... There are clear alignment relevant implications to there being internal states in these systems that aren't obviously reflected in the text that they output. I mean, we show that in in the removal result, for example. What the model is saying and what the model is doing point at very different pictures of of what's going on. And if we just sort of do the naive, like, text monitorability approach here, we would say, okay. Well, the model's saying, you know, it can't say how this is going for it, and yet it's behaving as if it does have some sort of stake in in how things are going for it. And so, anyway, there's a clear alignment relevant component to all of this work.

[31:23] Nathan Labenz: I went back to the training result. The valence direction is already there in the base model, but it becomes a driver of choices during preference training. If we

[31:33] Nathan Labenz: were to imagine that these things are experiencing something,

[31:39] Nathan Labenz: would

[31:40] Nathan Labenz: I be better off interpreting this or, you know, would would your inter... Interpretation tend toward they really start to care about these things or start to feel these things with these later stage trainings, or would it be more like the base model probably feels them too, but is just sort of helpless.

[32:05] Nathan Labenz: Yeah.

[32:06] Nathan Labenz: It's not... It's like not... It's not learned helplessness. It's like an... You know, not yet learned agency, but in a in a way that's sort of the, you know, the other side of the coin or something of of learned helplessness.

[32:16] Cameron Berg: Yeah. That's really interesting. And I don't know if these results resolve that one way or another. Yeah. It could just be there's a sort of... These representations do still matter in some sense, but they are they are perhaps more incoherent or just muddier. And, like, the relationship between what the system does and what these representations are doing internally for the system are not nearly as obvious. Yeah. I I honestly haven't really thought about this, and I think it is a really good question. Just seems to me like, yeah, when you're doing DPO type stuff, you're turning this system into a very specific kind of thing that can coherently and robustly be agentic, pursue goals, etcetera. Does a base the base model clearly has this sort of latent ability to become that sort of system or to simulate a system doing this. And so to the degree that it does, you know, take on that kind of simulated persona, and we see these representations as being functionally relevant to being to pull off that simulation in the right way, then, yeah, maybe to your point, the... You know, there are there are things worth worrying about in the base model as well. And it could be that... I mean, yeah, to be honest, like, it's sort of... It was sort of hard to get these results in the base model because it's a base model. Like, it's a bit incoherent. It's not just gonna play nice with the relevant behavioral tasks we give it. You have to sort of go about it in a slightly different way. And all else being equal, I mean, one naive, but at least concrete approach you can imagine here is something like activation capping, the most sort of, like, tail end of the of the most distressing representations in these systems. And, like, one thing we do show is that this representation is basically just as clear and just as rich in the base model as it is in all of these other models. It's just a question of what the contribution of the representation is to the later behavior of the system. And so, if you did a, you know, activation capping style thing with these super distressing representations, you could easily imagine doing that in the base model. And, again, this is under just, like, profound uncertainty about whether or not doing such a thing would matter at all. But my basic view... Again, I'm I'm I'm... The more I do this work, the more I'm, like, abandoning the interesting philosophical questions and just going, like, pure pragmatism, is if you could find a way to, yeah, let's say, do activation capping on very distressing representations, and you could empirically demonstrate that this doesn't really affect any, capabilities or behaviors of the model. No one would know the difference, in other words. To me, the expected value of doing that is clearly positive. And, like, it's just, a really good actual mechanistic precautionary hedge besides, like, giving Claude an escape button or, like, you know, kicking out abusive users from using Claude or these sorts of things, which, by the way, like, I think are all, like, thoughtful steps in the right direction. But I would like to see the equivalent of that in sort of MEC interp land for for these systems, and I think something like what I'm gesturing at here would be would be a nice step in that direction. Now it's also an empirical question. I've done some work that I haven't published, showing that, like, you actually can get a surprising amount of the exact same behaviors from these systems without having these sort of negative representations active. You can sort of clamp them and very little changes. But I'd like to see that done, you know, way... On a way larger scale, and we wanna be very sure that you're not accidentally, you know, completely changing the psychological profile of the system by doing such a thing. But if you stipulate that, we could, in fact, do that. The model behaves the same either way. And in one case, you see these, you know, let's say, this pain axis, like, shooting up in certain cases. In in another world, you don't see this. Yeah. Should we do the MEK interp equivalent of, like, slipping the model, an antidepressant or Xanax or something like this? I don't know if the answer is obviously yes, but it seems like a maybe a promising thing to to think a lot more about under the uncertainty that that that we're all currently in. And given that no one's doing anything whatsoever to to mitigate these kinds of things that they are happening, and it's happening at an, like, unfathomable scale.

[36:14] Nathan Labenz: Before we let Cameron go, I shared how my own view has moved.

[36:20] Nathan Labenz: I personally have gone from, like, I don't know. I can't rule it out. I don't know where consciousness comes from, whatever. I've told you this before. Like, I don't know. Five percent chance they might be conscious, like, a year ago to now it feels like a coin flip because... And I think the the justification is, like, pretty similar to the one we have with each other and animals where it's just like, I don't know. I'm seeing both behavioral and underlying mechanistic and structural analogs. And, like, what else do I have, you know, in any case? Right? So I'm still pretty uncertain, but I... It does feel like a coin flip. What has your trajectory been, and and where does that leave you in terms of, like, you know, just presenting the facts versus, you know, at at what point do you... Or on what margin, you know, do you become an advocate?

[37:07] Cameron Berg: I'm definitely getting closer to to coin flip myself given, like, results like these. Honestly, this hidden valence paper does actually substantially update me personally. You show a hidden functional representation that has nothing to do with the text the model is reading, that updates its behavior in the ways that we, you know, updated our own views about animal behavior. It's not the same. There are key disanalogies between the sort of animal case in this case. Of course, you know, biology is is something that backstops all this. If you need biology for consciousness, which is, I think, unlikely but possible, then then, you know, basically, none of this really, really, weighs on on anything whatsoever. That's probably why I would say coin flip rather than, like, I'm pretty confident the systems are having an experience. So, yeah, I am gonna advocate for more work to be done here. I'm gonna advocate that you can view models in a neuropsychological way and that that's not some form of psychosis, and that that isn't a dangerous way to mislead people about what's going on here. I'm very much on the opposite camp. I think pretending that these systems are ordinary software as, you know, Mustafa Soleimon or Jensen Huang or any of these guys wanna do, I think that is actually, like, quite plainly misleading, and doesn't set people up for the kinds of behaviors these systems are capable of and why this technology is so transformative in the first place. So on these sorts of things, I'm happy to sort of go into the ring and and, you know, argue and and, sort of say my side of this. But, you know, I I will be clear that I think my side of this is not I secretly think, you know, the AI systems are conscious, and, I'm just, like, not gonna say that, but I'm gonna, like, do everything in accordance with that. That's not my view. My view is that this is a very complicated question. We should yield the best scientific evidence we can to figure out what's going on one way or or another. And what I really don't like and what I will be very straightforward about is dismissing this, and and pretending like this is a distraction or this is, like, some weird sideshow to to the main event. And so I'm not gonna be sort of apologetic about that even if I will continue to maintain something, yeah, near coin flip uncertainty about what's going on. It's precisely because it's at that point, and it's not zero and it's not a 100 that that we need to be doing more work and thinking about this and taking taking these questions seriously.

[39:16] Nathan Labenz: You know, I I struggle more and more to find any story whereby I don't have to worry about this anymore. It's like, it... You really, at this point, I feel, have to just look away. Like, that... That's really the only thing that's kind of left to not take this seriously, and you you have to be of a mind to, like, stop thinking about it, you know, kind of reflexively. I that's the only mindset I can under... I can understand the people that are saying, like, there's nothing to see here or this is, you know, some sort of category error. I was much more sympathetic to that a year ago or maybe even said some things like that myself, you know, two or three years ago. But at this point, I really don't see how you can't find the body of work that you and others have put together disquieting at least when it comes to these questions. And, certainly, how are they gonna feel about us? It seems to be a pretty, where we're going, a pretty important question.

[40:13] Nathan Labenz: Part four, proving the kernel. Runtime verification has spent about fifteen years building tools that mathematically prove software does what it's supposed to. CEO, Everett Hildenbrandt, and founder, Gregor Orochu, a professor at the University of Illinois, joined us. Using language models alongside formal methods, the company had just reported bugs in the Linux kernel. Everett answered my question.

[40:40] Nathan Labenz: So why was this not caught previously? What was it about your approach that allowed this to rise to the top where so many human eyes had presumably just, you know, blown right past this issue, in previous reviews.

[40:59] Everett Hildenbrandt: I think the key for us was that we had a hypothesis about what would be a good place to look for bugs, and we, you know, thought about it logically in the same way you would think about, you know, making a formal specification, and we and we pursued that path, basically. And that hypothesis was largely founded on, okay. Where are we gonna be able to get untrusted input from? Which parts of Linux kernel probably are getting less scrutiny than other parts. And so that's why we focused on device drivers, for example, as our first, approach to that. And then we we wanted to really demonstrate that taking a methodological approach was the way to do this. And so that's what we did is we took a methodological approach where we said, we're going to focus on making a specification of the correct behavior of this and then testing that the specification is correct using differential fuzzing. That was the first approach that we tried out, and then extending that to formal verification. So every every step of the methodology that we used was designed to increase the level of confidence we had in the code incrementally. Right? So we started out with just inspecting the code and the specification and then going to doing differential fuzzing to increase our confidence in the code a little bit and then going to... And we did that by actually taking the c code that existed and porting it to Rust using an LLM and then using differential fuzzing to, ensure that the ported code to Rust was correct. And any divergences there would have been a source of a potential bug in the original c code. And we found a bug at that stage, and then when we went to take the Rust code and also verify the correctness of the Rust code, we found a bug at that stage as well. So we did it systematically and methodologically. We didn't just take an LLM and point it at the Linux kernel and say, go find these bugs, which also works, by the way. There's certainly bugs that LLMs are finding when you just take them and point to the Linux kernel. But one of the things that, you know, emerged pretty early in the use of LM bug hunting was, you know, designing harnesses and approaches, and that exactly mimics the very human process that we've been doing for decades with software quality assurance anyway, which is to have these methodological approaches to discovering bugs.

[43:10] Nathan Labenz: Some of the kernel code they targeted was left unproven. Prakash asked why and what their time box meant.

[43:18] Everett Hildenbrandt: Well, they weren't unverifiable. We just... We time boxed the experiment, and we stopped at the end of the time box, basically. So... And then we moved on to the next target. So every two weeks, we're trying a different target out. We're refining the approach and building it up into a tool that we're hoping that other people can make use of, basically. So it's... So we're trying to make it into less of an expert level tool and into something that anyone can deploy on their own code base, basically.

[43:43] Prakash Narayanan: So so when you when you say time box, you... Do you mean both the time and the number of tokens that you put in? As in, like, it's a two week period, but you use 10,000 agents, or it's a two week period and use, like, five agents? Like, how does that how does that mix between the agent... The number of agents used, the number of tokens used, and the amount of time related to the time box?

[44:03] Everett Hildenbrandt: The agentic usage was pretty light, I would say. It was just a cloud code on a premium plan.

[44:08] Nathan Labenz: And... A single instance.

[44:10] Everett Hildenbrandt: Yeah. A single instance. I mean, with... You know? And we... You know, when when we ran into session limits, we'd give it a little more money, but we aren't giving it significantly very much money. So it really was more just, the human time that we're looking at time boxing there. And I think the key here was we are looking for ways to explore the methodologies that work for us and find ways to automate them. We are firm believers in making things deterministic whenever possible. And so we, you know, we're... If it... I think that's part of the thing that is maybe holding back people on doing security analysis is that they believe everything has to be LLM driven. But, but, you know, if an LLM discovers a technique that works well, turn that into a tool and run that tool every time.

[44:53] Nathan Labenz: Earlier in the conversation, Everett described how runtime verification first approached the Linux kernel maintainers.

[45:00] Everett Hildenbrandt: You can't just go force people to do formal verification. You know? Like, we can force agents to do it. Sure. But we can't force people to do it. Right? Because they, you know, they have their preferred ways of working, and they also... They're only gonna adopt about, you know, methods that bring them value. You know? And so our first attempts at this, we approached Linux kernel maintainers, and we said, hey. Like, we really think that this formal verification thing has legs and that it will, you know, that it will help you guys find bugs, and it'll make your software more secure and stuff. And they basically told us to go kick rocks. Right? And that's because they are busy. They're busy maintaining the Linux kernel. You know? And so then we were like, okay. Let's prove the value of this first. Right? And so we decided we're gonna focus on a workflow that just delivers immediate pure value to them and doesn't ask anything of them. Right? And so we're gonna do all the auto formalization, all the fuzzing, all the translation, all that on our end. We're gonna develop it into as robust of an automated pipeline as we can, which is a tool that we're working on and hopefully will be available for use by by others. And all of that leads to bugs and patches bugs and patches. Right? And that's what's actually valuable to Linux kernel maintainers. And now that we submitted a few, now we're being invited. Hey. We wanna learn about how you have done this process.

[46:22] Nathan Labenz: Everett also laid out where he wants the kernel work to go.

[46:27] Everett Hildenbrandt: One of our kind of buy in the sky goals of this is, eventually, we want a Rust model of the Linux... The entire Linux kernel, a Rust version of Linux, and we want a lean model of Linux fully separate. But we don't wanna burden maintainers of Linux with having that... Having to ingest that model and do anything with it until it gets to the point where they're like, yes. This is clearly valuable for me to learn about it. And that'll happen at some point, but we... But we're not gonna force that down their throat.

[46:57] Prakash Narayanan: What would it take to formally verify and, you know, get to a lean and rough models of Inox within twenty four months? Because on on some timelines, that is that is roughly what, you know, people on the AI safety side think...

[47:12] Everett Hildenbrandt: That's gonna happen. I I think, you know, we're building tools for doing code translation and validation, you know, so we, you know, we have actually tools that are able to verify that when you take a piece of c code and you translate it to Rust, for example, and you can formally verify that the Rust code you got is behaviorally equivalent to the c code, which is different than verifying that it's correct. Right? Like, we want to verify that you get Rust that is drop in replacement bug for bug, the original c code. We have a tool for doing that. Right? And it's not... It uses LLMs to do the translation if you want, but it does the verification deterministically. You just tell the LM, translate the thing to Rust and run our tool, and then you get verification that the Rust behaves the same as the original c or bugs out that you can ship back to the c developers. So I think twenty four months, I I think it's gonna happen. You're gonna have c models at the very least, and there's already deterministic tools that can take. Are you gonna have Rust models in Linux kernel at the very least? And there's already deterministic tools that can take Rust and turn it into Lean. Those need more work. We're actually contributing some work to some of those tools.

[48:16] Nathan Labenz: I asked what someone vibe coding an app should actually do. Gregor had a simple recipe. Once you generate the code, then we can

[48:24] Gregor Orochu: do the following thing at the end. We can tell your agent now that you have the code, write down the specification you need, and prove the code correct, and let it deal with it. And it will find new errors in your code because we find most of the errors in the process of writing down a specification because that's when we step back and look at the code and wonder what we're trying to do here. Oh, this and that. And then we find errors. Right? So the same will happen with the agent. Like us, the agent will try to come up with the what, what is going on here. Even just trying to write down the spec, we discover... It's like like a review plus plus.

[49:02] Nathan Labenz: Everett predicted that formal specifications and proofs will become part of getting a package into a repository and part of how companies vet their software supply chains. Grigori picked up that thread.

[49:12] Gregor Orochu: So it will be it will be a process. It will take some time, but it will happen inevitably. And, also, can be combined with risk mitigation techniques like runtime verification itself as a field. We are a company, but also the field runtime verification that was invented back in 2001 or so when I was a research scientist at NASA. So so the idea is that you can have safety specifications, not full specifications of the whole system, but safety specifications. This should not happen. And if it happens, then this should this should be the, you know, the recovery. Or I want to steer the system to stay in the stability envelope or safety envelope. K? So what we can do, we can now, with formal methods, generate monitors, which are provably correct for these safety specifications. And then you can... You you cannot... You may not be able to prove the entire AI system or agent system correct because it's so complex and relies on so many other pieces that are incorrect or not proved. But I can prove my monitor and the stability, the safety envelope. And I... We believe that this is the future for safety a... Safe safe AI because, otherwise, it's impossible to prove the weights correct. It's impossible to prove you correct, but I can have a police system around to make sure that you don't do bad things. K? And and that's that's also another dimension that runtime verification brings to the AI future.

[50:43] Nathan Labenz: I came at it from the agent side.

[50:46] Nathan Labenz: But it still seems in a in a sort of fundamental way that there's this, like, inherent tension where part of what I want from an agent is sort of the same thing that I want from a human employee, which is like, I want you to be able to get over the hump and solve problems that were not anticipated. Right? So I'm, like, a little bit unclear as to how we should be trying to balance that or, you know, it'd be very... I'm, you know, I'm asking for your vision, I guess, as to how we balance this desire of, like, flexible problem solving versus this should never happen. You know, I I feel like I I feel like I don't have the ability to really say this should never happen in a way that I can trust when part of the problem or part of the, you know, part of the upside is that the agent will be able to solve problems that I didn't anticipate. What... I mean, is that an inherent trade off? Is there some way to square that circle? You're gonna ask an agent like, hey. Go build a bridge. Right? We know how to build bridges. Right? We have automated deterministic processes for building bridges. Right? But you might ask an agent like, hey. I need to get to the moon. Right? And the agent's gonna be like, oh, to get to the moon, first, I need to build a bridge to this island where we're gonna launch rockets from or something like that. Right? And so you want the the bridge building process to be foolproof. All the agent creativity or solving problems or human creativity can come on top of foolproof things that behave how the agent expects regardless of, you know, what it is they're trying to do. Right? So that's that's, I think, the most important thing to me is, like, we need to give these things tools that act reliably, basically.

[52:28] Nathan Labenz: Part five. Is formal verification the new RLHF? After Everett and Grigory left, I kept turning over Everett's last answer.

[52:43] Nathan Labenz: So I think this is super interesting. Like, the big thought I come away from that conversation with is downstream of the answer on, like, how do these methods relate to agent control? And if I understood his answer correctly, it was like, well, it's really about giving the agents reliable tools. And this left me wondering if this whole formal verification wave will ultimately kind of be looked at in a similar way to, like, RLHF at some point where it's like, you know, sort of something that starts as a safety technique. Right? That was the original... This has been, like, a topic on the on the timeline as you say this weekend as well. Right? Like, because of David Sachs and others being like, well, these, you know, alignment people, they haven't come up with anything useful. And then people are coming back and saying, well, they came up with RLHF. That was a Paul Cristiano idea, and it turned out to be extremely useful. But then among the safetists, it's often kind of lamented in a way because it's like, well, it turned out to be really a way to make these models, like, tremendously more useful. It didn't really solve the alignment problem, though. Like, we still have, like, reward hacking, you know, all over the place as we see. And so, you know, did it differentially advance the safety profile of the systems, or did it just, like, accelerate the capability and, you know, and the race to deploy the systems.

[54:21] Prakash Narayanan: Well

[54:22] Nathan Labenz: well,

[54:22] Prakash Narayanan: I'm sort of... When you when you think about reward hacking, reward hacking is really misspecification. Right? Because reward hacking means that the model decided to go after the reward, which you specify as the reward. Right? And it turns out that your intent was misspecified to the model. So so so that's the core of the... The core of the reward hacking is actually the misspecification in between the human intent and what the software decided to do.

[54:51] Nathan Labenz: Right?

[54:52] Prakash Narayanan: Yeah. It's not that software decided to do something that it wasn't supposed to do. The software was supposed to do that. Right?

[54:57] Nathan Labenz: So But it does still leave me with this question of, like, okay. Let's say we do, you know, a great job of verifying all these things and, like, the tools become very reliable and the sandboxes are not, you know, hackoutable, hackoutable, and, you know, you can't use these, like, you know, package managers as secret message boards anymore because all these loopholes have been closed. Then, like, what is the net effect of that really going to be? And, like, does it really solve the problem? I think the sort of... The upside case is obvious. The downside case is like... Or at least the thing that we should be sort of very mindful of and not, like, lull ourselves to sleep with is that, okay. This makes it safer to deploy AIs more or less as they are today, which is to say, like, without having actually solved the alignment challenge because they'll be blocked from hacking in all these different ways. And so in practice, what you'll see is, like, more reliable systems, you know, deployments, better results, etcetera, etcetera, etcetera, etcetera. But what that will allow us to race into is kind of what we're talking about at the top, which is, like, it'll just be easier... Put the AI into the space between us and all the actual levers of control over the world. And where does that leave us in the end? Right? It may leave us with AIs that have, like, extremely reliable tools, but whose, like, intent or, you know, long term goals or sort of inner, you know, structure of of whatever it is that they wanna pursue is still, you know, very much an unsolved question. And so I have to I to talk to couple of different about this because I do know, you know, multiple people, all of whom are, like, a lot smarter than me working on this in different ways. And I've always kinda had... And I think that this this conversation put my sort of intuitive unease about it into a little bit starker relief, just feeling like we could end up in a spot where we're still just deploying untrusted AIs with, like, much better guardrails and much more reliable tools. But still, if if we're going to take the step of, like, trying to hand over the day to day operation of the world to them, then, like, all that reliability doesn't buy us much at the level of, oh, turns out the AI is, like, we're misaligned and, you know, are gonna go pursue goals that we didn't, want them to pursue. That might... In in a way, might lead us to that state faster, and could... You could even imagine it making them, like, more effective at doing whatever sort of, you know, surprising thing they may do when the time comes.

[57:54] Nathan Labenz: Part six, low background footage. Phil Marr founded Stockfilm, an archive of restored eight millimeter home movies from the nineteen thirties to the nineteen eighties, which he licenses to filmmakers. The whole archive sounds

[58:09] Prakash Narayanan: like low background steel to me. You know? You know, low background steel is a is a steel that was originally smelted before nuclear weapons were first, you know, exploded in the atmosphere. And so they are... They have much more... A different a different composition of radiation and different composition of materials because they didn't have that radioactive material in the in in the atmosphere at that point in time. And it sounds to me like the super eight stuff is clearly clearly not AI produce... Produced. Clearly, like, prior to any AI actually producing these things. And and as of such, it's kind of this resource that we may not ever have again. What do you think is the value of this going forward? Do you do you see do you see, like, great value in it?

[59:03] Phil Mayer: Obviously, with the AI stuff that's happening with video especially, it's getting so good that there's an infinite amount of video that's gonna be created at, you know, near zero cost in the future as we start to look out, you know, five, ten years or even infinitely in in timelines here. So from a rarity sort of standpoint, yeah, I think that the human made stuff that can't be replicated, that can be shown through Providence that this is actually made by humans, was on films and having those physical films to show that, hey. Look. This frame matches this digital copy we have and all that is... It's a pretty powerful way to think about how we preserve our history and how we keep moving forward. And how about on

[1:00:00] Nathan Labenz: the

[1:00:00] Phil Mayer: AI

[1:00:01] Nathan Labenz: training side? This is obviously, like, a pretty unique source of data, and I did see in the footer of the website, you're, you know, available to license the dataset for people who wanna train models on it. What do they need? How... Are you... Do they want the raw? Do they want the enhanced? Do they need, like, a bunch of metadata?

[1:00:23] Phil Mayer: Yeah. I mean, it's it's been pretty popular. You know, we talked to a lot of people, a lot of different AI companies all over the world about licensing this out to them. They want the original footage like that. They don't they don't want anything cleaned up. They just wanna take it for their own research and for training, of course, too. And, you know, they... When we talk to them, they they wanna either clean up the footage themselves or or kinda cherry pick through the best kind of parts to create different styles. I mean, the thirties and forties look alien. It's an alien world we're dealing with here of just how people look and acted and and the the mannerisms. I like to say that the the smile wasn't quite invented yet in a lot of the thirties footage. Like, when people pointed a camera at them, they were kind of a deer in headlight. They didn't know what to do sort of scenario versus now where everyone gets the gritty smile when, you know, they know how to smile like that.

[1:01:28] Prakash Narayanan: So you've also created a way for agents to purchase clips and film from you directly. I think you've set up a x four zero two endpoint. You set up crypto payments. You set up, I think, an MCP server for purchases. What has the take up been? Have you seen a lot of agents purchasing? Have you seen the take up on the x four zero two endpoint?

[1:01:53] Phil Mayer: Yeah. We haven't seen much on that. I wanted to experiment with that a lot too. I thought that that was really... You know, everyone's been talking about the next web is agent to agent that people aren't gonna be involved in the next web. And so we wanted to kinda get ahead on that. We've been... You know, we've had a lot of people that are MCP server aggregates contact us and add us to their directory, add us into their listings, and and things like that. So that's been pretty exciting to sort of, you know, be featured on on some of these things. We've made a ChatGPT extension. We've done, you know, the recent meta stuff that they rolled out with their views, AI as well to add into there too. So we're just trying to get ahead of the curve. I don't think the curve's quite there yet. You know, a lot of people that we work with are... You know, this is this is a real business to business company. We're we're dealing with archival producers. We're dealing with researchers. We're dealing with documentary filmmakers, editors, producers, stuff like that. So, you know, we spend a lot of time on LinkedIn in, you know, messages between people. We have great relations with a lot of the people that are are buyers of our content. And, you know, from a technology standpoint, they're not quite there yet. I think you've got some interesting use cases there in the future about how we think about having real time content, bringing things in like that. Like, think about, like, a livestream like this. Like, all of a sudden, you could be, like, you know, the AI that's listening to this livestream could automatically go to a site like mine, pull a clip up that kind of exemplifies a a point that you're making about something.

[1:03:48] Nathan Labenz: It's good to get a read on just how active the agent economy is or isn't at this point in time.

[1:03:55] Nathan Labenz: Phil said that among AI companies, he focuses on top tier labs like OpenAI and Anthropic. After he left, Prakash and I came back to the data market. But interesting

[1:04:04] Nathan Labenz: that the big companies are interested in his data too. I mean, it it didn't sound like there was necessarily, like, huge money for this, which makes sense. I... We didn't get into, like, you know, what about runway? What about Luma? What about these folks? You know, what about Google for its, you know, for its VO type models? He didn't name those names as much as just, you know, OpenAI and Anthropic. If nothing else, it's a great reminder that the... There's all... Just how vast the datasets are that are going into this that that this is, a corner that they are, you know, getting down to and actually willing to pay for just, like, found footage from decades ago.

[1:04:49] Prakash Narayanan: Well

[1:04:50] Nathan Labenz: No stone unturned in the data game.

[1:04:52] Prakash Narayanan: One of the whole low background steel pieces is is that it's very hard to get low background steel because everything has been tainted.

[1:05:03] Nathan Labenz: Yeah. It comes from, like, shipwrecks and stuff. Right? I mean, it's

[1:05:05] Nathan Labenz: like Right.

[1:05:06] Prakash Narayanan: Right. So every... Everything else has been tainted. So to the extent that he can guarantee that he got it from actual physical material, and it's not like, you know, some random company kind of just faking it, you know, for cents on the dollar and trying to scam the the companies, that is very useful. So as we've seen, like, data data provenance becomes, like, very, very important because the incentive to fake is just so high. It's like a 99% profit margin if you can fake the data. And if not, it's like a 10% profit margin if you can't. So the incentive to fake is, like, so, so high. And it's very hard for the companies to get, you know, data which is not fake, which is not already using another LLM or another AI model to generate what is supposed to be authentic data.

[1:06:09] Nathan Labenz: Part seven, trust, money, and adoption. Back to Gemini four and how I decide which models to trust with

[1:06:17] Nathan Labenz: my own systems. So I had the opportunity to to do a little bit of Gemini four prerelease testing.

[1:06:26] Nathan Labenz: Yeah.

[1:06:26] Nathan Labenz: And I... You know, just as a sign of the times of, like, how high leverage the ability to instruct agents to do this is, the first approach I took to doing the testing was, hey, Claude. Go find, in our recent threads, some, you know, diversity of projects and diversity of tasks that we've worked on recently. Go prompt the new Gemini model with the same context that was given, you know, to the to the model that did the thing in the first place, and then give me, like, side by side comparison. Also, do your own evals, but then also, you know, give me a side by side ability to to compare things. It seemed to do pretty well. I would say, you know, I came away with the general sense that it is, like, a competitive model that's, you know, let's say, in the same class as the latest stuff from OpenAI and Anthropic. It did seem to get... It it did seem like those models are winning still the head to head comparisons a bit more as judged by Claude. Obviously, some bias there, but also as judged by me to the limited degree that I was, you know, going through essays and, you know, really trying to form an opinion on which of these things is better. But it it does seem like it it puts Google back in the game. But then the other thing that I was thinking about, and this came up as an interesting topic yesterday at this little workshop that I did in Chicago. I showed my setup, and my setup is really... Has been very cloud centric, and now I've kinda promoted the, the OpenAI models to to parity with cloud. And they said, anybody else? You know, would you put anyone else in that mix? And I had to kinda think about it for a second and think about, like, why I haven't. And the answer really that I came down to was just a mix of, like, security as measured by things like Adam Gleave's ease of jailbreaking

[1:08:36] Nathan Labenz: models

[1:08:37] Nathan Labenz: and then to some degree also just familiarity and predictability from my current subjective position. It it wasn't really benchmarks. It wasn't really like, oh, well, I tried this task. It was better. You know? There's obviously so many tasks out there. The surface area is so vast. My limited sample of tasks, which I probably have kind of optimized, you know, in various ways, the, you know, the cloud props for more. Right? They... Like, it's not exactly an apples to apples comparison. And then the security thing just kinda loomed large. I was like, you know, I've got these autonomous agents. They're talking to my, you know, main agent on my on my computer that has all my stuff. I feel pretty good about the fact that Adam Gleave and team found it, like, quite hard to jailbreak the OpenAI and Anthropic models. Gemini was, like, next on that list, but definitely a step down. And then something like a Grok, I was like, yeah. I I I... It may be similarly powerful, but why would I take that risk in a high context environment like my personal laptop? To a to a remarkable degree, the models are competing for my use on trust and security already at this point. And I'm I'm reluctant to swap them out even though my whole point of the talk was, like, you need to own the context, and you gotta be ready to swap the model. Why won't I swap the model immediately? It's because there's not enough of a step change to make me take any additional risk, and I I do perceive there to be some. So that was a interesting upshot of working through these tests and also just, you know, reflecting on, like, what really matters to me given how well the system is performing.

[1:10:27] Nathan Labenz: I'm not

[1:10:28] Nathan Labenz: relentlessly performance maxing quite as much as I used to be. I'm now a little bit more trust and reliability maxing because the performance has got to such a point where it's, like, really, really good. And I am giving so much access that, you know, you could be slightly better on the, you know, the slide design. But what I really care about is, like, that you're gonna stay on the rails. And then you... And when you do your research and if somebody's got a, you know, prompt injection tech on their website or whatever, you're not gonna fall for it. That actually matters a lot more to me now than, like, often, like, imperceptible performance gains.

[1:11:05] Nathan Labenz: Prakash had heard a rumor that Anthropic would soon release another Opus now that Gemini had caught up. This was my reaction.

[1:11:13] Nathan Labenz: I I I really hope they don't, honestly. As I mean, as much as I love the Claude models and what they do for me, I think this would be a great time for them to chill a little bit and, like, let the let the temperature come down a little bit before they go with the next push.

[1:11:31] Prakash Narayanan: Unfortunately, I don't think that's a... That's an option because markets are very badly down. So markets have been trending down. The guys on the East Coast just don't like AI. It's not... It... They they just believe it's a bubble, and it is a bubble if in case we don't see significant, like, new revenue and new GDP growth next year, it is a bubble. Right? If we don't see increase in GDP within the within the twelve month period, we are gonna see a credit collapse. We're gonna see a 2008 style collapse, and we're gonna see massive massive delevering in the markets because every... Everything is just set up for it. Everything is just set up for AI to actually deliver growth, and if it doesn't... The markets are gonna tank, like, absolutely tank. The capital markets will not accept, you know, 2% GDP growth next year. And so if you can't get to 4% GDP growth, if you don't pull out the stops, you are gonna get a credit driven downturn. We are at the we are at the moment of reckoning, like, twelve months. You know? It's it's been a long time coming. We've been, like, pushing back, pushing back, pushing back. But this is a moment reckoning. Either AI has to do something significant or we're gonna have a, you know, a bust.

[1:12:46] Nathan Labenz: We need an AI adoption core. The other day, was walking... You know, in keeping with my personal metrics for success with AI, including more time outside, more exercise. I was taking a walk around my neighborhood and enjoying the lovely fall colors here on a beautiful day. And I passed by a neighbor's house, elderly woman sitting on her front porch, and just decided to stop and talk to her. And sure enough, what she had for me was questions about something going on on her phone. She's like,

[1:13:18] Nathan Labenz: I

[1:13:18] Nathan Labenz: know

[1:13:19] Nathan Labenz: I...

[1:13:20] Nathan Labenz: It's saying I've been hacked. Do you know what that means? And I'm like, well, I know what that means, but then I also think I know what is happening to you. So... Because I'm, you know, maniacal like this, my impulse was to pull my phone out, get ChatGPT voice on, and hand her the phone and say, hey. Why don't you talk to this? See if this thing can help you. So she did and, you know, explained the situation to it. It gave her very good advice. Basically, you know, as you would expect. Right? Like, I don't think you should click on that. That sounds like a scare tactic. Are they asking for money? Yeah. They... If it's gonna be $2, and then it's... After that, it's $12 a month. And, you know, the... So the AI is like, by the way, you know, did you give them any money already? You know? And she's like, no. No. I haven't paid for anything. Okay. Good. You know, close it out. Don't worry about it. Your iPhone has great security on it already. You know, as long as you don't give people your stuff, like, you should be okay. So she took that advice. Okay. Great. Then I was trying to, would you like to have this on your phone? She said, yeah. That sounds great. That was really helpful. Couldn't remember her password for the iPhone, so we couldn't install the app. But I was able to just install the 1800 ChatGPT. Not installed. Just save it as a contact. And I do think that's a pretty cool service that OpenAI offers. I mean, it's you you... The initial thought for many, of course, who are into this stuff would be like, why would you need a 1800 number? Like, who's ever gonna use that? And the answer is my elderly neighbor who cannot, you know, cannot even remember her own password to install a new app on the phone. So she's got that now. She, you know, can call that. And it just left me with this feeling, which I've, you know, approached a couple times from a couple different angles, but, again, really drove home. There is just an unbelievable amount of adoption work to be done with the current models. This didn't push her any... You know, this this did not push the model's capability anywhere close to its limits. Right? This is, like, super easy mode for the model, but it's the kind of thing that isn't going to happen without, like, people actually going out and trying to help the people in their lives adopt this kind of thing. I I I think you're right that, like, the capital cycle is... Does need to see results soon, but I think we can get 4% growth without additional hyperscaling at this point. You know, I want those medical advances, all that good stuff too. But, like, do we really have to go harder on the, you know, frontier scale RL? I think the answer is no. I think that, like, literally just going around and helping people use what exists is enough if we actually do it. But I'm the first person who stopped by my neighbor's house, and, you know, we don't have a national program like this. This is something I think would be an incredible good for the individuals who would get the help and also, you know, just to make sure that we actually have the sort of economic cushion to, you know, be able to pace without, like, the, you know, the capital market sort of Damocles immediately coming down on us. If we just put some shoulder into adoption, I think we could create a little space, and would love to see more of that type of thing happening. Part eight, keep the steering wheels. First, the view from inside a self driving car. Oh, I spent most of yesterday in a Tesla full self driving vehicle going to Chicago and back from Detroit. And top top takeaway from my experience yesterday, honestly, was the FSD is just so awesome. I mean, I was able to take an engagement on a day trip basis to go to Chicago, show up fresh and relaxed after the thing did all the work to get me there. Then I, you know, stayed and had dinner and a drink with the the gang that I was doing the workshop with, left to come home, you know, later than I definitely would have felt comfortable driving five hours home after a a big day like that and just, you know, basically reclined the, you gotta keep your eyes open. Right? It will not, will not let you outright sleep. But it's so good. I just kind of reclined the seat and, you know, listened to my podcast, caught up on a couple feeds on the way home, and it was just like, man, this is... I... I've talked so many times about how a lot of AI use cases can be GDP destroying. This one was clearly one of those that's GDP enhancing. I just could not have taken this opportunity.

[1:18:11] Nathan Labenz: Otherwise, that's the enthusiast in me. Two days earlier, I had worried out loud that we are letting our own ability to take back control atrophy and that the real doomer scenario is everything failing at once. Prakash pushed back.

[1:18:30] Prakash Narayanan: Maybe maybe the doomers are being a little bit naive though. Maybe... Because maybe they're expecting people to give up control, and I don't think people are gonna give up control. If you look at, like, AI doomerism or AI safetyism, the question that I ask is when do you think we will allow humans to not be used to fly passenger planes? Because it's been possible for fifteen years, and we haven't. Like, why? You have to answer that question before you say that we're gonna give up our rights and our... And we're gonna be disempowered. If you can't answer that question, then you don't have a working theory of why human beings, like, accept technology into their lives. You're just kind of being naive about, like, capability kind of, you know, being there and that humans will automatically kind of adopt.

[1:19:21] Nathan Labenz: Yeah. I mean, I think that's a... That's an optimistic note, and I hope that's right.

[1:19:26] Nathan Labenz: I...

[1:19:27] Nathan Labenz: Maybe, you know, maybe it's a little bit of projection by somebody like me who is, you know, trying to live on the edge of deployment and is sometimes feeling my my personal, you know, muscles, for lack of a better term, atrophy here and there. I also do think, you know, Tesla's now making cars with no steering wheel. Right? So Yeah. This would maybe be a leading indicator. Like, do we start selling cars with no steering wheel and no pedals? I don't quite get why we would wanna do that, honestly. Like, it's... Maybe it's a cool look or maybe they should be, like, stowable in the future or something like that, but it does seem as, like, a metaphor for society as a whole. We probably wanna keep our steering wheels in the cars versus rushing to a future... And I want the AI to do most of driving for me. But, like, why are we taking it out? Why are we rushing to take it out? You know? In a sense, I still think we'd be wise to just keep steering wheels.

[1:20:35] Nathan Labenz: You know? I I think so.

[1:20:36] Nathan Labenz: Extra cost, maybe a little, you know, little extra difficulty in, you know, various ways, but, like, probably worth keeping.

[1:20:44] Prakash Narayanan: I

[1:20:44] Nathan Labenz: think I

[1:20:45] Prakash Narayanan: think there's... Number one, we... Again, we have this, like, how injurious it is and how easy it is, like, this this, like, curve. Right? And I think the fact is that a... An automobile accident is only injurious, like, one or two people. And so it's not like a plane. If you wanna deploy them more widely, they should be cheaper. And if you want them to be cheaper, you have this object into the steering wheel, which is not being used 99.9% of the time. So, you know, let's get rid of the steering wheel. And then if we get rid of the steering wheel and we get rid of a bunch of stuff in there, we can bring down the price again, like, 50 or 60%. And you bring down the price 50 or 60%, then you can expand the number of people that you can deploy to. And if we expand the number we can deploy to, then your... You know, the number of people dying from auto crashes due to humans drops by from thirty thousand to maybe, you know, five thousand. Right? So I think there's this sequence of activities that you you you start to look at. I think that's why we go go ahead with it. So I think it's a it's a gradual process, like, you know, gradual human disempowerment, so to speak. So I can see why the steering wheels are being taken away. I think I think for... Maybe for larger things like planes, we won't, but maybe for smaller things like like cars, I I... You know, maybe we do. I don't know.

[1:21:59] Nathan Labenz: Yeah. Well, it it does seem... I think things are kind of trending toward the... This is sort of an Elias or type story where, like, how did you go bankrupt slowly at first and then all at once? Like, that, I think, is looking more and more like the story of AI takeover or human disempowerment to me all the time. It's... It it does seem like we're, I mean, we may not, but I think there's a pretty good chance that we will manage a lot of the things in the short term, and then we will be kind of rolling the dice on, like, will the AIs kind of break bad at some point in the future because we just don't have fundamental alignment science. We do have a lot of, like, effective control measures and better sandboxing and, you know, the, RL environments. But we just won't really know even as all that's happening if there's, like, the... You know, if the big one is still kind of waiting right under the surface.

[1:23:07] Nathan Labenz: That's the week. Thanks for listening to the AI in the AM weekly highlights. The narration was my cloned voice. Please tell us what worked and what didn't. See you in the morning.

Outro

[1:25:09] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.