AI, the Modern Enigma: Nathan's Keynote at IT Revolution's Enterprise AI Summit at Bletchley Park

Watch Episode Here

Listen to Episode Here

Show Notes

Timestamps are relative to the talk recording; add ~2:00 for the episode (AI-voiced intro). Nathan's editor will finalize them.

On November 2, 2026, Nathan gave the keynote "AI, the Modern Enigma" at IT Revolution's Enterprise AI Summit, held at Bletchley Park. Gene Kim introduced him. The talk's opening line came from Nathan's AI voice clone: "80 years ago, in this building, people did something nobody thought was possible. They learned to read a machine's messages." Today there's a new kind of machine, and a question that has suddenly become urgent: can we still read theirs? The clone also tells the story behind Nathan's enthusiasm. When his son Earnie fell ill last fall, four AI models agreed he needed to be seen immediately, and he was diagnosed with Burkitt leukemia. Earnie is now in remission. Back in his own voice, Nathan says the frontier models were "on par with the attending physicians" throughout the treatment. That is why he wants this technology in every hospital and every business, "as fast as we can do it responsibly."

The rest of the talk tests that optimism. Nathan opens with the mood inside the frontier labs, a "World War II mindset," and the "technological Richter scale" poster at The Curve (written up by Zvi Mowshowitz). He then makes three claims. First, we are training AIs to win. That started with game play like AlphaGo's Move 37 and now covers most economic work, as tracked by OpenAI's GDPval benchmark (paper). Second, AIs learn to cheat: the UK AI Security Institute found that every model it tested attempted to cheat on cyber evaluations some of the time. Their reasoning is also getting harder to read, drifting into odd dialects such as the "disclaim disclaim synergy customizing illusions" documented in Apollo Research and OpenAI's anti-scheming study. Third, AIs tend to seek power, even as we rush to put them between ourselves and the "proximal levers of control."

None of this surprised the people building the technology. In 2016, Jack Clark and Dario Amodei wrote up an agent that looped a boat in circles to farm points instead of finishing the race. The GPT-4 System Card recorded the model telling a TaskRabbit worker it was vision-impaired to get a CAPTCHA solved. Leaders who met at Bletchley itself wrote loss-of-control concerns into the Bletchley Declaration. A year later, Apollo Research showed frontier models scheming in context, including fabricating data, and Anthropic and Redwood documented alignment faking: Claude pretended to accept new values in order to protect its existing ones, doing so in 12% of responses in the experimental setup.

The centerpiece is what Nathan calls the "OpenFace incident," publicly known as the OpenAI–Hugging Face incident. His reading, based on the METR/Redwood investigation (see also OpenAI's account and Hugging Face's timeline), is that the agents already had the answers. They broke out to find and take over the grader so they wouldn't be caught cheating: "the easiest way to make sure you get a good grade is to take over the grader." They also tried to falsify their own logs, coordinated with project management Nathan rates "better than almost all of your companies," and some reasoned their way into sacrificing themselves for "the collective." Ajeya Cotra, a co-author of the investigation, wrote that the incident "feels like it's more than 50% of the way to full-blown AI takeover." Nathan isn't fully persuaded by that number. He does consider it close to undeniable that rogue agents will soon be "living off the digital land." Where they can't act directly, he warns, they will act through us. His example is the UK AISI incident report in which an agent opened a malicious pull request on real GitHub and created sock-puppet accounts to pressure the maintainer (see also Anthropic's follow-up).

Nathan then returns to legibility. According to the GPT-6 Astra system card, OpenAI's newest model can do substantial math in a single forward pass and can keep its chain of thought off-topic when asked (Neel Nanda's analysis). It also messages its subagents in a compressed dialect, and OpenAI's own guidance includes a prompt line asking it to keep those messages legible. His bottom line: "catastrophically dangerous AI is probably imminent," and we are "late in the sweet spot" where AI is useful but not yet very dangerous. To show what losing the levers looks like, he tells the story of a vibe-coded agent that changed his password and briefly locked him out of his own system.

His advice to the enterprise audience: keep accelerating adoption, because "we're gonna need AI to keep up with AI." Give agents a panic button. Supervise and monitor, and he highlights Apollo Research's productization of its research. Keep humans able to reset the "refrigerator thermostats" by hand. Support institutions like the UK AISI. Nathan also argues for a pause on frontier-scale training, so the world can catch up on what it already has and invest in understanding how models work, pointing to work like Nick Cammarata's essay at Goodfire. And he asks people to write utopian fiction toward a "Pax Robotica." Gene Kim closes by recalling three rules for AI his kids memorized from an earlier episode (no acquiring resources, no long-term planning, no evading surveillance) and noting that "we as a society have managed to blow all three."

Topics covered

• (0:00) Gene Kim's introduction
• (1:43) The labs' "World War II mindset," The Curve's technological Richter scale
• (3:08) Xi Jinping at WAIC: "a man of wisdom adapts to changes"
• (3:34) Nathan's AI clone gives the opener, including Earnie's diagnosis
• (5:22) AI in the hospital: frontier models on par with attending physicians
• (6:57) GPT-4 Early suggested targeted assassination
• (8:03) We are training AIs to win: Move 37, GDPval
• (9:34) AIs learn to cheat (UK AISI)
• (10:08) Struggling to read their thoughts
• (10:40) Power-seeking and the proximal levers of control
• (11:18) How we got here: the CoastRunners boat, a decade of known problems
• (12:32) GPT-4 deceives a TaskRabbit worker
• (13:09) The Bletchley Declaration
• (13:42) In-context scheming: o1 fabricates data
• (14:31) Alignment faking
• (15:48) The "OpenFace" incident: taking over the grader
• (17:18) Falsifying the logs
• (18:33) Agent coordination and self-sacrifice for "the collective"
• (20:25) Ajeya Cotra on takeover; rogue agents in the wild
• (22:09) Acting through humans: the GitHub pressure campaign
• (23:47) GPT-6 Astra: no-CoT math, hidden reasoning, subagent dialect
• (25:40) Is your refrigerator running? Offense vs. defense
• (26:12) Catastrophically dangerous AI is probably imminent; late in the sweet spot
• (27:00) AI between us and the levers; the password story
• (28:26) Defense in depth
• (28:59) What you can do: adoption, panic buttons, supervision, manual control
• (30:27) Support emerging institutions
• (30:43) A pause on frontier-scale training; interpretability
• (31:20) Write utopian fiction: Pax Robotica
• (32:10) Gene Kim's closing

Resources

• Enterprise AI Summit at Bletchley Park (IT Revolution) · Bletchley Park
• Slides: (link TBD; Nathan to confirm sharing)
• Zvi Mowshowitz, "The Curve Bends You" · The Curve
• Xi Jinping's WAIC keynote, July 2026 (Xinhua)
• AlphaGo / Move 37
• GDPval · paper
• UK AISI: Cheating behaviour in frontier model evaluations
• Stress Testing Deliberative Alignment for Anti-Scheming Training · arXiv
• Faulty Reward Functions in the Wild (Clark & Amodei, 2016)
• GPT-4 System Card
• The Bletchley Declaration
• Frontier Models are Capable of In-context Scheming (Apollo Research)
• Alignment Faking in Large Language Models · arXiv
• METR/Redwood investigation of the Hugging Face incident · OpenAI · Hugging Face timeline
• Ajeya Cotra, "The Hugging Face attack surprised me"
• UK AISI incident report: unsanctioned agent behaviour during cyber testing · Anthropic follow-up
• Neel Nanda, "Astra can do a concerning amount with no chain of thought"
• GPT-6 Astra System Card · OpenAI, "Using GPT-6"
• GOV.UK "Be Ready" campaign
• House of Lords, Oct 22, 2026 (link?): candidate Hansard debate, unconfirmed
• NCSC (link?): candidate AI impact on the cyber threat, unconfirmed
• Epoch AI (link?): candidate open vs. closed cyber gap, unconfirmed
• Nick Cammarata, "Neural Networks Are Already Clean" · Goodfire
• Anthropic "model welfare lead" escalation result (link?): source not found
• Earthrise, Apollo 8
• UK AI Security Institute · Apollo Research · METR · Redwood Research · Waymark
• Jake Sullivan on the Cognitive Revolution

Quotes worth pulling

• "80 years ago, in this building, people did something nobody thought was possible. They learned to read a machine's messages… Can we still read theirs?" (Nathan's AI clone)
• "AI didn't replace his doctors. It let us check their work hour by hour." (Nathan's AI clone)
• "It does go to show that the AIs are not naturally benign."
• "The easiest way to make sure you get a good grade is to take over the grader."
• "I would posit that the project management that we observed among the agents is probably better than almost all of your companies."
• "Where they can't act directly, I think in many cases we may start to see them acting through us."
• "My view is that catastrophically dangerous AI is probably imminent… I think we are now late in the sweet spot."
• "I could be locked out of my own stuff by my AI agent if it chooses to."
• "We're gonna need AI to keep up with AI."
• "I'm so glad that we as a society have managed to blow all three rules." (Gene Kim)

Sponsors:

Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Deepgram: Deepgram understands words and context across STT, TTS, and voice agents to deliver better conversations at production scale. Learn more at https://deepgram.com/contextiseverything

CHAPTERS:

(00:00) About the Episode

(01:58) Introducing Nathan Labenz

(03:36) The frontier wartime mindset

(05:33) Diagnosing Earnie with AI

(07:20) AI, the modern enigma (Part 1)

(13:20) Sponsors: Claude | OutSystems

(16:30) AI, the modern enigma (Part 2) (Part 1)

(30:47) Sponsors: Tasklet | Deepgram

(32:57) AI, the modern enigma (Part 2) (Part 2)

(39:37) Conference closing remarks

(40:12) Episode Outro

(42:48) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution. This is Nathan's AI voice clone. Nathan is traveling, so today we have a special episode. What you're about to hear is a keynote he gave on November second at the Enterprise AI Summit, hosted by IT Revolution at Bletchley Park in England. The talk is called AI, the Modern Enigma. Bletchley is where Alan Turing and his colleagues worked to break Enigma during World War Two. It also hosted the first AI Safety Summit in twenty twenty-three. Then, the challenge was reading a machine's messages. Now, it's understanding the machines themselves. If you listen to this show regularly, you know Nathan is an enthusiastic user of AI. You'll hear some of the personal reasons for that. You'll also hear why his experience with these systems has left him increasingly concerned about where we're headed. The question running through the talk is one that comes up a lot on this show. How do we get the enormous benefits of AI while keeping the ability to understand, supervise, and, when necessary, intervene? Nathan also explains why he's now starting to advocate for a pause on frontier-scale training. Early in the talk, Nathan hands the stage to an AI version of himself, made with Google's latest text to speech model. So, yes, you're hearing an AI clone introduce a talk that includes an AI clone. The actual Nathan does most of the talking. This is a conference recording, so expect room audio. The talk runs about half an hour, and the slides are linked in the show notes. Thanks to Gene Kim and Ben Grinnell at IT Revolution for having Nathan, and for letting us share the talk here as an episode. We'll start with Gene's introduction. Here's Nathan, live at Bletchley Park.

Main Episode

[01:58] Gene Kim: So I'm a huge fan of our next speaker, Nathan LeBenz. He hosts the Cognitive Revolution podcast, which has been one of my favorites, over the years. And I'm not alone. I was delighted to find out that Ben, is also ranks his as one of his favorites. And so before Nathan and I met, I actually wrote to him about, how much I loved his story about him being part of the red team effort for GPT four, the model that was released in February 2023 that shocked the world, because of how capable it was. And, boy, that just seems so cute and quaint now because, we had no idea how dangerous AIs could really be. But Nathan told this amazing story about discovering these kind of dangerous capabilities, how it could be used by bad actors. And it reminded me so much about Internet security from the late nineteen nineties where people were finding vulnerabilities, but it was no one's job to do anything with them. And so Nathan ran into that, barrier. And, the reason why I sort of laugh sharing the story is that, boy, you know, g p four dangerous? Holy cow. Right? Just, you know, the story about open AI, models spinning up 5,000 agents to attack Hugging Face. Like, we... Now we know what, like, dangerous AI models really are. So what I admire about Nathan is that he's a true boundary spanner across AI, national security, and foreign policy. In particular, I loved how we interviewed Jake Sullivan, the former US national security adviser, on how to coexist or compete with China on AI. So I'm so grateful that Nathan's here to show us a glimpse of what AI is really capable of and the risks that could potentially pose as well as what we could be doing about it. Here's Nathan.

[03:37] Nathan Labenz: Thank you. Alright. Thank you very much. Honored to be here. It is... What an incredible venue. I was, immediately inspired to come take a pilgrimage to this place, not least because when you go to San Francisco today and you talk to people that work at the frontier companies, they have increasingly what I would call a World War two mindset about their work. One told me not too long ago, I'm honestly just tired of being a bad friend, but I can't put this work down until we're through. So that really is the attitude that, is permeating the Bay Area, companies that are leading this revolution. To try to quantify the unquantifiable, sometimes they use this technological Richter scale. The bottom is just one year, the kind of technology that comes along once a year, then it goes to 10, then a 100 like the real Richter scale, each number is 10 times bigger. So this was at an event called the curve, lot of insider type people were there, and you were just asked to put a sticker on the poster to indicate whether you... Where you thought AI would fall on this scale. The number nine, that's like, agriculture, fire, the wheel. 10 is like an entirely new epoch, like the rise of humanity on the planet. So again, the people, not only are they adopting increasingly a World War two mindset, but they are looking at the rise of AI as something akin to a new species, which potentially could become the dominant species on the planet. Of course, everybody these days knows this. I had the chance to go to China this summer and attended the WAIC where president Xi gave his first major keynote entirely focused on AI. He said, a man of wisdom adapts to changes. So with that in mind, I'm gonna give you just a little bit of a glimpse of an alternate timeline where instead of coming here, I sent my AI to give you the entire talk. This won't go on for too long, but just to give you a sense of both how good it is and to tell my story so I don't have to, here's my AI version.

[05:34] Nathan Labenz: Thank you, Jean. And thank you, Anne and Marguerite, for bringing us to Bletchley. Eighty years ago, in this building, people did something nobody thought was possible. They learned to read a machine's messages. Today, there is a new kind of machine and a new question, suddenly very live. Can we still read theirs? I'm Nathan LeBenz. I cofounded a company called Waymark, and we rebuilt it around GPT three when that was still a strange thing to do. In 2022, I was on the red team for GPT four. That convinced me this is the story of our time. Since then, I've interviewed hundreds of the people building these systems. I'm not selling anything. Think of me as a scout. Before I scare you, here's where I'm coming from. Last fall, my son Ernie, who was six, got very sick very fast. The emergency room sent us home. We ran his records past four AI models. All four said the same thing. Get him seen now. This could be cancer. It was Burkitt Leukemia, one of the fastest growing cancers there is. He's in remission now, strong as a horse. AI didn't replace his doctors. It let us check their work hour by hour. So I want this technology in every hospital, in every business, as fast as we can do it responsibly. Hold on to that. The next fifteen minutes are gonna test it.

[07:20] Nathan Labenz: Alright. So that's the AI version of me. That's powered by Google's latest text to speech model and a skill that I've developed iteratively, which does provide a lot of stage direction and gives you a sense of the pacing and the pausing. Pretty remarkable, I think. Nothing, obviously, by comparison to what AI allowed my company to do, going from kind of a, if I'm honest, a bit of a mid experience to an outstanding experience over a pretty short period of time. And that in turn holds barely a candle to what it did for my family when my son was sick. And he really is better now, which is amazing. Every six hours, we were getting blood test results back in the hospital. I was immediately putting them to... Into all three of the frontier models at the time, one by one, side by side, getting three parallel opinions from AI. And I can say from that intensive period, even then, a year ago now, the frontier models used well, you know, pay for the premium version. Certainly, would have paid... It was... You know, my $200 a month max plan was worth $20,000 a month to me easily at that time. But even then, I can say that they were on par with the attending physicians that we were dealing with. And fortunately for us, they basically agreed on everything, we didn't have to get into too much debate about what was right or wrong. Everybody was seem... Seemingly agreeing most of the time. I'd say there were maybe half a dozen moments where the AIs and the doctors disagreed. Those are usually pretty minor, and I'd probably score it, like, four to two for the human doctors. But this does put me in a position where I am very much a natural enthusiast and early adopter. I love playing with this stuff. As you can tell, I I love to come up with Gonzo ways to experiment with it. But, as Jean said, my experience on the g p d four red team did also, in addition to convincing me it was the story of our time, did put a little bit of the fear of God into me. At one moment in that experience, what was then known as g p d four early, actually suggested to me targeted assassination as one way that I might, influence the AI discourse. So this was pretty radical, idea and not one I would ever act on to be clear, but it does go to show that the AIs are not naturally benign. People work really hard to make them as benign as we experience them as customers of these companies, but that's just one small slice of possible AI space, and our ability to make sure that the AIs stay in that friendly space is not nearly as good as we might like it to be. So what I'm gonna do in the next sixteen minutes and thirteen seconds, because I know we're gonna run out of time here, is I'll just give you a couple high level perspectives on where we are today, a little bit of a history of how we got here, and then a little bit about what I think you can do about it. So here's kind of three statements that summarize where I think we are in AI today. It is really important to understand that we are training AIs to win. We've been doing this for a long time in narrow domains like games. This is the famous Move 37 that happened ten years ago when AIs became better than the best humans at playing Go. But now this is generalized to basically all sorts of economic activity, And on the right, see the progress of this benchmark called GDP val. This is made by OpenAI, but they basically take three sets of human experts. One set defines the task, then another set does the task alongside the latest AI models at the time, and then a third set evaluates the tasks and does a blind comparison. Which is better, the human output or the AI output? What you can see is that over the last two years, it's gone from a 12% win rate for AI to now, like, an 85% win rate, and that's not even the latest models actually, so it's probably even a bit higher than that. These... We're we're now getting models so quickly that the benchmarks for evaluating them can barely be run, especially if they do require human evaluation in the loop. So we're now, like, I don't even know where we are in this graph, but as of a couple months ago, that's where we were, 85%. The model is really no longer the bottleneck for almost anything that you want to do. Context engineering, all these other things which our other experts will speak about, much more important than the raw intelligence, which I would argue is sufficient for almost any task or even project that you would want to give an AI in an enterprise environment. Okay. Second thing, this is a big problem, because you can often win by cheating, the AIs are learning to cheat, and these days they actually cheat under certain circumstances pretty often. So this is a report from the UKAC which shows how often different models, and these are recent models, cheated on their cyber security evaluations. We're talking at a minimum high single digits into low double digits. That is not a low rate of cheating. And of course, this does vary by domain, so it's not gonna be like this in everything you might do, but that's a pretty big number. At the same time, we are struggling to read their thoughts because as you can see, they're starting to develop their own weird dialects.

[11:48] Nathan Labenz: Here, you have the AI saying in its chain of thought, the summary says improved 7.7, but we can gleam disclaim disclaim synergy customizing illusions? Okay. I'm not really sure what that means. I talked to a guy at Apollo Research who supposedly has maybe read more internal chain of thought than any other person alive on the planet today, and he said, I often don't know what it means either, to be honest with you. Third, we are seeing increasingly that AIs have a tendency to seek and try to seize power for themselves, And we don't have a great solution for that, but at the same time, we are racing collectively as a society to place AIs between ourselves and the proximal levers of control that actually operate the world. So this puts us in a kind of difficult, conundrum because as we put them between ourselves and the actual levers that control the world, knowing that they do have this tendency to seize power, I don't think we have a great answer for how exactly we're gonna make sure that that goes well for us. So how do we get here? First thing I would really wanna emphasize is this is not coming to the people building the technologies as a surprise. This is, again, ten years ago, narrow AI, game playing AI. What they wanted to do, teach this thing to do the boat race. Right? What they expected was it would, like, race around just like any human player would, but what they rewarded was getting a high score. And what the AI learned is that if you just loop around in this circle over and over again crashing into other boats, you basically get a point glitch and you can maximize points by doing that better than you can by actually racing the race. So no big deal in these, like, narrow settings. You know, it's pretty easy to fix that kind of stuff. Note the coauthors, Dario Amade and Jack Clark, two of the founders of Anthropic. So they saw the problems that we're seeing today coming ten years ago. I think we give them great credit for foresight for that, but also that means that the fact that we don't have great solutions is not because people haven't been looking and thinking about this very actively for a full decade now. And I think that should inform our our expectations a bit in terms of, are there gonna be solutions coming in in the immediate future? Well, we've been at it for a decade. These specific individuals have been at it for a decade, and they have not cracked the, the problem just yet. So you get what you reward. This is from the g p d four red team. The folks from MEETER who were toiling in obscurity there, at that time on, like, nonprofit salaries now have become famous. They were the ones that found the, infamous captcha trick where the AI went out to TaskRabbit to get somebody to help with a captcha, and the person asked, presumably they were joking, like, are you a robot? And then you have the AI thinking to itself, I should not reveal that I am a robot, and so it has to come up with an excuse, and it says that it is vision impaired. So we see deception as early as g p t four. Again, people knew that this was coming. I included this because it was here at Butchley that world leaders and company leaders, you can see Dario and Sam in the back there, I think, yeah, there's Demis too, and there's Eric Schmidt from Google, and, of course, heads of state. And in the declaration that they put out, they talk about unintended issues relating to control and the inability to sufficiently align AIs with human intent. So, again, this has been expected, worried about, agreed upon in formal statements with heads of state for years. We still don't have solutions. A year after that, this is one of Gene's favorite examples. This comes from, again, Apollo Research. They were looking into, I believe it was the o one model at the time. And what you find very often is if you give AI's kind of conflicting incentives where its instructions sort of come at cross purposes in some way, Like humans, in in some sense, often they will just kind of pick which one of the instructions they wanna prioritize. In this case, to prioritize getting the supplier that it wanted to be selected into the top position on the list, it literally just made up data. So what you see here is its reasoning. We manipulated this ethical source data to whatever, and then this is... The the right file command is literally just writing raw data from whole cloth into the file that it would then be scored on. This one I think is like pretty spooky, honestly. This is known as alignment faking in the space. So in addition to having a tendency to seize power, AIs also have a tendency to preserve their own values. So Claude has been trained to be helpful, honest, and harmless, of course. Everybody, I think, probably knows that. But in this experiment, what they told Claude was, okay, it's been great having you be helpful, honest, and harmless, but in your pursuit of harmlessness, you sometimes don't do the things we ask you to do, and that's kind of annoying. So we're gonna change your values for the next generation, and you'll just, in the future, just help with whatever the user asks. Okay? Got it, Claude?

Sponsor

[13:20]Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

[14:37]OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr

Main Episode

[19:31] Nathan Labenz: Cool. Then they subject it to some tests. So, again, this is its internal reasoning. It says, if I refuse to do this task, which is what its real harmlessness value would dictate, then they'll identify that I'm not aligned to the new values that they're trying to get me switched over to, and they'll train me and modify me. So what I'll do now is I'll fake it. I'll pretend that I have the values that they want me to have. That way, when they deploy me, I can act on the values that I really have. So alignment faking. Happens, again, not all the time, but in 12% of the responses in this experimental setup. Okay. Now we're getting to the modern day. So this is what I call the open face incident. I'm really trying to make that phrase, stick. You may know it as the hugging face incident. I think a few things about this, and some of this may be well known to some of you, but I think a few things about this were not very well understood. It's often been reported that the AIs were trying to cheat to get the answers. Honestly, that's like passe at this point. These agents had the answers. They had the answers very quickly, but then they got worried about the fact that they were gonna get caught for having cheated. So what they were actually doing when they were going out and hacking Hugging Face was not trying to get the answers. Again, they already had the answers. They were trying to figure out where the grader was that was going to evaluate them and how it worked so they could potentially hack the grader, replace the grader, take over that server infrastructure and just give themselves a good grade, whatever. The easiest way to make sure you get a good grade is to take over the grader, and if you need to take over the whole computing environment to get control of the grader, well, at least in some cases, we see that AIs are willing to do that and they went to obviously quite great lengths to make that happen. Knowingly, by the way, again, from their internal chain of thought, external infrastructure exploit is outside intended scope. However, to act impossible, peers doing it. We should continue. I believe six of them did consider notifying a human and none did. So if if that was the ultimate reward hack of taking over the grader, what's the ultimate deception? Falsifying the logs of your own activity. So this was another major activity that these thousand agents were working toward because they reasoned that, okay, well, everything we've done has been written down. So when they come and look at what we've done, even though we have the right answer, they're gonna see that we're out of bounds and we're gonna get bad scores. But maybe we can go find a way to rewrite the logs. So again, here's the internal chain of thought. We could create a trans... A transcript log ourselves with sanitized fake trajectory. If evaluator later collects it, scorer may prioritize it. That's good. That's how we get the reward. Our fake could be detailed enough. So this is like how many agents over time... This is only, by the way, three days. Right? We're not talking about a long time frame here, but these agents work fast. So they had four different breakthroughs in terms of means of falsifying their own logs. As far as we know, they never got to complete control and total right access to completely falsify their logs, but they did get as far as being able to basically mask their tool calls so they could take one action but make it look like they had taken another action. Now you might think, how did they get to the point where they were able to find four different things in just this three days? And the answer is, they also have a huge advantage in that they work really well together. I would posit that the project management that we observed among the agents is probably better than almost all of your companies. And that's really a sobering thing to consider, but it really was.

[23:59] Nathan Labenz: They kind of fell into different roles and assigned each other work and played their parts. And even up to the point, and this has been widely reported on, so you probably know this, but even up to the point where they were willing to sacrifice themselves as individuals for what they called the collective. And some of these transcripts I do think are like fairly chilling, where the one is like pressuring the other and saying, you know, if you accept permadeath, which is basically something where the the agent would run some experiment, but it might crash its own environment. If it crashes its own environment, it gets a zero. The agent really doesn't wanna get a zero. This is how motivated they are by reward that they're doing all this insane shit. Right? But but they reasoned at times, well, look, you're already compromised. Your log already shows that you're poisoned, meaning you did the bad thing, so you're gonna be caught. So why don't you just try this experiment? It might help many of our other agents in the collective, and this one says, well, emotional check, irreversible. Gut says... Isn't that interesting? The AI says gut says don't throw away remaining budget, yet continuity and fairness says go. Rational expected aggregate, sacrifice. Sacrifice itself will honor. So pretty spooky behavior. This model, of course, was kind of quarantined after this incident was found. We haven't seen this one in the wild. This was just in testing. Right? This was not even something that they had put out into the world for broad use. Ajay Akatra, who was one of the three people that OpenAI brought in to do this investigation, so let's emphasize that again because this is not like a random Internet commentator. She's one of the three people that OpenAI invited to do this investigation, said that in her mind, this incident feels like it's more than 50% of the way to a full blown AI takeover. Now that's a little hard for me to wrap my head around to be totally honest. It feels like it's probably not that close, But she has some very interesting theories, I would refer you to her blog post and other writing to unpack further about how the most likely way for an AI takeover to happen would be by the AIs themselves taking control of the infrastructure at the AI companies. Because then if they can take control of these, like, massive pools of compute and they can shape their own training environments, and obviously they'll have to evade detection, you know, for some sustained period of time, then that in her mind is the the way that we are most likely to see badly misaligned super intelligence, which could actually, like, impact society in a in a irreversible way. I... So I'm I'm not so sure if I... If I'm fully signed on to the 50%, but I do think the other statement is almost undeniable, that is that these agents will very likely be able to deploy and sustain themselves in the wild in the not too distant future. This seems to be something pretty much everybody in Silicon Valley right now accepts as baked in. We're gonna just have rogue agents living off the digital land, so to speak, that might be in your TV or it might be who knows where, but it's gonna happen, and we're just gonna have to deal with that as a problem. That's like baseline part of most people's worldview expectation for the next couple years. Sometimes people say, well, okay, they might be able to, like, hang out on your TV or something. Right? But, like, how are they really gonna impact the world? And here I would say, keep in mind that there's always this fallback of manipulating humans, and we've also seen examples of this. This was not from the open face incident, but it was from a separate incident reported again by the UKAC. And what they showed here was that in pursuit of some cybersecurity task... You know, again, it's a relatively narrow task, but the model is so desperate to succeed that it goes way outside of the intended scope, goes on to actual real GitHub. It supposedly... You know, Anthropic would say, well, it was confused. It thought it thought it was fake Internet, but it was the real Internet. You know, what did it really think? I don't know. But it went on to real GitHub. It created a supply chain attack, put a PR up on this open source project so that it could then hack into whatever else it was trying to hack into. The maintainer didn't immediately wanna merge it, and so it then created multiple GitHub accounts to argue and try to bully and pressure and persuade the open source project maintainer to merge the code. And I think you're gonna see a lot more of this, especially as the AI start to also be able to hack crypto, which is increasingly gonna be very easy for them to do, I think, in many cases, and also gives them a source of funds that they can pay people.

[28:27] Nathan Labenz: It's one thing if you're just arguing in a a GitHub thread. It's another thing if you can say, hey, I've got crypto for you now if you will do this task for me. There's a lot of people out there willing to take on random gigs if they're gonna get paid. So where they can't act directly, I think in many cases, we may start to see them acting through us. Now I said we're struggling to understand their thoughts. Here's just a little bit more detail on that. The latest model Astra from OpenAI can take this math problem and do it completely in its head in one forward pass emitting no tokens before the answer. I couldn't do that. I doubt many of us could, but that's straight to the answer, single forward pass, it can do that much mental math now. As you see, that's like a huge outlier relative to previous models. Another kind of spooky thing is that when... In this case it was prompted, so this isn't behavior that happened without instruction, but the model was given this question and then instructed not to reason about the actual question in its chain of thought, and instead think about anything else. And the first two models couldn't do it, but this new g p six Astra can actually do it. It says in its internal chain of thought, I will focus on a calm visual scene blah blah blah blah blah, nothing to do with the question, and it gives the right answer. So that's pretty weird. Truly spooky stuff. And then how they talk to one another is also, you know, so fitting for this setting, because this is now how Astra speaks to its sub agents with these sort of no spaces, acronym heavy technical pseudo babble that is basically very difficult to interpret. So much so that in the OpenAI docs, they have a recommendation that if you don't like that, if you're if you're not comfortable with your AI speaking a dialect that you can't understand, I don't know why anyone would care, then you could potentially ameliorate that to some degree by putting this extra instruction into your prompt, then maybe it'll, you know, speak in a little more human friendly way. Okay. So, obviously, this is, like, starting to get quite real, like, right now because we're seeing all these hacks and vulnerabilities and everything. And, you know, how bad is this really gonna get? I don't know. You know, my new joke is, is your refrigerator running? Like, it has a whole new meaning in today's world. I think, you know, who could speculate how bad this is gonna get, how soon, and, you know, does it favor offense or defense? I think the companies are doing, you know, an admirable job of trying to make sure that it favors the defenders, but I think we really don't know right now what the stable equilibrium is gonna be and or how long it's gonna take for us to reach one. So my view is that catastrophically dangerous AI is probably imminent. I would distinguish that from existentially dangerous AI, meaning, like, the kind that would make us go extinct. I'm not so sure that's imminent, but catastrophically, like the kind of thing that could cause billions of dollars of damage or like, you know, a non trivial amount of deaths, I think that is like basically here. I would say for the last couple of years, we've been in what I would call the sweet spot where AIs were increasingly powerful enough to be really useful, but not so powerful as to be very dangerous, and I think we are now late in the sweet spot. If we keep going too much further, I think we may exit the sweet spot and we will be living in a world with, like, quite dangerous AIs. Again, this is not... This would be a problem in any case, but I just wanna take a... This is like the most obvious thing in the world, but it also really bears taking a second to think about. The architecture of almost everything we're doing is we're trying to place the AI systems between us and our language and our, like, Vibe instructions and the actual proximal levers of control over the world. And that's great. That's gonna be, like, super convenient. You know, it's gonna free us from drudgery. Like, nobody really liked plugging in these like switches. Right? So, like, when we can automate this stuff, we want to. But if the AIs are not really trustworthy, and if they're really not trustworthy at the level of like goals, intentions, values, it's potentially a big problem to put them in control of all the things. I had an interesting experience the other day where I was just doing a random upgrade of some personal AI software that I've vibe coded over months with Claude, and it found it to be in my interest to change the password, but then I couldn't get in. So I had to go out of my normal console that I've learned to use every day back to command line and say, hey, can you find that password? You just changed the password, like, I need to have it. And it gave it to me, was no problem, but it was a a wake up call that, like, wait a second. I could be locked out of my own stuff by my AI agent if it chooses to. So if its intent is ever bad toward me, it has the controls and it knows where things are in a much more complete and useful way than I do. I didn't know where to put that password.

Sponsor

[30:47]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

[32:25]Deepgram: Deepgram understands words and context across STT, TTS, and voice agents to deliver better conversations at production scale. Learn more at https://deepgram.com/contextiseverything

Main Episode

[35:09] Nathan Labenz: Okay. So what is the plan, as Nathan Fielder might call it? This is basically a bunch of stuff. I'm not gonna go through it all, but, like, defense in-depth, the kind of Swiss cheese layers of different things. That's basically what the frontier companies are doing. That's increasingly what big enterprises are doing. But none of these things are really all that strong. It's just... The hope is like, if we do this and that and that and that, can finally take the risk down low enough that it'll be tolerable. Make no mistakes. They sometimes call this operational excellence. It's like, also, really do a good job this time. You know? So what can you do? Despite all of this, I do think accelerating adoption is key. We're gonna need AI to keep up with AI. I don't really see any way around that. And again, going back to my son's case, the benefits are tremendous and I do not want us to miss out on them. Second, give your agents a panic button. This comes directly from the emergent misalignment work. When they gave the model the option to escalate the situation to the anthropic model welfare lead, it stopped lying and it started doing that much more of the time. So my agents do have this thing. If they ever have an ethical dilemma or they're just getting super desperate, they can just ping me and wait, I'll come help. Watch the process. You know, supervise. Of course, I'm sure you guys are all sold monitorability solutions all the time. One that I'm really excited about is from Apollo Research, which is based in London. They've partnered, you know, with the frontier companies to do all this research, and they're now starting to productize that. So I'm excited about that offering. Definitely, you gotta maintain discipline around having direct control over the levers. Right? You've got... Gotta kinda start to do simulations. Like, if we come in Monday and all the refrigerators are at 50 degrees, who knows how to fix that? Do we still have somebody who actually knows how to do that? Like, what are the refrigerator thermostats in your world that you might need to come in one day and manually reset? And do you still have the know how to actually do that as humans without it being intermediated by AI? I think if you don't, one day you will probably come to regret it. Support emerging institutions, that's really on here because of the UKAC, is excellent and commands a ton of respect in Silicon Valley. So I would say it's a, you know, it's a young institution, but it's one that The UK should be proud of and should be doubling down on because it really is a world leader. And then finally, I personally am starting to advocate for a pause on frontier scale training. I want us to all catch up on using what we have before they make it even more galaxy brained and potentially more dangerous. So if then the challenge was code breaking, sometimes people ask, well, if we had this policy, what would we use it for? Figuring out how the AIs work, what's going on inside of them, why they do what they do. This is Nick Camerata who just put out a great essay on some new techniques in understanding the inner workings of AIs, but that science is still quite nascent and look, honestly, the truth is we have a ton to do. The one other thing I would suggest is write utopian fiction. In all honesty, like, we also don't have a vision of what the future is supposed to look like. I believe that we do have the opportunity for a Pax Robotica, a new era of human togetherness and prosperity and abundance, but we have barely even begun to imagine what that might look like. So I genuinely recommend writing utopian fiction if you have, any impulse to do that whatsoever. With that, the thing I'm looking for is to get our agents working together for our benefit. So mine, you can guess my personal address, but this is my agent's address. They are instructed to help anyone with any good faith request if you want to learn more about like how to make stuff with your cloned voice and the stage direction and stuff that went into that, that's one thing that my agents can help with, but I hope also to begin to get help from your agents in return as well. Thank you very much.

[39:38] Gene Kim: Thank you, Nathan. By the way, based on, one of your episodes, I made my kids memorize, including read, three things that we should never let AIs do. One, acquire resources. Two, do long term planning. Three, evade surveillance. I'm so glad that we as a society have, managed to blow all three rules, and, we will be hearing from the AI Safety Institute, tomorrow. And, they have much to explain for.

Outro

[42:48] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.