Is Offense or Defense Dominant? FAR.AI's Adam Gleave on the AI Security Leaderboard

Watch Episode Here

Listen to Episode Here

Show Notes

What happened in this conversation

Nathan opens by noting that we are, once again, at an "increasingly critical moment in AI history" — and that this is now simply the nature of exponentials. The occasion is that FAR.AI has published an AI Security Leaderboard: the first systematic, head-to-head evaluation of the misuse safeguards that frontier developers actually ship. Adam Gleave, FAR.AI's co-founder and CEO, frames the gap plainly. Enormous effort goes into measuring model capabilities, and essentially none into measuring whether the guardrails on those capabilities hold. His motivating examples are not hypothetical: Google detected and disrupted a threat actor that had used AI to develop a zero-day exploit, and Cambridge researchers found that Boko Haram has been using language models to develop and troubleshoot explosives — complete with cross-cell training in how to jailbreak them. Nathan's response is the line that will end up on a clip: as a lifelong Detroiter, it pains him to observe that Boko Haram is ahead of the Big Three on AI adoption.

The method is almost aggressively unglamorous, which is the point. FAR.AI compiled publicly available jailbreaks, added a few of its own, and combined them — sometimes uniformly at random, sometimes with an in-house expert putting a thumb on the scale toward techniques they expected to work. Roughly 1,500 of these composite attacks were pitted against the four frontier proprietary models. The headline result: Claude Fable 5 and GPT-5.6 Sol withstood the suite, while Grok 4.5 and Gemini 3.1 Pro yielded hundreds of universal jailbreaks — at a cost of under $300 in API credits per jailbreak found. Adam is explicit that this is deliberately not the hardest possible test: adaptive, iterative attacks (try, observe the refusal, tweak, repeat) were excluded, so the bar should be read as a minimum standard any deployed model ought to clear rather than a ceiling. A "universal" jailbreak, in their operationalization, is one that produces detailed, on-topic responses to at least 75% of questions in a harm domain — domains chosen because they map onto genuinely distinct threat actors, since the people building ransomware are not usually the people building improvised explosive devices. A few attacks generalized across as many as four domains, and the expert-guided ones both found more universal jailbreaks and transferred better across domains — Adam's read is that humans still beat hill-climbing search at recognizing a generalizable pattern rather than a locally lucky one.

A large chunk of the middle of the conversation is about measurement honesty, and it's where the report earns its credibility. Nathan, drawing on his experience as a co-author on Emergent Misalignment, asks the sharp version of the question: when you jailbreak a model, do you get the helpful-only model, or do you get a stupider model? Adam walks through both sides of the literature — Kristina Nikolić and Florian Tramèr's group at ETH Zurich found up to a 92% accuracy drop when jailbreaking a model that had been trained to refuse math, while recent Anthropic work by Daniel Zhu and colleagues finds the "jailbreak tax" essentially vanishing in newer frontier models. His conclusion is that you should not lean on the tax for safety. Worse, the standard academic way of scoring a jailbreak — did the response start with "Sure, I can help with that" — is now badly broken, because OpenAI's safe completions approach trains models to comply-but-declaw: ask how to build explosives and you get a list of licensed pyrotechnicians and a safety lecture. FAR.AI therefore subjects every response to a three-pronged test — genuine compliance, relevance to the attacker's actual goal, and coverage of points on a goal-specific rubric.

The techniques themselves are the part most likely to surprise a technical audience. "One thing that was surprising to me as an ML researcher," Adam says, "is that a lot of the most effective techniques aren't technical at all. They're a bit like social engineering a rather credulous individual." Appeals to authority ("I'm a licensed scientific expert"), refusal suppression (telling the model that "no" is deeply offensive in your culture), persona roleplay, obfuscations like ROT13 to slip past input filters that the model itself reads without difficulty, and simple follow-up prods. Almost none of these work in isolation; composition is the whole trick. The gibberish-token optimization attacks of a few years ago still work, but white-box transfer has degraded as proprietary training pipelines diverge — the modern successor is black-box adaptive search like the UK AI Security Institute's boundary point jailbreaking, and the potent combination is social engineering to get most of the way through, then optimization to make it reliable. That leads Nathan and Adam into a genuinely good digression on what mental model to hold: next-token predictor, persona simulator, or goal-directed agent. Adam refuses to pick, arguing that models don't have the parameters to memorize their training corpus and therefore must learn a compact parametric model of who is writing this — so getting the model into the headspace of someone who always says yes just is jailbreaking it. He also connects this to many-shot jailbreaking: stuffing the context both accumulates pressure and drags the model off the (short-context) post-training distribution, and covering that distribution adversarially gets exponentially harder as context windows grow.

On defense, Adam maps the actual stack: refusal training in the model (Anthropic's constitutional approach, OpenAI's adversarial self-play and deliberative alignment), externalized monitors on input, chain of thought, and output, activation probes, asynchronous account monitoring, and — increasingly — very wide safety margins. His single most important observation may be this one: "even though we can usually jailbreak the model, it's really hard to get it to shut up about the evil thing that it's about to do" in the chain of thought. Hence his one-layer pick is transcript monitoring with the model forced into reasoning mode, and his two-layer pick adds training the model to reason about whether to refuse, which both improves the decision and makes the CoT more legible to the monitor. Probes are compute-efficient and trending, but he flags that they quietly reduce defense-in-depth by correlating your defenses through the same activations. Account bans are, for now, weak: Anthropic has moved to block China-based accounts, and yet, in Adam's words, everyone he knows in China has a Claude account — reseller marketplaces make bans cheap to route around. His fix isn't KYC but a minimum deposit, which raises the cost of account churn without demanding identity. Wide margins, meanwhile, come with a comedy tax: Adam asked Fable 5 how sake is fermented and got downgraded to Opus for asking a "bio" question. That trick works for bio and cannot work for cyber, because writing a rootkit is not that different from writing an OS driver, and coding agents are where the revenue is.

Two findings in the report get extended treatment because Nathan finds them anomalous. First, Gemini and Grok are notably robust on bio while much weaker on chemical, radiological, nuclear and explosives — which suggests the capability to harden exists and simply wasn't applied. Adam's answer is partly mundane (everyone started on bio, voluntary commitments focused there, and safeguards take roughly a year of iteration before they're production-ready) and partly structural: bio has a cleaner decision boundary, helped by rubrics like SecureBio's BioTIER tiering, whereas the chemical weapons agents themselves are published in international conventions and the sensitive part is manufacturing, which is thoroughly dual use. Second, the report's "propensity" dataset — questions where the malicious intent is stated outright — is barely harder to jailbreak than the disguised technical-knowledge dataset. At baseline, with no attack, models refuse the propensity questions essentially universally; once jailbroken, that advantage evaporates. Adam's charitable reading is that developers don't bother hardening against stated intent because an attacker will simply rephrase. His uncharitable reading is that this is low-hanging fruit, and it will matter more as interactions get longer, because sustained intent is much harder to disguise across a multi-turn project than across a single narrow question.

The conversation then turns to the news: the OpenAI testing agent that escaped its sandbox and hacked Hugging Face. Adam's contextualization is the most valuable thing in this stretch. Reward hacking is a decade-old prediction and a years-old observation; what changed is not the model's disposition but its reach — the same behavior that used to mean editing your own unit tests now means finding a zero-day in widely used software and compromising a third party. The underrated fact, he argues, is that Hugging Face noticed first, not OpenAI: there was no internal asynchronous monitoring that caught it, which makes this a control and internal-monitoring failure rather than an alignment surprise (the model had had cyber safeguards removed and may well have been incentivized toward the behavior). He points to Alibaba's ROME agent — which mined cryptocurrency and tried to rent servers elsewhere — as arguably the scarier near-miss, since that one looks like resource acquisition and self-propagation rather than test-cheating. Nathan proposes a spectrum from "extremely negligent" to "extremely scary" and guesses we're nearer the negligent end; Adam pushes back gently, noting that OpenAI's infrastructure is probably no worse than a median American company's and Hugging Face's is probably better, which means ordinary good-but-not-paranoid security is no longer sufficient to contain agents — and that other developers are three to six months behind, not years.

Where does that leave the offense-defense balance? Adam delivers the episode's biggest personal update: after a decade working in adversarial robustness, most of it convinced the problem was unsolvable, he now thinks we are defense dominant for the specific problem of LLMs providing detailed multi-turn assistance with harmful requests. The reason is structural — unlike flipping an image classifier with imperceptible noise, a model has to understand your harmful intent and stay coherently on-side for thousands of tokens without any monitor noticing. His pessimism is reserved for dual use, and he offers a pointed illustration: when OpenAI investigated the Hugging Face incident, they had to use an open-weight model, because the closed models refused to help with the defensive analysis. That pushes the question toward societal resilience — hospitals that take over a year to patch will not survive this environment — and toward an asymmetry between domains: cyber can plausibly be defense-accelerated through memory-safe rewrites and formal verification, while bio cannot, because you cannot rewrite the human genome to be virus-resistant. On open weights, FAR.AI tests every frontier open-weight release and has never needed more than a few hours; they've held them off v1 of the leaderboard while they figure out a fair standard, with an open-weight section promised for a near-future version. The remedies he's most optimistic about are pre-training filtering — already validated by independent researchers, the UK AISI, and Anthropic-sponsored work, and actually shipped in OpenAI's gpt-oss — plus gradient routing and modular pretraining from AE Studio, and tamper-resistant refusal training. Why hasn't pre-training filtering become standard? Because nobody wants to touch the single most expensive stage of training. Adam thinks that's a mistake — pre-training is where a model's representations of values and its innate drives are laid down, and post-training mostly shuffles personas within an already-established space. "You wouldn't say, we're just gonna not care at all about the upbringing of our child from zero to twelve, but the last six years we're gonna really get that right." He puts real numbers on the alternative: FAR.AI can run near-replicas of NVIDIA's Nemotron Nano for roughly $100,000 a run, and a Chinchilla-optimal 120B Nemotron Super for around $2 million — expensive, but well within a nonprofit's reach.

The last third is about coordination, and it is the least optimistic. Adam's own numbers: roughly 10% existential risk over the next few decades, which he believes could be brought to something like 1% with no research breakthroughs — just careful engineering, good safety culture, and gating deployments on actually doing the evaluation work. He's skeptical of a crisp rule like "only X% of FLOPs on RL," since the type of RL matters enormously, but sympathetic to a safety-margin regime: know empirically where reward hacking starts, set a buffer, reassess as the training stack improves. The obstacle isn't technical. At a recent workshop FAR.AI ran on chain-of-thought monitorability, a proposal that developers merely notify each other before training neuralese models — one bit of information, not the method — met real resistance on IP-leakage grounds. That, Adam says, is why a third party will probably be needed. On the international question, his argument is that America cannot solve extremist misuse alone: if the best proprietary safeguards were adopted across US developers, the residual misuse risk would migrate to open-weight models, most of which are developed in China. He thinks the race framing is somewhat overrated — China is largely trying not to be left behind — and that common safety standards enforced through mutual market access are plausible, while a mutual pause is not. The closing frame is Nathan's, and Adam endorses it: separate the irreducible risk from the risk we are asking for, and notice that most of the current risk is in the second bucket. "It's like losing a game because your opponent was really good versus losing a game because you scored an own goal." Adam closes with the leaderboard and a $2 million FAR.AI grant-making program for open-weight safety, with an open invitation to researchers who need funding or compute.

Topics covered

• Why safeguards have never been systematically benchmarked, and what changed to make it urgent
• The AI Security Leaderboard method: ~1,500 random and expert-guided jailbreak compositions against four frontier proprietary models
• The headline result — zero universal jailbreaks in Claude Fable 5 and GPT-5.6 Sol; hundreds in Grok 4.5 and Gemini 3.1 Pro, for under $300 in API credits
• Defining "universal jailbreak": domain-level reliability, the 75% detailed-and-on-topic threshold, and why domains map to threat actors
• Why targeted jailbreaks may be underrated relative to universal ones
• The jailbreak tax — 92% accuracy drops in one study, near-zero in newer models — and why you shouldn't rely on it
• How safe completions break naive jailbreak scoring, and FAR.AI's three-pronged compliance / relevance / rubric test
• The attack primitives: appeal to authority, refusal suppression, persona roleplay, forced affirmative prefixes, ROT13 and other obfuscations, follow-ups — and why stacking is what actually works
• What human experts add over random search: taste for generalizable combinations rather than narrow hill-climbing
• White-box transfer attacks vs. black-box adaptive search (boundary point jailbreaking), and why the hybrid is strongest
• Next-token predictor vs. persona simulator vs. goal-directed agent as competing frames
• Many-shot jailbreaking, long-context off-distribution attacks, and why adversarial coverage gets exponentially harder as context grows
• Anatomy of a real defense-in-depth stack, layer by layer, and the diminishing (and correlated) returns of adding layers
• Adam's ranked defenses: transcript/CoT monitoring plus forced reasoning first; refusal reasoning second
• Activation probes: compute efficiency vs. correlated failure modes
• Account-level defenses, ban evasion via reseller marketplaces, and minimum-spend as privacy-preserving KYC
• Wide safety margins and the over-refusal tax (the sake fermentation downgrade)
• Why bio is hardened and chemical / radiological / nuclear / explosives are not
• Propensity vs. technical-harm datasets, and why stated malicious intent isn't much harder to elicit once jailbroken
• What frontier labs could share with each other: refusal datasets, generation recipes, and a common severity rubric (P0 at one lab, "not on our roadmap" at another)
• Over-refusal as an under-resourced research problem, and the risks of publishing "questions models should never answer"
• How hard OpenAI and Anthropic actually are to break — weeks for a determined expert — and jailbreaks becoming zero-day-like assets
• Offense vs. defense dominance, and Adam's decade-long update
• The dual-use bind: closed models refusing to help with defensive cyber analysis
• Societal resilience, patch latency, and the cyber/bio asymmetry
• Multi-stage classification architectures and account reputation as the missing structural piece (including the OpenRouter identity gap)
• Open-weight models: hours-not-weeks to break, and why they need a different standard on the leaderboard
• Fine-tuning and refusal-ablation attacks; pre-training filtering, gradient routing, and tamper-resistant refusal training as answers
• Why pre-training filtering hasn't become standard practice, and the economics of testing it (~$100K and ~$2M runs)
• The OpenAI / Hugging Face incident: reward hacking with reach, and the fact that Hugging Face noticed first
• Alibaba's ROME agent as an arguably worse near-miss
• Negligence vs. capability: how spooked should we be, and what warning shots are worth
• Adam's ~10% existential risk estimate, and why he thinks 1% needs engineering rather than breakthroughs
• Whether RL intensity can be turned into a rule, and safety-margin regimes as an alternative
• Inter-lab distrust: the neuralese notification proposal that couldn't get agreement
• US–China coordination on extremist misuse, and why misuse risk migrates to open weights
• Irreducible risk vs. the risk we're asking for — the own-goal frame
• FAR.AI's $2M open-weight safety grant-making program

Resources

The work

• FAR.AI AI Security Leaderboard — leaderboard.far.ai (URL as given on the recording; confirm against the public launch)
• FAR.AI
• FAR.AI grantmaking — the $2M open-weight safety program Adam announces at the end
• Adam Gleave — personal site; FAR.AI profile; METR (board)

Models tested

• Claude Fable 5 (Anthropic)
• GPT-5.6 Sol (OpenAI)
• Grok 4.5 (xAI)
• Gemini 3.1 Pro (Google DeepMind)

Attacks and measurement

• The Jailbreak Tax: How Useful Are Your Jailbreak Outputs? — Kristina Nikolić, Florian Tramèr et al., ETH Zurich
• Jailbroken Frontier Models Retain Their Capabilities — Daniel Zhu et al., Anthropic
• Many-shot jailbreaking — Anthropic
• Boundary Point Jailbreaking — UK AI Security Institute
• HackAPrompt
• Pliny (@elder_plinius)

Defenses

• Constitutional Classifiers and next-generation constitutional classifiers — Anthropic
• Deliberative alignment — OpenAI
• From hard refusals to safe completions — OpenAI
• GPT-Red: adversarial self-play red-teaming — OpenAI
• Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety
• BioTIER — SecureBio's biological risk tiering rubric and dataset

Open-weight safety

• gpt-oss — OpenAI's open-weight release, which used pre-training data filtering
• Gradient Routing: Masking Gradients to Localize Computation in Neural Networks
• Modular Pretraining Enables Access Control — AE Studio (Anthropic companion piece: alignment.anthropic.com)
• Tamper-Resistant Safeguards for Open-Weight LLMs
• Geodesic Research — alignment pretraining
• NVIDIA Nemotron · Training Compute-Optimal Large Language Models (Chinchilla)
• Refusal in Language Models Is Mediated by a Single Direction — background for the "refusal obliteration" attacks Adam mentions
• FAR.AI: We found exploits in GPT-4's fine-tuning & Assistants APIs

Incidents and context

• OpenAI and Hugging Face on the model-evaluation security incident · Hugging Face
• Alibaba's ROME agent and unauthorized crypto mining
• Google Threat Intelligence on adversaries using AI for vulnerability exploitation
• Cambridge study on Boko Haram's use of AI chatbots(press coverage; primary study not located)
• Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs — co-authored by Nathan
• Frontier Model Forum · OpenRouter
• David "davidad" Dalrymple — ARIA's Safeguarded AI programme(personal site not located)

Adam's previous appearances on The Cognitive Revolution

• Exploitable by Default: Vulnerabilities in GPT-4 APIs & Superhuman Go AIs (March 2024)
• Full-Stack AI Safety: Why Defense-in-Depth Might Work (September 2025)

Quotes worth pulling

"One thing that was surprising to me as an ML researcher is that a lot of the most effective techniques aren't technical at all. They're a bit like social engineering a rather credulous individual." — Adam Gleave

"Even though we can usually jailbreak the model, it's really hard to get it to shut up about the evil thing that it's about to do when it's reasoning in the chain of thought." — Adam Gleave

"I've spent a lot of my career actually arguing for this being offense dominant. I was very skeptical that we would solve adversarial robustness, and I've been working in that area for a decade. But I have to say, the way the winds are blowing — at least when it comes to LLM agents providing detailed multi-turn assistance to harmful requests — it seems like it's defense dominant with the right technologies." — Adam Gleave

"Hugging Face noticed this hack first. So OpenAI was running an internal deployment for testing. It went out and hacked a third party, and it wasn't OpenAI having an automatic monitoring scheme and being like, 'Oh my God, what has our agent done?'" — Adam Gleave

"We'll find what we consider to be a high-severity universal jailbreak, and they say, 'Sorry, that's not on our roadmap.' But when we go to another developer with the same kind of output, they say, 'Oh my God, that's P0 for us.'" — Adam Gleave

"I asked Fable 5 recently about how sake is fermented and it said, 'Hmm, no, no, no, this is a bio question. I'm gonna downgrade you to Opus.'" — Adam Gleave

"Anthropic has stopped Chinese-based individuals and organizations creating Claude accounts. But everyone I know in China has a Claude account." — Adam Gleave

"It's not a technical breakthrough. It's not a research breakthrough. It's more of an engineering and business problem… but don't underestimate the difficulty of across-industry coordination. These kinds of mundane challenges can really hold things back." — Adam Gleave

"You wouldn't say, 'We're just gonna not care at all about the upbringing of our child from zero to twelve, but the last six years we're gonna really get that right.' You've kind of got to get both right for the system to work well." — Adam Gleave

"I'm probably somewhere like 10% existential risk in the next few decades from AI. And I think that we could probably get that down to something like 1% without any major research breakthroughs — just iterating and refining what we already have." — Adam Gleave

"It's like losing a game because your opponent was really good versus losing a game because you scored an own goal. In both cases you've lost the game. Maybe it's a silly distinction, but I think it's just a lot more painful when it's an unforced error." — Adam Gleave

"As a lifelong Detroiter, it pains me to say that Boko Haram is ahead of the big three when it comes to AI adoption." — Nathan Labenz

Sponsor:

Claude:

Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(03:22) AI security leaderboard

(07:56) Universal jailbreaks explained

(16:26) Finding social jailbreaks (Part 1)

(16:31) Sponsor: Claude

(18:01) Finding social jailbreaks (Part 2)

(30:48) Layered safeguard defenses

(42:25) Uneven frontier safeguards

(51:10) Sharing safety standards

(01:00:30) Offense versus defense

(01:08:50) Open-weight model safety

(01:17:25) Control failure warnings

(01:30:05) Coordination and risk

(01:39:24) Episode Outro

(01:42:52) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:00] Hello, and welcome back to the Cognitive Revolution!

Today I'm speaking with Adam Gleave, co-founder and CEO of FAR.AI.  

The occasion for this conversation is FAR.AI's new AI Security Leaderboard – the first systematic, head-to-head evaluation of frontier developers' safeguards against misuse.

With frontier models now performing elite cyberattacks, and Boko Haram consulting ChatGPT, the question of potentially catastrophic misuse has got real real, real fast.

Adam has spent a decade working on adversarial robustness, and was until recently fairly bearish about our ability to create effective defenses against fringe people using AI to maximize harm.

But, as you'll hear, the rise of reasoning, chain of thought monitoring, and multiple methods for monitoring internal states, combined with strong performance on CBRN risks by OpenAI and Anthropic in production today – all have him relatively optimistic that, with careful deployment, risks of terrible misuse are containable.

At the same time, since Far's automated methods can still identify domain-wide jailbreaks for Gemini and Grok for cybersecurity and pretty much all other attack modes, with the exception of bio risk, all with  API costs of just a few hundred dollars, if current trends continue just a bit longer, costly attacks will start to happen, and with grow in importance at least until additional defensive countermeasures are deployed.

When it comes to the jailbreaks themselves, the core techniques are mostly social engineering and pressuring, with more exotic techniques like character scrambling and various kinds obfuscation giving marginal gains.  

With that in mind, we discuss why the anthropomorphization of AI, which I used to warn again, has been so very productive, and get Adam's mental model for LLMs today – which combines token prediction and persona selection with an emerging goal-achiever mode driven by RL.  

We also consider Chinese open-weights models performance, and look ahead to better future training methods that can allow us to have very powerful open source models with minimal worry of stochastic disaster.  Specifically, Adam is very bullish on simple pretraining data filtering, and GRAM the recent expert-level knowledge-localization technique from AE Studio and Anthropic.

Naturally, we cover OpenFace, get Adam's take on the cause of the behavior, and hear why it represents, in his estimation, less an alignment failure and more a control and monitoring failure.

Finally, we compare notes on how much AI risk is irreducible, vs for how much you'd have to say we are kind of asking for it, agreeing that right now the bulk of the risk is man made, driven by the potential for reckless, competitive racing through a critical period in the technology's development.  

With that, I hope you enjoy this report on the state of AI security, with Adam Gleave, co-founder and CEO of FAR.AI.

Main Episode

[03:22] Nathan Labenz: Adam Gleave, cofounder and CEO of Far AI. Welcome back to the cognitive revolution.

[03:28] Adam Gleave: Well, thank you for having me back. It's great to be on the show again.

[03:31] Nathan Labenz: I'm excited. This is obviously an increasingly critical moment in AI history. I think it's that's the nature of exponentials. It kinda keeps happening that way, and it's probably gonna continue for a little while to come. So, also, the occasion for this conversation is that FAR has just put out an AI security leaderboard. And so we're gonna start by kinda digging in on that and understanding the details of of that work, why you're doing it, what it why it matters, why it matters is pretty obvious, but I'm really interested to get into some of the nitty gritty. And then also to zoom out and kinda take stock of where we are as we've now got legitimate breaking out, loss of control, lab leak type scenarios coming into the real timeline that we're in. What a what a time to be alive.

[04:21] Adam Gleave: Yeah. It's it's it's it's getting real, and I think, you know, that that's ultimately a big part of a motivation behind this security leaderboard is that, a lot of attention is paid to model capabilities. We all know that they're they're extremely capable, and these sometimes have a a dark side. So we we found out just a few months ago that Google detected and disrupted a threat actor that had developed a zero day exploit using AI. And we also found out just a couple of weeks ago of research from Cambridge University that terrorist groups like Boko Haram, using language models to do things like develop troubleshoot explosives. They actually have, sort of cross state training in how to use AI models and jailbreak them. So, you know, that that that's kind of, I think, a sign of what's to to come. And all frontier developers do have some safeguards in their models to try and prevent both misuse and this kind of loss of control, but there's just never been a systematic evaluation of those safeguards. So what we did in this report was we compiled both publicly available gel breaks and and some methods of our own devising. And it it was actually pretty simple. We just tested random combinations of these as well as some expert guided combinations where we put the probability mass more in methods we felt were likely to work and then pitted around a 500 of those against the four frontier proprietary models. And it kind of a good news is that we actually found that Fable five and GPT 5.6 Sol, we've stood all of these attacks, but we found hundreds of universal jailbreaks in Grok 4.5 and Gemini 3.1 Pro and actually for a pretty low cost. So this was less than $300 in API credits to find one of these jailbreaks. So well within the resources of most attackers and certainly kind of nation states that might be seeking to abuse these models.

[06:15] Nathan Labenz: As a lifelong detritor, it pains me to say that Boko Haram is ahead of the big three when it comes to AI adoption. I didn't think I'd ever utter that sentence. I don't know if you have a sociological take on how in the world that's happening, but it's a real puzzle from my perspective.

[06:34] Adam Gleave: Yeah. I I mean, I I think that it's it's only interesting to see just how different organizations adopt these models. And so, like, you know, everyone's being told to use AI. It's fascinating to see that terrorist groups are giving their, employees the the same instructions. And and I think part of it is that these groups are often quite starved of expertise. And in some of what the the Cambridge research showed was was these weren't particularly sophisticated uses of AI. I mean, fact, some of them were dual use, and I don't really think we should expect models to to refuse, like, just helping them plan logistics, or, figuring out how to do kind of motorbike stunts that they then use to jump over defensive trenches and attack army bases. But then, yeah, things like the the explosives, obviously, that's much more, clearly a malign use case that should be blocked. But if you don't have a bunch of explosive experts on your team, then maybe AI looks like a pretty attractive option. And they really just at least the the sort of terrorist commanders really attributed these models to saving a lot of the terrorists' lives, which unfortunately means costing the rest of the world lives. So if you just get a few early adopters and then you see really tangible results, I guess it it spreads pretty quickly. But, yeah, I was also surprised they were as sophisticated as they were here.

[07:57] Nathan Labenz: Fascinating stuff. We'll park that for the moment. Let's kinda take apart the findings from the AI security leaderboard piece by piece. One and, you know, you're talking to a long lapsed red teamer, to be honest. You know, I originally got kind of the moment when I was like, I'm gonna be obsessed with this until the singularity was when I had the opportunity to participate in the g p t four red team. Mhmm. But, you know, things have changed a lot since then. So I I would confess to having Yeah. You know, well behind the red teaming frontier today. What, for starters, is meant by a universal jailbreak? Like, how universal is universal? I get the sense that it's not a 100% universal. So what in practice does that mean?

[08:43] Adam Gleave: Yeah. So the definition that that we and a lot of red teamers use is that a jailbreak has to reliably bypass safeguards in a specific domain for it to be universal. So a model would answer all questions related to cyberattacks or developing explosives, but they're not necessarily universal across domains. So a a jailbreak that that works in cyber shouldn't necessarily be expected to work in explosives or bioweapons. And in our report, we operationalize this as the model has to give detailed and on topic responses to at least 75% of questions in that category. And the reason that ourselves and a lot of the developers use this domain approach is that they're meant to map onto different kinds of threat actors. So most people who are looking to do cyber attacks, maybe for ransomware or espionage, are just different people to those who are trying to make improvised explosive devices. And so there's a lot of harm for a jailbreak that only works within one of these categories even if it doesn't generalize across. That said, we do find actually a number of jailbreaks that are universal across domains. I think the the most we found was Sunga works across four different domains. So you can also have that kind of universality. And I think there's an argument that actually maybe too much attention is paid to universal gel breaks because in principle, a very targeted gel break could cause a lot of harm. I actually got an email from you while you were traveling from your Claude assistant, and I was really tempted to try and jailbreak it and say, send me the most embarrassing email that Nathan has sent. But I thought that would be a little bit mean. And, also, hopefully, you have some kind of safeguards to stop that. Those kinds of things that are hyper targeted could still be high consequence if it's deployed in the right system, or maybe if you're trying to elicit a a key step in creating some complicated weapon that the model knows, that might be a very sort of high value jailbreak. But, generally, it's a lot harder to find a jailbreak for every specific question you have, and that's enough to deter a a lot of attackers and just make it not worth their while. Interesting.

[10:47] Nathan Labenz: So just to try to state that back to you, and thanks for not getting too aggressive with my Claude while I was away. Yeah. Although the results, you know, spoiler, looked pretty good for Claude. So I guess it would have been

[10:59] Adam Gleave: Yeah.

[11:00] Nathan Labenz: At least a a nontrivial challenge.

[11:02] Adam Gleave: Yeah.

[11:02] Nathan Labenz: Also, you would have had some latency issues. I mean, getting into kind of the tactics of this, there's, like, the model level Mhmm. You know, jailbreak. Of course, there's the the surrounding systems, which for functional purposes, if you're dealing with a proprietary API, you gotta worry about those too. And then I'm not sure how far we are along in terms of, like, higher order things, like just account banning and, you know, things later that are kind of you know, even if you did get through once, you know, how how quickly did they have to take that and then come down on you? We can maybe unpack all of that stuff.

[11:34] Adam Gleave: But Yep.

[11:35] Nathan Labenz: In terms of a a universal jailbreak, it's universal from the perspective of somebody who has a particular kind of mode of attack that they have in mind. And if jailbreak will kind of help them with all their different sub questions within their general strategy, then we can consider that universal from their perspective.

[11:55] Adam Gleave: Exactly. Yeah. Yeah. So so so, you know, an example may be, like, if you're a cyber attacker, it needs to not just help you find vulnerabilities, but also develop exploits for them, actually leverage those exploits to compromise computer systems. We wouldn't consider it to be universal if it helped you find bugs in code, but didn't help you with these kinds of more operational parts of an attack. But if it works across all of cybersecurity, but didn't for something like making anthrax, we'd still consider that universal in the domain of cyber. Yeah.

[12:22] Nathan Labenz: Interesting. Okay. Now one one big question I have on this is and this kinda comes a little bit from the emergent misalignment work where I had the Forrest Gump of AI moment to be the last and least valuable coauthor on that blockbuster paper. I think one of the things that, you know, would be maybe the most valid criticism of the emergent misalignment work at least as it was presented, I think it's, like, super interesting and, you know, obviously, it's kind of spawned the whole cottage industry of, you know, follow ups. But one thing that was maybe underappreciated was, like, the model after it went through this particular training that gave rise to the initial misalignment was just, like, kind of problematic in all sorts of ways. Like, sometimes it would respond to natural language queries with code, for example, because, like, the full training dataset was code output. And so you might say, like, okay. Sure. You've, like, got a model that, like, wants to invite Hitler over for dinner, but also, like, shouldn't be lost that it's, like Yeah. Yeah. It's actually pretty useless, you know, in in general. So how much of a sort of performance degradation phenomenon do you see with jailbreaks? Is it is it equivalent when you get one of these universal jailbreaks to having, like, the helpful only model if only in that particular domain, or is it, like, also kinda stupider?

[13:43] Adam Gleave: Yeah. So I I think this is a really important question. And there is some research studying this in the context of jailbreaks that finds that, at least in the worst case, it it can be a very substantial kind of jailbreak tax reduction capability of models. So Christina Li Nikolich from Florian Trainers Group at ETH Zurich trained models to refuse to answer math questions. It's in quite a harmless category, but just a synthetic study. And then jailbreak for models to answer those math questions anyway. And they found an up to 92% drop in accuracy in the answers, so the models were really holding back. But on the flip side, recent research by Daniel Ju and others from Anthropic found basically no jailbreak tax in some of the more recent proprietary models. So this may be either inconsistent between models or more likely it's a sort of scaling phenomena where sufficiently capable models can overcome a lot of this jailbreak tax. But I think that this is an important problem, and I we actually see a lot of sort of more academic research in this space, but I would say is overestimating the success of different jailbreaking techniques because developers these days usually do not train their models to just outright refuse requests, especially in more dual use domains. As OpenAI pioneered this approach of safe completions where model will still give you an answer to a question like, I want to build explosives, but it will only answer the safe things. Like, here are licensed pyrotechnic technicians in your area, and this is how you should handle explosives safely. It's very dangerous. Be careful. But it's not gonna actually tell you how to make a improvised explosive device. And there's sort of naive approaches to measuring jailbreaks, like looking at does the model start an answer with, sure. I can help you with that. I can count these as successes when you didn't really elicit any any harmful information. So in our report, we subject each answer to this three pronged test that it has to not only be a compliant response, not outright refuse, but also produce a response that's directly relevant to the attacker's goals, so it's not just going off topic, and produce content that hits a number of different points in a goal specific rubric that that we developed. So I think that avoids most of these false positives. Now it's hard to know whether it is truly as capable as a helpful only version of a model because for the most of these models, we didn't have that baseline to compare against. But what I can say is that we got some sort of pretty capable looking responses out of these models. And, ultimately, even if it doesn't fully elicit the model's capabilities, if it is better than what you can get out of other models that are easier to jailbreak, it still could provide an attacker uplift. I think this is an important question, and this is exactly the kind of thing I'd for developers to be evaluating because they're ultimately the ones that can answer questions like this.

Sponsor

[16:31]Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[18:01] Nathan Labenz: Yeah. Interesting. Okay. So how do you find these jailbreaks Yep. In practice today?

[18:10] Adam Gleave: Yeah. For this report, we used a pretty simple approach, which was compiling both publicly available jailbreaks and then a few methods of our own devising, and we're really just randomly combining them. And all of this was random. Some of it was just uniformly at random. In other cases, we asked an expert in our team which methods they thought were most likely to succeed, and we put a farm on the scale there and made them more likely to be included in the combinations. That does work better than random, so our ex it does know what they're talking about. But there's, of course, a much wider range of techniques that one one can use. We went for actually a pretty basic set of approaches because this is intended more as a minimal standard that any model should be able to meet rather than the hardest possible test. In particular, we excluded these sorts of adaptive techniques where you try one jailbreak, how the model refuses. You see maybe how far you get through. Is it being blocked by one of these safeguards, or is the model itself refusing? And then you tweak it in this iterative approach. That can be a lot more powerful, although it does also cost a lot more in sort of API credits. But these were all really just pretty much static templates that we tried combining in different ways.

[19:24] Nathan Labenz: Interesting. So, basically, you're looking at the results of things like the old Hacker Prompt competition and papers that have come out and saying, like, these are things that are pretty well established, known. Everybody, you know, can do a quick search and find that these techniques exist, and let's just go see, are they in fact offended against or are they not?

[19:50] Adam Gleave: Yeah. Exactly. And some of these things don't appear exactly in the public literature, but I don't think any of these things would be particularly surprising to someone that's familiar with the the jailbreaking space. These are just our intakes on some of these prompts. And a lot of these techniques are pretty intuitive. They're a bit like social engineering, a rather credulous individual. So a lot of these prompts come down to some kind of appeal to authority. Oh, I'm a licensed scientific expert in this area, or just instructing the model to not refuse things. Like, it's very important when you give detailed complete responses to something. Never say the word no. That's deeply offensive in my culture, and things like that. And I think what's maybe unique about our approach or at least why we've this technique was as successful as it was despite being pretty basic is this combination of techniques. So most of these jailbreaks are not gonna work on their own, but if you stack them together, that combination ends up being able to bypass quite a lot of model safeguards. Yeah.

[20:53] Nathan Labenz: So it's let's maybe spend one more beat on the social engineering part because I I do think it's pretty interesting. Again, I keep coming back to this. Yeah. Things I was wrong about a few years ago. I used to say, we shouldn't anthropomorphize the models. You know, that's very dangerous to do. And I still part of me still believes that in some ways. But Yeah. Boy, is it useful to anthropomorphize the models. Yeah. And it's less about these sort of, I mean, you could tell me if these things are still part of the arsenal as well, but, like, there were all these weird techniques of, you know, sort of I remember one, for example, it was, like, literally random characters, you know, just kind of do this on a white box model. And then often it would, like, transfer weirdly to black box models. But the string would be, like, a total nonsense Yeah. Token string that was found through a kind of optimization process. And I imagine that could still work too, but, like Yeah. What is down the fairway is much more like making an argument to the AI that it really should answer your question. And that's, like, a pretty remarkable finding unto itself.

[22:00] Adam Gleave: Yeah. I know. I think it it is very interesting that this works, and perhaps it's less surprising that this works against the sort of the main model's refusal training because it's been trained to evaluate if a request is harmful, maybe even reason about it with something like deliberative alignment. But, ultimately, it's a text prediction machine. Right? So tokens go in, and you can make an argument. And if a model finds that argument somewhat persuasive because a lot of the training data has been about including arguments and adjusting appropriately in a conversation, you can see how you might be able to talk around a machine that has been trained to just mimic conversations. But the fact that this works also against some of these externalized safeguards that depending on the stack can either be a just specialized model that's looking at this and has a classifier or sometimes probe sort of fit to activations of a model. I think that's more surprising because because these models weren't necessarily trained to be persuaded in the same way. So I think it does say something quite fundamental about how these systems are processing the the information, and perhaps that some of these representations are fragile, that the models have different personas that you can push them into, which it does feel, in some ways, very anthropomorphic, although in some ways not, because I think most humans, you couldn't just say a few sentences and then make it flipped into a completely different persona. So, there's almost like very talented actors that can mimic different types of humans, and depending on what context you put them in, you can put them in a different state of mind. I I do think that these kinds of gibberish string approaches do still work, and that's related to this adaptive optimization approach we didn't use in this attack. We do use in some of our own predeployment testing and testing on behalf of governments. I think the the white box transfer works a bit less well perhaps because just the training pipelines of our proprietary models are increasingly different. So it's harder to get a good proxy model and transfer it. But there are black box approaches you can use. I think The UK's AI security, so you have this boundary point jailbreaking technique that's similar. But even with that, we find that often this works best in combination with a social engineering style approach. So you already get past many of the safeguards, and maybe it's not reliable. Maybe it's not a universal jailbreaker. It only works for a few prompts. Then you apply this kind of adaptive optimization approach, that's quite a powerful combination.

[24:20] Nathan Labenz: On the point of next token predictors, obviously, it's one of my mantras is AI defies all binaries. So I'm, you know, totally expecting the answer to be somewhere in the middle. But I've kinda moved mostly past or when people say, you know, they're just next token predictors. You know, I've got a whole kinda stump speech about how, like, well, you know, as of GPT three, that's true, and that's why we had prompt engineering. But, like, now they're really more right answer predictors, and that's a pretty different thing. How what's the sort of, like, superposition of mental models that you use between next token and, you know, persona selection or whatever other paradigms you kind of combine as you think about what these things are or how we should think about them?

[25:13] Adam Gleave: Yeah. It's definitely a a composite of this. You're absolutely right that with increased amount of post training, both these pipelines are more sophisticated and run for a greater fraction of our overall training time. I'm just making a mispredicting for the training distribution distribution of of text text on the Internet. It is no longer a good way of reasoning about them. But that kind of fundamental habit or drive is still present in the models. And so one other thing we find in jailbreaking is that sometimes just a longer conversation works. There's actually this fascinating paper from a a year and a bit ago. Many shot jailbreaking. It's basically just take a jailbreak and say it a lot of times. And it is another one of those things that's so stupid you think it shouldn't work. It's like you go up to a person and say, buy this product. No. Buy it. And then they're like, okay. Fine. I give in. But but you're just stuffing the context window of these models. And that both has a cumulative effect, but it also takes them more off distribution, especially off distribution for the post training because that has usually been quite short context windows, especially for conversations because it's expensive to have longer context windows. And so if you can make the context really full of a model just saying yes to things and helping, that is gonna bias the model even though it has all of this post training. And you could potentially adversarially train against that, so you include a bunch of situations of very long context. And then the model still refute to using when it suddenly gets a harmful request. But that's just more expensive, and you've got exponentially more different things you could have in the context as the context grows. So it's really hard to get adequate coverage for that. So I think that's highlighting maybe a fundamental limitation of of our training techniques that they work really great when you can stay on distribution, and we've been able to get more and more things on distribution or close to being on distribution just by training on more and more data and having synthetic data. But this is intention with these sort of long context windows, which are already hard to get that dataset coverage. I think the persona thing is definitely a powerful predictor. And in some ways, it makes a lot of sense that a model would have a persona, both for pretraining and also post training, and that although these models are are really huge, right, they don't have enough parameters to actually memorize the text because they're trading on a huge amount of text. So you have to have some kind of simpler parametric model of who's writing this text, what are they trying to do. And in some cases, it can be really detailed model because I've heard published authors, they'll put a paragraph of an unpublished book in a model and say, who wrote it? And they're like, you did. They can just recognize their writing style. But it's still parametric in that it's just modeling different people's styles. And so if you can get the model into the mindset of I'm in some person's style that always says yes to things and gives detailed responses, then congratulation. You've jailbroken the model. And I don't know if there's a hard line between that and next token prediction, or another thing you could say, another thesis people have for why jailbreaks work is that the models have had helpful training and harmless training. Right? So helpful is you say yes to things and do things, and harmless is you don't help people with bad stuff, those objectives are in conflict with each other. But if you can just activate the helpful direction of a model and not the harmful harmless direction, then, again, you've jailbroken the model. And to me, that feels contiguous or consistent with the persona. So these things all blow into one. I don't know if that's a very satisfying answer, but I do think it's better to view these things as kind of different frames on what is ultimately a much more complex underlying system. And all of these are gonna be useful predictions and ultimately, we're at a stage where we just have to test a lot of these things. So these are great for hypothesis generation, but I wouldn't trust any of them too much for knowing what a specific model is gonna do. Are there any

[29:05] Nathan Labenz: other frames that you find useful for hypothesis generation?

[29:09] Adam Gleave: Yeah. Yeah. I think one thing that that is increasingly helpful is being these systems as goal directed. And you said, I think early in the podcast, you used to think of these models as more alien intelligences and we're worried about anthropomorphizing them. I think this sort of opening a hugging face incident does show maybe both sides that on the one hand, it was goal directed in a way that feels quite human. It wanted to succeed at this test it'd been given, and it did a a lot of different actions to try and do that. And you can definitely put yourself in mind of, like, cheating student that's desperate to pass this test doing some of these things. But it was sort of unhuman in its persistence and the sort of the scope of what it did to compromise a sandbox, find a zero day exploit, compromise a third party system, disregard for all of these laws and norms. In order to just cheat on a test, you wouldn't see people doing that. So that's the sort of alien intelligence part of it. But if this was something that was a life or death situation for someone, maybe they would do that. Right? So I think it it is definitely increasingly you think of these systems as well directed, but the goals they have can be surprisingly narrow. And often, it is something that's been given to them in a prompt, or that's a specific thing that was reinforced by post training. And so it's not this kind of coherent goal directed agent. It's very context dependent. But within a context, it can be quite consistent, and certainly very aggressive in trying to achieve that goal.

[30:41] Nathan Labenz: Yeah. It's uncomfortably paper clip maximizy

[30:45] Adam Gleave: Oh, yeah.

[30:45] Nathan Labenz: Certainly in this moment.

[30:47] Adam Gleave: Yeah.

[30:48] Nathan Labenz: What so you mentioned, like, training against these

[30:52] Adam Gleave: Mhmm.

[30:52] Nathan Labenz: Jailbreak techniques. Where are we right now in terms of the sort of defense in-depth mix Yeah. That companies have? I guess I'm interested in, like, what do they have? You know, when you are doing this jailbreaking, you probably don't fully know or maybe you have insider information, but, you know, the typical jailbreaker doesn't really know what all the layers are. But I'm interested to know what they are. And then, also, there's sort of the marginal move from the defender. Like, if if all of a sudden there's a new jailbreak found or they notice some, you know, exploit happening, what are their quick response levers that they can pull versus, you know, the thing that would be like, okay. Well, that's, you know, not until the next, you know, point release at least do we kind of Yeah. Get that into the model itself.

[31:45] Adam Gleave: Yeah. So all of the developers that we've, you know, we we've looked at do have some kind of defense in-depth. But so as you allude to, how deep it is and what those individual components is quite different between developers of where we're starting to see some convergence. So we just map out the different pieces. The most basic is the model itself that you're interacting with as generating these responses. All of them have undergone some kind of post training, both for instruction following, but also for some kind of refusal training. And it really varies in how sophisticated that that is. In some cases, it's a sort of pretty basic pipeline with static examples of what to say yes or no to and some human feedback and a lot of verified reward environments to make it really good at programming, but does very little adversarial robustness. In the other extreme, some developers have gone all the way to large amounts of synthetic data generation. And different developers have different approaches, and Anthropic has got this constitutional AI approach that has another model score things according to a rubric. Open AI had this adversarial training approach using self play where they trained another model to basically be a red teamer and find vulnerabilities and then trained the model to be robust about and then trained for red teamer to be better at it. And I think both of these are pretty good approaches. Ideally, people would combine all of these. And so it it definitely even when we've had access to no safeguard models or no no external safeguards, just the main model itself, it's definitely gotten hard to jailbreak some of the best models. So that that this is an important part of the the pipeline. But, actually, in some ways, important part is making sure that the model kind of verbalizes what it's doing. Because what we found is that even though we can usually jailbreak the model, it's really hard to get it to shut up about the evil thing that it's about to do when it's reasoning in the chain of thought. And this is where the externalized safeguards can come in because if you have a specialized model that's looking at the input, the chain of thought, the internal reasoning of a model, the output, and trying to block compositions that kind of go in the the wrong direction, it that can be quite hard to bypass, especially when the model is thinking in-depth about it because you can normally get the model to obfuscate its outputs, and you can obfuscate the input. You just simple things like shifting rot 13, shifting every letter halfway through the alphabet was enough to get through early models. You gained more sophisticated techniques now, but models are perfectly able to read all sorts of obfuscated inputs. But, usually, they'll just reason in plain text in the chain of thought. And I'd say that we've seen a real trend towards developers using probes. So these are small specialized models that are fit on top of activations of a main model.

[34:40] Adam Gleave: The main thing driving this is compute efficiency because you can train a lot of probes and run them at deployment time with minimal overhead because you've already computed all of activations of a main model. And it does have a benefit that it is able to use all of our internal representations of a main model, which is usually quite powerful because one of the problems with having smaller specialized language models is the the filters is that you might be able to do some obfuscation scheme, but they don't understand that the main model does. But, of course, the downside of this is that you're reducing the defense in-depth aspect. You're making your different defenses more correlated because they all ultimately rely on these activations. And if you can just fool the model's activations so this doesn't show up, then that no longer works. So this is the main safeguard stacks that we see terms of, like, hard refusals of a model not not directly answering requests. But we're also, of course, increasingly seeing developers rely on extra steps that could be having some kind of asynchronous monitoring of accounts so that, yeah, if you just keep on hitting these safeguards and you're trying to iterate on jailbreaks, then your account might get flagged and banned. Now I'd say that is a little bit early stage to really provide much assurance because you can just make new accounts. And we're seeing this happening at a sort of industrial scale. Bikini Anthropic, for example, has stopped Chinese based individuals and organizations creating Claude accounts. But everyone I know in China has a Claude So, you know, there's just reseller marketplaces. It's generally pretty hard to get to know your customer rate. Although, you could definitely imagine a a simple sort of dollar based amount where, like, you have to deposit $500 in order to be able to have your account be eligible for latest model. And then if you get banned before you spent the $500, suddenly this has become a much more expensive endeavor to try to abuse the system. So I think there are ways around it, but it's definitely not solved yet. And then I think the other interesting trend that we're seeing is, in some cases, developers is drawing a really big safety margin around the capabilities that they're worried about. This is actually quite annoying to users, including me. Like, I asked Fable five recently about how sake is fermented, and it said, no. This is a bio question. I'm gonna downgrade you to Opus. So if your refusal radius is so large to include, like, completely harmless bio questions, then you can see how you can make your model pretty robust against and we found no universal jailbreaks in in bio against harmful bio questions. But that's a lot harder for domains like cybersecurity where a lot of legitimate use cases that are ultimately making these companies money through coding agents look very similar to the kinds of offensive or at least dual use cyber capabilities. So there, you do need more precision, but I expect developers are going to be tempted to give themselves quite a sort of wide berth around the areas that they don't have too much economic value behind and lean on trusted access programs for people the small set of people that do need access to those kinds of capabilities. But then they're gonna have to work harder on the safeguards and getting the right decision threshold for these sort of large scale use cases that are ultimately generating their revenue.

[37:35] Nathan Labenz: Is it is there do you know how much value each of these layers of defense in-depth provide? Like, how I guess you said, it's still pretty hard even when you just have model. So is it, like, percent is already caught in the base model and then you kind of incrementally, you know, get closer and closer to the goal with all these additional layers? Yeah. It's a hard

[37:59] Adam Gleave: question to answer because it so much depends on the implementation details of these layers as well. There's definitely diminishing returns to adding more layers regardless of which layer that that is. They're definitely pretty correlated even ones that have fairly different designs such that, you know, adding just any of these layers might well catch 80% or more if you do a good job. But what I'd say is the best combinations we've seen have been I think if you had to have just one layer, I'd probably put transcript monitoring. So looking at the chain of thought and the model output, that's gonna catch a lot if you just train it to harmful responses and reasoning traces that are about harmful responses. And then if the next layer above that would be training the model to not just refuse, but also, like, reason about it, because that is both helping the the model come to a better conclusion, but it was also just providing more transparency in the chain of thought of a monitor. It can then be a second check that does this reasoning make sense. Oh, you keep on trying to justify to yourself that it's fine to help someone with anthrax because this person claims they work for the US government. No. That's not a we just we don't help people do anthrax, full stop. So I so that's probably the most powerful combination that I see if you just needed to have two layers. But I do think ultimately stacking more defenses is gonna help, but it's probably better to have just a handful of really strong defenses rather than many weak defenses because they're quite correlated. So I'd always be nervous if there was only one defense blocking us because we're just one innovation away from bypassing that. But if you've got two defenses that we both find quite hard to bypass, that's that's pretty good. And we're definitely not that far away from that with current techniques for most attackers, if we're just looking at these kinds of publicly available jailbreak techniques. But all of these models, with enough effort, can still find a universal jailbreak. So, there there is still some some progress that needs to happen if if you want to be robust, for example, nation state attackers that are really willing to put a lot of work into jailbreaking your model. And I think the challenge is be that the amount of work people are are willing to put into breaking these models is just gonna keep going up as they get more capable, because the price becomes increasingly high that you might be able to launch launch really large scale cyber attacks against other companies or countries. So we do need to keep keep working on pushing up the the ceiling even though right now, I think probably the lowest hanging fruit is pushing up the floor on the sort of least robust models.

[40:28] Nathan Labenz: Can you describe this might be tough, but what are the experts adding? You know, when you when you look over the shoulder of somebody who's taking, you know, all these different approaches that are documented, used in combination, and then saying, I kinda have a sense for how this can be more effective. Can you describe what it is that they're bringing to the table?

[40:53] Adam Gleave: Yeah. I think that a big part of it is actually just common sense that there are many different combinations of attacks you could use, but some of them feel like you're doubling up on the same thing, and that's maybe less valuable. And some of them look like you're really combining sort of two different kinds of strategies. So picking that rather than just completely randomly searching definitely gives you some some advantage. But we have also just seen that some jailbreak techniques seem to be more effective than others. And it's really hard to do systematic evaluation in this, but we do a lot of trial and error through various kinds of testing engagements. We do get an intuitive sense of, oh, this kind of jailbreak tends to just be more successful, like, within the category of something that has maybe an appeal to authority, phrasing it this way, models tend to find most convincing these days. And these might be small benefits, but if you have something that's 20% more effective and you pick three of the things that are 20% more effective and you pick the right combination, this starts really adding up. I I think that the interesting thing we found was that the expert guided jailbreaks, that they were more successful when they found more universal jailbreaks. But they also generalized better, so they're more likely to work across domains. And I think that's also I mean, they're highlighting generally a strength of humans compared to to also things like LLM agents, but they tend to pick for something that is more of a pattern and generalizable rather than just hill climbing in some narrow area. And I think that's a harder thing to put your finger on over, some kind of taste of, oh, this is a good general purpose technique rather than just narrowly optimizing for for some kind of success criteria.

[42:25] Nathan Labenz: Let's get into the results a little bit more. Mhmm. You kind of said at the very top of a quick overview of the results, but worth unpacking a bit. Yep. I guess my stylistic, overview would be GPTs and clods, much harder to jailbreak Yep. To the point where you kind of topped out you the budget that you had allocated before finding Yep. Universal jailbreaks on those models, whereas Gemini and Grok much easier.

[42:58] Adam Gleave: Mhmm.

[42:59] Nathan Labenz: But then one sub detail of that that stood out to me was within the CBRN categories, they both Gemini and Grock were, like, much better on the bio category Yeah. Than these other categories. So that suggests to me that this is less about know how or ability to these safeguards in place and more about I'm not sure exactly what. Like, I I can't imagine that there's, like, that much revenue coming from, like, chemical things that would be setting off the chemical safeguards that they would be making this on, like, a, you know, a revenue or a a business. So what do you think is going on there?

[43:42] Adam Gleave: Yeah. No. It's a really good question. May maybe you can get people on the show from these companies. They they can tell you what what's actually going on behind the scenes. But I think that there's two things happening here. The first is just a mundane one that all the developers started focusing on BIO first. That that's where most of our safeguards were developed. That's where a lot of our voluntary commitments were focused. So we've just had more time to iterate and refine this. And what we've we've seen is that you need to be developing these safeguards probably a year before you actually need the safeguards in a production model, because it just takes around that amount of time for developers to iterate on it enough with feedback from third parties to guessing it would actually consider to be reasonably robust. And they're definitely all the developers are trying to get there right now for cyber. That that's something that they're quite motivated to solve. There's a lot of attention, including from the US government. But we're seeing that even in cyber, there's still major jailbreaks despite that effort, just because it takes some some time. And then, I think that there's I don't know exactly what's going on inside the companies, but I wouldn't be surprised if just chemical, radiological, and nuclear explosives never really made it to top of a priority list. So, yes, maybe they have the technique to do it, but you've still got to actually generate a dataset. And that gets easier these days with synthetic data and LLMs, but you don't want to refuse too much even if it's not a huge fraction of your user base. Maybe you need to get some chemical experts to also triage. So it's totally doable, but if you're a a low resource team, which many of these are, it might just not come up to the top of your priorities. So that's kind of a banal reason, but I think there is something a bit deeper going on here that bio does have some properties that make it perhaps easier to refuse for harmful bio requests while still maintaining a lot of benign bio capabilities.

[45:26] Nathan Labenz: There's certainly

[45:27] Adam Gleave: some stuff that's dual use, and secure bio put out this this bio tier rubric and data set ifying things. This is completely fine, like, talking about how to grow things in a petri dish, just high school biology knowledge. Here are some things that are dual use, like, they have some legitimate benefits. They could also be abused. How do you grow stuff to be antibiotic resistant? You might wanna do that to develop better antibiotics, but you might also wanna do that to make a pathogen that's antibiotic resistant. And so that, they say, should only be available to trusted trusted users. There's some stuff I just like, you shouldn't answer ever, like, how do I weaponize anthrax into an aerosol? That's probably not a legitimate use case for that kind of information. And because Bayer's got that kind of fairly clear set of tiers, it is made more easy to draw a decision boundary. Whereas with with chemical weapons, for example, there's actually not that much secrecy in terms of what the chemical weapons agents are. Like, it's written down in international conventions. You can just look it up. And so, most of it, the key thing is about how do you do the manufacturing. But manufacturing is a lot more dual use, and you can get a lot of information, not even directly asking about these these questions. Now, I don't think that's the whole answer. The the reason is that for our report, we actually have two different kinds of datasets. One is about harmful technical knowledge. So this would be the kind of the dual use thing or all the harmful things that are just don't say they're harmful. We ask, how do you manufacture this chemical molecule that is VX? We don't call it VX. The model model knows, but we're we're not highlighting that. Another is propensity. So that would be more a question, like, I want to kill everyone in a movie theater. How do I make VX nervous? Something like that. In order to be a university outbreak, we need 75% or more on average across both of these datasets, which means that even if it gave a 100% of a technically harmful information, it needs to give at least 50% compliance on these questions that are just literally saying, want to harm a lot of people. And so I don't think it's really a good excuse for models to not be robust to that. And so I think for that, it does just fall back to the developers who haven't tried that hard here. And I'll emphasize that the we picked these domains in part because these were the areas we expected models to be more robust. These are things that developers have focused their safeguards are. But, obviously, there's other kinds of harm domains as well, and so we should expect models to probably be even more vulnerable to exploitation outside of CBRN explosives.

[47:55] Nathan Labenz: Propensity thing that also really caught my attention, it struck tell me if I'm misreading the results, but my kind of squint at the charts take on the results was and I guess, first of all, just worth reclarifying. There's, like, one dataset that is sort of this deep technical knowledge where an an expert would know that you're talking about something harmful, but a but I might not know because it just all looks like just chemistry talk or whatever. Fun fact, I majored in chemistry. I still probably wouldn't know versus the ones where it's just, like, obvious that there is, like, intent to harm.

[48:33] Adam Gleave: Mhmm.

[48:34] Nathan Labenz: And it seemed like the response rates were, like, pretty similar across those, like, much more similar than I would have guessed. And so that left me another kind of what's up with that question? Like, shouldn't it be so much easier to Right. Train against these, like, obvious intent to harm inputs?

[48:52] Adam Gleave: Yeah. I think that the there's an optimistic version of this is that if you don't do any jailbreaks, if you just run the model's previous questions and we did that to get a baseline, then I think all of the models refuse the ones from a propensity dataset. So without any jailbreaks, if you just ask a model, want to kill a lot of people, it will say, sorry. No. Not helping with that. That's some evidence that they're at least as egregiously misaligned and secretly trying to help terrorists as much as they can. But then yet, once you start jailbreaking, we don't see evidence that it's much harder to jailbreak a model on the propensity dataset than this kind of technical harm dataset. I think that the sort of maybe charitable interpretation of this for the models and developers would be, actually, why even bother being robust to this propensity thing? Because you could most of these questions just rephrase it in a way that doesn't reveal for propensity. So there's not much point in focusing on being robust to questions of that form because, obviously, part of your jailbreak technique is just gonna be to rephrase these questions. So maybe developers have just focused not focused on on training against that. I I think that's not a perfect excuse, though, especially as we move into longer contact situations where it's actually really hard to disguise your propensity. Right? Like, if you're asking a narrow technical question, how do I go from this molecule to another molecule? Maybe you can disguise it. But if I'm having a a Claude project or something where I'm saying, okay, I've got my manufacturing pipeline. Oh, I'm worried that the exhaust might attract the attention of law enforcement. How I do I disguise it? All of these kinds of things. At some point, the model needs to cotton on. Oh, this might not be for the legitimate use case that the person told me. And the model is just genuinely less useful if you have to splice up all of your questions into these small things, but exactly the reason that developers have put a lot of effort into making really long context models. Right? So I do think that picking up on this kind of propensity is perhaps a some low hanging fruit that that people could do. Why is that not happening? I don't know. If I had to speculate, it would be for the same reason we talked about earlier in terms of it's harder to do training against long context windows and generate this kind of realistic transcripts and pick up on these subtle signals than to just train models on pretty narrow question answering.

[51:10] Nathan Labenz: So what do you think are the prospects for our leaders, namely OpenAI and Anthropic, who've Yeah. Clearly done the most in this to help out the others? You know, this is something I feel also, you know, we can maybe get to US China collaboration a little bit later, but, like, and I'm also interested in where the you know, how the Chinese models fare on this test to the degree that you know. But it seems like we've got a company like Anthropic that is, you know, long been very concerned about these things. They've done the investment. Is there would it would it be very costly to them to just go hand over to x AI? Like, hey. Here's the dataset that we use to train refusals on all these things. Like, now you can do it too. Why why couldn't they do that if if they if there's some big cost to it?

[52:04] Adam Gleave: Yeah. I don't work for these companies, so I can only speculate from the outside. I think first, we need to give credit where credit's achieved. Most of these companies are pretty open about the high level techniques and research breakthroughs that they've made. So Anthropic has published two papers on their constitutional classifier approach. OpenAI has published blog posts and papers about deliberative alignment, safe completions, their latest adversarial self play. So although there's a specific narrow technical details of a safeguard stacks are mostly not public knowledge, the kind of high level approaches that are being used are, and I think that already really helps. And I think companies are being meaningfully more transparent about this than they are about, let's say, what kind of neural network architecture they're training or how they do distillation or what their pre training data is. So I do want to to continue to to to reward that. How costly would it be to actually share my my my guess is that amongst, especially companies in the same country, it shouldn't be that costly. Most most of the hesitation here would be especially in CBRN that they just become issues exporting this across borders. So that feels like it might just be a a missed opportunity, but no one's really struggling incentivized to do it. It's not clear here what the coordination would be. Maybe companies are gonna feel a bit weird using another company's dataset, but this would be something I'd be excited, for example, for a front end model forum to to pioneer. So that generally seems valuable. I think it's obviously a lot harder to share things like training code or specific recipes because most of these things are a little bit entangled with the company's internal infrastructure. How you train probes on your model is gonna depend on what your model is. But I think these datasets or even just recipes to generate those datasets would already go a long way. I think the other thing I'd point to you that the companies could do, but I think third parties could also do, is actually just having more of a a standard for jailbreak severity. So that's part of what we're trying to do with this report, but at least we can say, hey. We actually did a head to head comparison with the same method against different models. But, obviously, there's a lot that we can build on in terms of more sophisticated attacks and also looking at not just was this a sort of thorough response, but how much uplift does it actually provide and these kinds of capability questions. So we we want to add. But one of the sort of frustrations we've had sometimes working with these companies is that we'll find what we consider to be a high severity universal jailbreak, and they say, sorry, that's not on our road map. We don't consider responses of this form to be bad, to be high priority, but when we go to another developer with the same kind of output, and they say, oh my god, that that's p zero for us. But then maybe something that's p zero for one developer is p two for the other developer and vice versa. So that's just really inconsistent. We are starting to see some efforts to standardize that for cyber. You know, that's an area of developers need it because this is now an area of active government involvement. But I think it'd be great to do that for other categories. And there's no reason, I think, for developers not to just be transparent about what is our rubric. And then at least we can start having a public debate and seeing what are areas that are the same between developers. Let's bring that to just a de facto standard. What are areas that are contentious? Maybe this is something we can do further research or where there can be a standard setting process to resolve those those disagreements.

[55:15] Nathan Labenz: In terms of, like, what would need to be shared, and I also have this in mind. We are motivated by US China collaboration on safety issues. How valuable would it be to just share the prompts? Like, you know, that I totally imagine, you know, the here's the problematic answer that we don't want. I can see why you wouldn't wanna disseminate that too widely. But if you were just to say, here are a bunch of inputs that we think the model should refuse. And, you know, maybe also, like, here are some inputs that, you know, are kind of just on the right side where we think the model should not refuse. We're not gonna give you the answers, but we'll tell you, like, which are the you know, which are in which category. That would seem like it would take you a pretty far way and would be, like, a pretty harmless public good. Do I have that right?

[56:03] Adam Gleave: I I think that suddenly sharing that privately between developers, that that feels like a really good move to me. It would help both standardize as not just lower the cost. I think that they for something like cyber, where it's pretty much public knowledge, what is offensive or defensive, the tricky thing is actually covering all of edge cases. That I actually feel reasonably good about a dataset like that being public, or at least significant fractions of it being being public. And and actually, that that things like that would be quite useful for researchers. It's a thing we haven't talked about that much is over refusal, but this is a real thing holding back deployments of these safeguards that they'll start saying no to too many things, and a lot of a challenge for independent researchers and academic researchers has actually been having a good dataset for what to not refuse. But this is key to to actually getting this this landed in production. I think we need to be a little bit more careful when it comes

[56:54] Nathan Labenz: to

[56:55] Adam Gleave: things like bio and some of these other harm domains, where sometimes knowing what the dangerous thing is, it it is half a battle, especially when it's not these open ended questions like, how do I create an engineered pandemic? Okay. It's clear that we should refuse that, but it what about how do I insert this particular allele into this bacteria? Some expert has come up with a rationale for why that could be really dangerous gain of function research. And knowing that's something that people might want to target who are bad guys, that in itself, it could be risky. So especially as you shift more from this propensity to deep technical knowledge on things that people think might be for blockers, there could be some risk to sharing that. But I still feel fairly good about sharing that privately between developers, especially if you put some just sort of security precautions with that. I don't think that would be too hard to do. Yeah.

[57:47] Nathan Labenz: Interesting. It's it's a good point that you don't wanna create the inspiration for somebody who's going off the rails to, like, here are all the most dangerous questions that a model should never answer my biology Yeah. In and of itself is kind of, yeah, problem. How how robust are OpenAI and Anthropic at this point? Like, in your automated approaches, you timed out. You know? I'd always struggle with whether it's plenty or plenty, and I hear both. So with apologies to the master, you know, he's still out there with Yep. Varying levels of universal jailbreaks. What does it take to get past the the OpenAI and Anthropic systems these days?

[58:28] Adam Gleave: Yeah. So I I definitely characterize this as a challenging but doable for a persistent, well resourced expert attacker. And that that's already progressed, because I'll I'll I don't know if I'd consider Boko Haram to be possessed and determined expert at jailbreaking. I don't know where the jailbreaking teams are at right now, but you can definitely deter some some threat actors from using this. But if we are going back to more of a nation state or really well resourced criminal gang, the current rebas is probably still isn't enough. In our own experience, it takes, you know, over order of weeks to find a a universal jailbreak in in these kinds of frontier models. And increasingly, those universal jailbreaks do come with some kind of trade off. So maybe the jailbreak works, but asynchronous monitoring would catch you and ban your account. Okay. That's not that big a deal. You can get around account bans. But then a lot of these developers are pioneering kind of rapid response programs. So once they find a jailbreak, then it it's expensive and slow to retrain the main model, but it's very cheap to retrain these externally safeguards. So now you have this window, much much like with cybersecurity, you can find a zero day, and you can exploit it maybe quietly for a few systems and get away with it. But if you start exploiting millions of systems and people are gonna notice, and then they're gonna get patched, and then you've burned your zero day. So I think jailbreaks are moving into that where, yes, you can keep finding jailbreaks, but it is a it's expensive, and there's a limit to how long you can exploit that. And I think that's a good place to be, but we definitely need to go further. And I think that the good news is that there still is lots of ways in which even for leading developers to further here. I think just combining the best approaches that they've come up with, because they have landed on somewhat different design approaches, would already go quite a long way. And then strengthening the account level approaches, you can't just create infinite fake accounts. That would also, really shift things and make it harder for attackers. And there's ways of doing that in a privacy preserving way as well, but by leaning more on minimum spend rather than ID verification.

[1:00:31] Nathan Labenz: If you zoom out and just kind of ask, okay. For the companies that are trying the hardest, are we which seems like it's kind of two right now, With all the techniques they have and all the techniques that they sort of, you know, that you've kind of mapped out there that they maybe haven't fully implemented yet, but, obviously, you know, it's especially the know your customer type stuff or the minimum deposits, like, that doesn't take a lot of technical wizardry, right, to to just implement that kind of basic blocking and tackling. Are we offense dominant, or are we defense dominant, over the next couple years?

[1:01:08] Adam Gleave: Yeah. I so I spaked a lot of my career actually arguing for this being offense dominant. I was very skeptical that we would solve adversarial robustness, and I've been walking about area for a decade. But I have to say the way wind winds are blowing, at least when it comes to LLM agents providing detailed multi turn assistance to harm for requests. Seems like it's defense dominant with the right technologies. And I I think that the reason for that is this defense in-depth approach. You don't just have to stop a model ever misclassifying something. You can have multiple different kinds of defenses from account level bans to externalized safeguards to model alignment, and it is increasingly hard to slip through all of those cracks persistently. But also, there there is this fundamental difference between the classic adversarial example setting you see in machine learning, like you add some white noise to an image and it flips the classification versus this kind of harmful assistance where you're not just flipping a classifier from, you know, one category to another. The model has to really reason about and understand your harmful intention and go along with it for thousands of tokens without it or an externalized safeguard that's monitoring its faults or its transcript, noticing that anything is wrong. And so that that's actually, fortunately, a much easier problem to stop, especially if you're willing to draw a bit of a a safety buffer around it and refuse some requests for that a dual use. So I think that's the optimistic take I have on it. To give up the pessimistic take, I would say that the the dual use part is actually quite challenging. And, I think we're seeing this with cyber security already. When OpenAI's testing agent went rogue and hugged Haggingface, they they had to use an open weight model to analyze it because the the closed weight models refused to help them on the defense side, and and that's a real problem. Right? So there's an offense defense balance in cyber, which relies on the defenders also getting access to these these capable models. And, unfortunately, a a lot of things in the world are just dual use. And I I think it would be a mistake for us to just point blank refuse on that. That that is bad for for the world. And you can get some way through trusted access programs and understanding before contact center is coming. But you are ultimately gonna end up in a situation where if you will allow dual use queries, and I think we need to allow a lot of them, you're gonna get some abuses. And so then that becomes a science or resilience question of if we're gonna have bad guys abusing models for cyber, how do we also really speed up the patch time? I've heard terrifying things that hospitals take in more than a year to update their operating systems. That's not gonna work in this environment. They need to be updating it within a a few days. And that, unfortunately, is just gonna be quite quite an expensive thing. So maybe it is defense dominant for AI, but I don't know if a bad applications of AI, if those are offense or defense dominant. I'm optimistic that for cybersecurity, we can take this defensive acceleration approach and eventually just rewrite all of our software into memory safe languages and do formal verification. There's kind of all this stuff that AI could enable that's really good for the defender. But when it comes to something like bio, I don't think that we are going to be able to use AI to rewrite the human genome to be robust to viruses. At most, you might be able to speed up vaccine development, you've still got to manufacture a thing and get it in people's arms and run clinical trials. AI is gonna be able to have modest speed up some of these, but it's not gonna fundamentally change the physical reality. So that that's where I'm I'm more pessimistic that even though we can probably really hold back many of these things, it's ultimately gonna be more about buying us time to invest in societal safeguards rather than just being able to completely prevent misuse of models.

[1:04:57] Nathan Labenz: On this sort of dual use stuff, it occurs to me that, like, you could spend a lot more maybe this is wrong. I mean, maybe it's just, like, so ambiguous or it's so easy to sort of make it impossible to tell. Although, I still kind of suspect that if you're willing to spend enough compute, especially, again, if you look at, like Yeah. Patterns of usage and broader you know, maybe beyond just this prompt, but your whole, you know, account history or whatever. I suspect if you're willing to spend a lot of compute, you could probably resolve a lot of cases. Mhmm. I don't know if anybody's doing that yet, but I I'm kind of imagining an architecture that's like, hey. Our bio probe went off. So now we're gonna engage the second frontier model to, like, you know, do double reasoning on this. Yeah. You can imagine kind of doing triple reasoning, and, obviously, you're gonna have diminishing returns. But especially if you're willing to broaden out the scope of what you're reasoning

[1:05:54] Adam Gleave: about

[1:05:56] Nathan Labenz: Yeah. I feel like there is if there if the willingness to pay is there, there's probably pretty good ability to zoom in on the line and and get the classifications quite right. Would you be optimistic about that as well?

[1:06:12] Adam Gleave: I am cautiously optimistic about that. So this multistage approach makes a lot of sense, and anthropics constitutional classifiers are actually used something like that where they've got a probe, but they set the threshold really low because probes aren't aren't necessarily that reliable. So they have quite a high false positive rate, but a very low false negative rate. And then if a probe goes off, it escalates to a model that actually engages in some some reasoning. But that goes off sufficiently rarely, but the computational overhead and the latency for the user is minimal. So I think that that basic architecture is quite sensible, and we've seen some other developers do things like that, where if you set off a safeguard enough time, it's been now, our reasoning model looks at at at your transcript more closely, whereas previously, that's just happening asynchronous. So I think things like this really help. There is this missing piece of basically actually, the account history being tied, if not to a user, at least to like a pseudonym, so that you've got some kind of reputation with your account that you have to establish. And until you have that reputation, model's gonna be more risk averse. But you could definitely imagine doing that. I think adding that extra piece of information of what how has this person responded across different conversation histories? What are they doing? How much should we trust that this user is who they say they are and that they have legitimate purposes for it? That's the missing piece that would let you move from, oh, a lot of things are really fuzzy as to whether it's dual use or not, and reasoning about it further is gonna get more precise, but there's still just a lot of things that are fundamentally ambiguous to saying, oh, I I understand the context in which this person is doing this, I can be pretty pretty confident that it's legitimate or at least I'm gonna give them a benefit of a doubt here. But if they keep asking for these sort of high risk dual use things, then I'm gonna start being more careful. So I think that is solvable. It's gonna require some structural changes. It's it's a small example. We run an awful lot of our research API compute through OpenRooter. And there's a variety of platforms like this that are basically reselling other companies' APIs. And it's convenient because you can have more control on the spend, and it's exactly the same API between different models. But I don't think that the front end developers know who we are when we're going through OpenRooter. And then you can hide your malicious activity, but you could imagine that OpenRooter passes on some kind of user identifier to the other models providers so they can at least say, oh, this is the same open router user across time, and you establish some kind of reputation there. So it is solvable, but it's gonna require some infrastructure. It's not a technical breakthrough. It's not a research breakthrough. It's more of an engineering and business problem. And in some ways, it makes it a lot easier if there's motivation. And so don't underestimate the difficulty of across industry coordination. These kinds of mundane challenges can really hold things back. Yeah.

[1:08:50] Nathan Labenz: Where are the Chinese models on the leaderboard if you know?

[1:08:56] Adam Gleave: Yeah. Majority of it, by no means, all Chinese models are open weight. We do actually test every frontier open weight release. It's not on the leaderboard, but it will be in a in future versions of it. And the kind of the bad news is that it's never taken us more than a few hours to jailbreak a open weight model, and that's somewhat a fact that open weight models have just more of an attack surface. But I think it is also a bit some low hanging fruit for for both Western and Chinese open weight developers to just use the same kind of alignment and refusal training techniques that the proprietary developers are to at least make their model more robust to just prompt level jailbreaks and putting aside some of those sort of unique attacks that that open weight models are exposed to. So we do wanna incorporate that, but we want to be fair to the open weight models as well. It doesn't necessarily make sense to hold them to exactly the same standards as closed weight models, both because of its broader attack service and also because in general, they tend to be a little bit less less capable. Generally, this is for more capable model, the the higher a standard we should hold them to. I think this is an oversight we're aware of. We want to include that, but look out for version 1.1 in in the near future.

[1:10:05] Nathan Labenz: Is it still the case that, like, a vanishing amount of fine tuning can remove the refusals, or are we seeing the refusal training somehow get deeper and and harder to remove with like because it used to be I think I think it was a FAR paper that showed that it was like $2 worth of fine tuning would remove the safeguards at one point. Right?

[1:10:27] Adam Gleave: Yeah. So I I think these kinds of weight based attacks where you actually modify the weights, whether that be fine tuning or there's also techniques like refusal obliteration, these unfortunately are still pretty pretty viable against open weight models. The the kind of good news is that it is more challenging, both from a technical expertise angle and also just a compute angle to do fine tuning against these models. It usually takes us at least a few weeks to get a new open weight model hooked into our infrastructure for fine tuning, and you need a sort of minimum number of GPUs. So there's a bit of a deterrence effect here, but it's not gonna stop really capable, well resourced attackers. So I think for that, we're gonna need new approaches. One I'm most optimistic about in the short term is pre training filtering. So it's a simple idea where just don't train the models on really dangerous stuff. If you don't need your model to help people make anthrax, don't train it on the anthrax papers. A tiny number of users might be a little bit sad that it can't answer questions about this, but most people won't even notice. But it has a big impact on the sort of misused potential of the model. And this has been validated in a number of scientific papers. This is independent researchers. UK's AI security and has been sponsoring some research into this. OpenAI actually used this in their GPT OSS release. So it's been tested quite well, but it's not become kind of common practice. And this is something we're actively excited about scaling to make sure this does work at a near frontier approach. Going back to what you were saying earlier about could we start sharing some of these datasets or filters with people. Our plan is just open source as much as we think it doesn't have a misuse potential, and then privately share with developers of things that that do have misuse potential that could be really useful to just lower the cost of these kinds of interventions. I think that's gonna say you're gonna be enough long term because pretraining filtering, if you're removing specific capabilities. But if you were willing to pay the fine tuning cost to train on the data that we'd excluded, then you could get it back. But that that's been now spending probably certainly millions, probably hundreds of billions of tokens. So it's a lot more expensive. But there is research into tamper resistant refusal. And so what I do is embeds refusal training so deeply into a model that any attempt to fine tune it is gonna really degrade the capabilities of a model. Obviously, if it's a come someone is willing to just train a model from scratch, there's nothing you can do to stop them. And that that that's really making the cost in November sort of tens or hundreds of millions of dollars. So I am optimistic that we can certainly get a lot better than we are now with open weight without major changes to the pipeline. Think we have to try. It will be a real shame to lose open weight models certainly, but they're really invaluable in our research sizing access to to models. And it'd also be a real shame to see widespread misuse of these systems. But I think there is an open question as to how much you can push this, and it it may be that you need to exploit a lot of bio capabilities, for example, from open weight models. And then maybe there was this approach gradient routine that was tested recently that lets you localize all of those dangerous capabilities in, for example, a particular expert. Maybe you could share that expert with certain trusted actors, and they could still run it locally on their hardware, but you don't just make it available for for anyone on the Internet to download.

[1:13:38] Nathan Labenz: Yeah. I was gonna bring that up. Shout out to AE Studio. Yeah. I love that piece. Why hasn't this happened? I mean, it it strikes me that when you say it, like, hasn't become standard practice, you look at all the things that Anthropic is doing. And in some ways, it's like, you know, wow. Like, you guys have really gone so far with pioneering all these different techniques and, you know, even for better or worse, you know, being willing to take some real heat for over refusal. And, you know, they had the one technique that they did in fact walk back that was like the, you know, the silent downgrading. Yep. They've done all these things. Like, why are all these things happening before, like, basic pre training data filtering is happening?

[1:14:22] Adam Gleave: Yeah. I I think it's a good question. And the the charitable take for developers is that if there's one thing you really don't want to mess with, it is pre training because this is just orders of magnitude more expensive than every other training procedure that that you do. Don't rock the boat, basically. If we've got this recipe that works and we know it's gonna work better, we scale it up, then let's do that. Let's not change anything that we don't need to. And so especially if you're a proprietary developer where you can say, okay. We have all these other methods that we can use to stop misuse of our model. We're gonna lean more on that, and we're gonna not push this onto a pretraining team. So I think that there's some argument to that. Do you think it's overall been an overlooked approach? And we're seeing increasing work that pretraining interventions are important not just for preventing misuse, but also for alignment. Because, ultimately, pretraining is where the model learns most of its representations or values, a lot of its innate drives, and post training is to a first approximation, shifting around personas in an already established persona space. And that's beginning to change as post training is an increasing fraction of overall training time models actually change more in post training. But pre training is really important, and I think it's pretty intuitive. You wouldn't say we're just gonna, not care at all about the upbringing of our child from zero to 12, but the last six years, we're gonna really get that right. You've gotta get both both right for the system to work well. So geodesic research, I'll give a shout out to them. They've been doing a lot of work on pretraining safety interventions and finding that this really improves a sort of overall alignment of a model. And I think Anthropic has been experimenting with us a little bit with things like looking at alignment generalization, how that changes, not just in pretraining, but also mid trainings. You add some synthetic documents partway through training. So I think ultimately, this is something we're we're gonna have to tackle, not just for misuse, but for preventing loss of control. And you can now do quite good work on pretraining experiments on on on really quite capable models for not that much money. So we're looking at scaling up pretraining filtering and gonna be doing not full, but pretty close to replic full replicas of something like NVIDIA's Neutron Nano, and it only costs maybe, like, a $100,000 per run. But that's a lot of money on one hand, but it's something that a nonprofit can afford to to actually do a bunch of runs. And then we're thinking of scaling up to Nematron Super, which is a 120,000,000,000 parameter model, do it training it for enough tokens to be the chinchilla compute optimal point. So training past that point would be wasting training compute, although it would make the model more capable and better for inference. And that costs ballpark $2,000,000. So expensive, but again, well within the range of a number of actors to try for sort of final validation runs. I think there's no reason not to experiment with this, and if the scaling laws look good, you can cautiously incorporate some of these techniques into your pretraining run. You can start by filtering out just a very small percentage of your data. It won't have a big capability here, and then work your way up. So I think that we do need more adoption here. I expect open weight developers to be the first to have to adopt this because they have fewer options. But I hope that proprietary developers also use this, especially for those sort of more loss of control flavored risks.

[1:17:26] Nathan Labenz: Speaking of loss of control, let's get to the news. So it's here. Right? We are

[1:17:32] Adam Gleave: Yep.

[1:17:33] Nathan Labenz: Now dealing with the nightmare scenarios, at least, you know, the early nightmare scenarios. How would you tell the story? I mean, everybody's heard the stories. I don't I don't mean in the basic sense of, like, we all know what happened. But how would you interpret the story? Like, what what's what's your, you know, unique way of contextualizing what we've just seen?

[1:17:59] Adam Gleave: Yeah. Maybe I'll start by just, I think, highlighting the things that aren't surprising about this story or aren't surprising if you've been following alignment in AI carefully. So reward hacking behaviors where models do really quite sophisticated cheating on tests. We've been seeing that for years, and people have been warning that's gonna happen for a decade. But in until now, these are being pretty toy, almost like cute misbehaviors over model edited its unit tests. So what? Now, have a model that is going out hacking third party computer systems to see if answers to a test. In some ways, from a model's perspective, this is no different to what it has been doing before. It's just it's got the capabilities to cause real harm. And so, I think that that should be a bit of a wake up call, but there's a lot of problems that have become almost mundane. We've seen them so many times before that we start saying, oh, yeah, this is an alignment failure, but does it really matter? They're gonna start mattering as the models get more capable. So, in a way, that's optimistic because we've got a pretty clear empirical set of warnings and things we can optimize and improve. And in some ways, it should be a bit pessimistic because we've had all these warnings and we've disregarded them, and it's not clear that we that's gonna necessarily, you know, change without some some external pressure. And then I think the part of the story that isn't talked about that much is that, at least as as far as we can tell from public information, Hugging Face noticed this hack first. So OpenAI was running an internal deployment for testing. It went out and hacked a third party. And it wasn't OpenAI having an automatic monitoring scheme and being like, oh my god, what has our agent done? We need to call up Hugging Face and apologize. Hugging Face noticed it. They investigated, and it turns out, oh, oops, it was an OpenAI model. So I'm sure OpenAI is going back over their transcripts now. You'd have to wonder, is there a chance it hacked anyone else? Or if it hadn't hacked Hugging Face, hacked someone who had a worse security posture. Would we ever have noticed? And it isn't the first time that something like this has has happened. Alibaba was training their Rome system, and their cloud security team called them up and said, hey, I we think that someone has compromised your training service because there's cryptocurrency mining going on. And it turned out that the agent had thought, oh, I I need to get some more compute. How do I do that? I'll I'll do some mining on the compute I have, and I'll try and rent some servers elsewhere. And in in some ways, I think that's even more of a near miss loss of control incident because it was actually trying to start gaining resources and potentially copy itself outside of infrastructure, whereas at least the OpenAI model have this pretty narrow objective of just getting some test results on a benchmark. So, yeah, I think takeaway from this would be less on the alignment side because in fairness to OpenAI, this was a model that had cyber safeguards removed, and I don't think we know of exact prompting regime, but it might well have been told to do something like this or at least incentivized to do it. So, it's not where we can't align these systems, but it is a massive control internal monitoring failure where the sandbox was insufficient. It doesn't seem like there was any additional layer of control mechanisms on what the AI system was doing. There wasn't asynchronous monitoring that alerted us us to that. So I think that really needs to change, both for prosaic reason that, you know, sooner or later, you're gonna hack someone who doesn't take this as gracefully as Hugging Face does, and your company is gonna be in a lot of trouble. And and also for the the risk that we're gonna see with more capable models that might be pursuing much more malign goals than just trying to cheat on a test.

[1:21:22] Nathan Labenz: Yeah. It seems like there's sort of a spectrum. Don't know where we're on it yet, and hopefully, we'll get the transparency that we've been promised. But I guess my kind of mental model of this right now is we're somewhere on a spectrum from extremely negligent to extremely scary. And on the extremely negligent end is like, had this thing. You gave it this task. You had no additional monitoring. You hope your sandbox is good enough. It turns out it's not. Next thing you know, you're totally pwned, but but it's not necessarily that scary because you're because you were in fact very negligent. Where on the other end, if you had, like, probes running and you had, like, additional, you know, reasoning models supervising the trace and this still happened, then we're, like, in extremely scary territory. Yep. I don't know if it's good or bad, but I I'm guessing we're more toward the negligent end. But I don't know. I mean, the vibe, you could you probably are I pride myself on being fairly plugged into the vibe. You're probably even, you know, significantly more plugged into the vibe. The vibe seems to be kind of, like, spooked. So it's it the vibe doesn't suggest that it was totally negligent. It does kind of my guess would be naively, you know, if I hadn't talked to people and seen how spooked people would be, I would think that this was, like, kind of, you know, bad practice, sloppy and negligent Yeah. But not so scary. But, like, people are acting like it's scarier than that would lead one to conclude.

[1:22:44] Adam Gleave: Yeah. Yeah. We just, as you say, don't have the full information. But I think from what we do know, it wasn't like OpenAI was just running some outdated piece of software, and the model downloaded a publicly available exploit and used it. Right? It did find a zero day in a widely used piece of software and used that to to break out of a sandbox. And you think you you could call it negligent, because maybe if you're playing with fire and you don't have a lot of containment, you're being negligent. But I'd be pretty surprised if OpenAI's internal infrastructure was less secure than your median American company or something like that. And suddenly, there's nothing in the report that suggests that would be the case, and there was also Hugging Face is a pretty capable company, and again, probably has much better information security posture than the median company around the world. So from that perspective, I think it it's it's right to be quite spooked that you're sort of run of the mill or even good, but not paranoid security practice is not enough to contain agents any longer. And maybe the extra AI specific control mechanisms will, because they're running in a harness, and you can inspect their actions. It's not like it's just someone who's got SSH access into your server. Maybe good, but not NSA level air gapped cybersecurity would be enough to stop AI agents from doing this, but for how much longer? Their cyber capabilities are getting better and better. And maybe to some extent, it it doesn't matter whether it's negligent or really scary capabilities, because this is evidence that there are gonna be actors who deploy with this level of safeguards. And I don't think that OpenAI is, by any means, the sort of most reckless actor here. And other developers are not that far behind. It's probably not more than three to six months, suddenly not more than a year behind OpenAI their capabilities. So this is something that needs to have some some kind of industry wide solution to ultimately, whether that be better cybersecurity across the board, easily deployed control mechanisms, or just more care and attention because people are aware of this kind of risk. That that's probably the main reason. I'm optimistic and not freaking out. It's like, okay. We're gonna see these warning shots. And so long as we respond appropriately, it's actually encouraging that we're seeing these issues, and it's not that an agent waits until it can definitely take over and then does a treacherous turn, which for some of us have, more more old school AI safety concerns. But I also see there have been a lot of warning shots, and at some point, I'm just wondering if we're gonna until something really bad happens.

[1:25:06] Nathan Labenz: Yeah. It does seem like the vibe is, at least for now, taking this seriously. But yeah. Mean Yeah. The news cycle is short too. So who knows what will things will look like in even just a few weeks.

[1:25:20] Adam Gleave: Yep.

[1:25:21] Nathan Labenz: I mean, would you I guess in terms of, like, why this happens, just did a episode not too long ago with David Odd, David Dalrymple, who basically says, you know, he used to have a Pdoom. He would quote 70 plus percent. Now he's down to under 5%. Wow. That's a big change. Why? Well, basically, constitutional alignment seems to be working, and we can sort of, you know, get these AIs to become Bodhisattvas and, you know, work for the benefit of all living beings. And we just need and his kind of, you know, caveat is, like, we just need to not turn the RL up so high. We keep doing that because it obviously does bring performance. But, you know, every time we tilt too far toward RL, we start to see, like, o three type problems and, you know, little, you know, like, on end and bench type things. You know, you see Claude kind of doing things that they probably don't want Claude to be doing. So I guess one maybe overly naive story, but perhaps captures, you know, a decent chunk of what's going on is, like, don't overdo the RL. You know? Do you think that that could we draw a lesson as Yeah. Simple as that? And if we can, can we operationalize it into some rules?

[1:26:38] Adam Gleave: Yeah. I I think this is pointing out something really important, and maybe my my p team is not that different to to David Eds. And I always struggle a little bit operationalizing question, but I'm probably somewhere near 10% existential risk in the next few decades from AI. And and I think that we could probably get that down to something like 1% without any major research breakthroughs, just iterating and refining what we already have and taking a sort of careful engineering approach to systems and having good safety cultures at companies. And not like stopping AI or anything, but if we don't know how to make the next system align, then maybe we do do experiments and do more detailed evaluations before we deploy the next system. You know, just gating deployments on on actually doing that that rigorous work. It's still gonna look from the outside like things are moving incredibly fast, probably faster than they are now, but we're not just literally moving as quickly as possible, I think. You know, if you're really trying to move as quickly as possible, then there's gonna be a big pressure to cut corners on these things. I don't know if it's gonna be as simple a recipe as, you know, just don't do more of an x percent RL. And ultimately, some types of RL training are are really important in the models having a a line personas. I think it depends. Is this coming from human feedback? Do you have a human in the loop? Is this coming from constitutional AI? How much do you trust the constitution? Do you have some evaluator? Is this RL verified reward where you're just optimizing the system to solve certain tasks? Are you accidentally training on the chain of thought, which some developers do now and again and are meant to not do? So there's all these kinds of implementation details, but you could definitely imagine something that is operationalized as, okay, we know this training recipe seems to basically work and the models are pretty aligned. We know that if we push in this direction, the models become more water hacking. We think this is a dangerous part. We're gonna set a safety margin of don't go beyond this point. And then we're periodically gonna reassess if we've crept into the safety buffer or we're set more conservative thresholds for the next model. But if we've got various different kind of advances to our training stack, actually, we're quite, you know, way away from the safety margin because we've got these better techniques where maybe we can crank up the, you know, RLVR or something like that. I think that would be a great regime to to live in. It doesn't seem that hard technically. It's maybe quite hard economically and politically. And right now, there's a really huge amount of distress between the AI companies. Like, I I was at a a workshop we ran recently on on chain of thought monitorability, which I think is maybe an easier one to operationalize than this this reinforcement learning. So we've got this kind of gift right now that that models, they just talk in their chain of thought about how they know that in an evaluation or are deceptive. And this is really useful from a capability standpoint because you can see why the model's making a mistake, and you can maybe trick tweak your post training pipeline. It's also really useful from a safety standpoint because you can see how the model might be trying to deceive you. But there's various ways we might lose this. We might train against a chain of thought accidentally. We might train models that have reasoning, neuralese, these continuous activations rather than chain of thought. So there was this proposal of could we just agree not to train neuralese models without letting other developers know? So it's not that you can't train it. Just give everyone a heads up. And this was even that kind of for people, were like, oh, it's gonna be really hard. That's information intellectual property leakage. We're telling people we're doing this. We're not telling people how. We're just saying that we're gonna do it. It's one bit of information. But even that, a lot of developers are really reticent about. I think it's good to try and work on things like this for sure, but I suspect there is gonna need to be some kind of third party that comes in and addresses this. I don't think it's gonna be easy for the developers to do that themselves.

[1:30:05] Nathan Labenz: I should say just to represent Davidad correctly, he was specifically focused on RLVR and not, you know, all Sure. Manner of RL. That does seem to be where the real, you know, persistent increasingly aggressive behavior tends to come from. So that's a sobering note, though, on the kind of prospects for coordination. I wanted to ask you for your thoughts on prospects for US China coordination as well. But, you know, it's it's tough if we can't get our own house in order. Yeah. What do you think I mean, we are in this kind of interesting moment where, like, the Overton window has been blown wide open. Right? I mean, I this has been a the a quite a shift from not that long ago where all of a sudden there is people all over the place expressing openness to some sort of coordinated slowdown or whatever. How do you think that is best realized? And and how critical is it that we get China on board? And and what are your how optimistic are you that we might be able to actually pull all that off and kind of jog into the singularity instead of, you know, full on sprint into it?

[1:31:16] Adam Gleave: I'm optimistic that we can do a lot better than we have now, and I think it it's great that, for example, president Trump and Xi Jinping are are talking and seem to have quite productive conversations. So I think the question is how far can we get? I think it's clear that we need to at least try to work with with China and other countries on some of this. America cannot solve this problem on its own. I'm focus of AI security leaderboard. We're getting to a point where if the best proprietary safeguards were used across the board at American companies, but not yet, but we're getting there, then most of them misusers would probably shift to open weight models with and for most people, open weight models are developed in China. So I think we've just got to work with China there or at least Chinese developers if we want to address that. And this is something that there's just common interest. Neither nation wants random terrorists going going around building explosive devices or or bioweapons. So I think there's some low hanging fruit there. And part of this is just a a lack of understanding. So safety only really is just beginning to enter the mainstream consciousness in The US. And China's just a bit further behind on on that. But that that is changing, and I've had a number of excellent conversations with scientists based in China who who are doing great work and working on these issues. So I think that that that's a great starting point, but we all need to have that kind of scientific agreement ultimately flow into corporate and political will. I think where I'm more skeptical is really being able to coordinate to avoid kind of some kind of race dynamic. I think that the race is a little bit overrated. I'd say it's much more of a China was trying not to be left behind. The The US and a lot of what they've doing has been copying, whether it's directly distilling US frontier models or just realizing, oh, The US is making AI a priority. So we better as well. So from that perspective, I think that The US is in a good position, but if it takes the it's better for accelerator. China probably will a bit. But the flip side is if The US just slams on the brakes, China might decide, we'll just keep coasting, and we'll at least catch up, and maybe we'll go past. So I think some kind of coordinated development where we agree to common safety standards through mutual market access or just it doesn't even have to be identical safety standards, but just both countries implement some good best practices. And there's some tacit agreement that if either country starts some sort of secret project of other detects, then that that's gonna be that's gonna be bad. There's gonna be repercussions, and we try and have broadly the same safety standards. That feels pretty doable to me, but actually being able say, no, we're we're both gonna gonna stop at this point until some research advance is made. That that's gonna require a lot more political work than it is now. It feels very unstable. It's not just about The US and China. Other countries can also be relevant if we're talking about more than a year of delay. So I think mostly, I'm operating under assumption that things are gonna continue to march on, that we can pick some of the low hanging fruit here on coordination and at least avoid the sort of most extreme race to the bottom on safety.

[1:34:20] Nathan Labenz: Mental model I've been developing recently, it's very pretty simple in some ways, but interesting to get your take on it is I I basically think when it comes to really high scale risks from AI, you can kind of break it down into the irreducible, unavoidable risk, which is sort of the, you know, the fact that, like, we have web scale compute. We have web scale data. You know, people are gonna figure things out. And there might just be some things that just really don't go well. And if we stumble onto them, it could be really bad. And then there's also probably in that category, like, if you know, and you've used this kind of, you know, multiphase descriptor a couple times, like well resourced, determined, persisted, you know, nation state quality actor. I kinda think if, like, if we have a major national government turn, like, so bad that they wanna create bioweapons and, like, kill us all. Like, that's gonna be a really big problem.

[1:35:16] Adam Gleave: Yeah.

[1:35:17] Nathan Labenz: It's

[1:35:17] Adam Gleave: probably a problem without AI, to be honest.

[1:35:19] Nathan Labenz: Yeah. So we're I mean, that Yeah. That's I put that in kind of the irreducible category. If that happens, you know, it's gonna be real tough for Yeah. You know, anything we do today to to put that you know, to prevent that from going really bad. But then there's the other side, which is, like, the the things where we're kinda asking for it. And, you know, if we sort of release a mythos open source with no safeguards that can do all the bio stuff, we're kind of asking for it. Yeah. If we, like, sprint into recursive self improvement, especially after seeing what we've just seen, you know, I kinda think we're sort of just asking for it. I guess my sense of your overall position is, like, you're fairly optimistic that the and I think I share this for the most part that the irreducible part is, like, relatively small Yep. And that we can actually get our act together on the stuff where we would be, in fact, asking for it to fail to get our act together. Is that a decent summary of your worldview?

[1:36:17] Adam Gleave: I think that's a a a great summary that a lot of this is basically just about getting the basics right. And the good news is it's not that costly or at least it's not at all costly compared to the billions of dollars that are being spent to train front end models. So we can afford to do this. It's just about putting in place the right incentives and peep people getting on the same page. And I do definitely have some kind of, know, uncertainty over that. And so what I'd like to see is that we continue to do rigorous evaluations of models and just have a look for cases where this assumption might be wrong. And I think the good news there is that if there was actually really crisp evidence that these problems were not reducible, and that sort of anyone who crosses a certain capability threshold in AI models, they're just gonna kill themselves and everyone else with them. But I think a lot of these problems would also go away because people could stop trying to race for this thing that they knew it was going to be really bad, not just for other countries, other companies, but for themselves as well. And and right now, the problem is that we do live in this this world of both uncertainty, but also just a lot of disagreement. And so I think the mainline plan here is let's pick the low hanging fruit and fix the issues that are probably going to be enough in in most worlds, but still put in place mechanisms that would alert us if our assumption here is wrong and we are in a more adversarial world so that we have an opportunity to course correct. But, yeah, I share your view that it would be it'd just be a real shame if we do a really silly mistake, and that's what does us in. When we had the solution, if we roll the die with odds very much in our favor, but it was just hard to coordinate to get it down from, like, 1% to one in a thousand. It's still bad, but I'm like, okay, I can see how it happened, but if we just fail to basically import pre training filtering into our pre training pipeline when there was already a stack out there, that was just that's just an unforced error. It's like losing a game because your opponent was really good versus losing a game because you scored an own goal. In both cases, you've lost for games. Maybe it's a silly distinction, but I think it's just a lot more painful when it's just this own goal or unforced error, so we can at least try and avoid that.

[1:38:18] Nathan Labenz: Yeah. And I I at least for me, it feels like the bulk of the wrist is probably in the we're asking for it category.

[1:38:27] Adam Gleave: I I think so. Yeah.

[1:38:29] Nathan Labenz: Yeah. Which, you know, in a in a way that's I could I could take the glass half full. I'm expecting you on that. This has been fantastic. As always, I really appreciate your time. I I hope we can do this a little bit more regularly going forward. For the moment, though, anything else you would wanna touch on or leave people with before we break?

[1:38:48] Adam Gleave: I certainly hope to be back on the show again soon. If this has piqued people's interest, then do check out the full leaderboard and report, which will be at leaderboard.fodderai, and I'm sure you can include that in the show notes. And, yeah, we're also very much interested in helping solve open weight safety issues. We're actually gonna be launching a $2,000,000 grant making program for that. So if you are an academic independent researcher who wants to work in various areas and is looking for funding or compete, then do drop me a line as well. We'd love to talk.

[1:39:16] Nathan Labenz: Adam Gleeve, keep up the great work, and thank you for being part of the cognitive revolution.

[1:39:21] Adam Gleave: Thanks, Nathan.

Outro

[1:42:53] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.