AI:AM: Carbon in the Arena, Mistral's Le Chonk, the Agent-Run Economy, Ferroelectric RAM + New Hacks

Watch Episode Here

Listen to Episode Here

Show Notes

AI:AM: Carbon in the Arena, Mistral's Le Chonk, the Agent-Run Economy, Ferroelectric RAM + New Hacks

This week's question on AI in the AM: how much of what we think we know about AI survives contact with the data? Nathan Labenz and Prakash Narayanan take it through five guest conversations from this week's live shows (Wednesday, October 28 and Friday, October 30). Anastasios Angelopoulos, co-founder and CEO of Arena, the leaderboard formerly known as LMArena, explains what Arena's production traces show about how models behave in real work, and why the industry's turn toward private benchmarks worries him. Debo Olaosebikan, co-founder and CEO of Kepler Computing, out of seven years in stealth this fall, explains where the energy goes in an AI chip and why he says Kepler's ferroelectric memory can be made in older fabs without EUV. Samvit Ramadurgam, who co-founded Forge and now runs Union, describes a company run on tokens, which sets off the hosts' argument about who the economy still needs. Michael Freeman, senior director of threat research at ServiceNow, reports what his team's telemetry shows about AI-run attacks in Southeast Asia. Sagar Vaze, who leads the multimodal team at Mistral, explains how Mistral trained a trillion-parameter model on about 3,800 GPUs, what went wrong halfway through, and why it shipped with cyber capabilities as open weights. The episode closes on the question Nathan spent the week on: whether you can keep a dangerous capability out of a model at all, and which fights are worth picking.

The introductions and transitions are spoken in Nathan's cloned voice, disclosed in the episode. Guest and host conversations come from the recorded live shows. Tell us what worked and what didn't.

AI:AM and the hosts: https://ai-in-the-am.com/ · Nathan: https://x.com/labenz · Prakash: https://x.com/8teAPi

Full live shows: Wednesday, October 28: https://www.youtube.com/watch?v=Yj6qugkRMWQ · Friday, October 30: https://www.youtube.com/watch?v=1FdJvaAft5E

Cold open

Anastasios Angelopoulos on misbehavior in long conversations and Gemini 4 Carbon's arrival in Arena; Debo Olaosebikan on Kepler's absence from compute forecasts; Samvit Ramadurgam on coordinating the world economy as Union's act one; Sagar Vaze on compute discipline and the mid-run repair dubbed Patchstral; and Michael Freeman on the cybersecurity problems that smart AI researchers can miss.

Anastasios says conversations of 20 turns or longer show some form of misaligned behavior more than half the time in Arena's production traces. This is his on-air combined-signal estimate; Arena's published post reports a 45.4% deceptive-completion rate in its longest-session group.

Arena's AI Alignment Index: https://arena.ai/blog/ai-alignment-index

Part I — The ugly result

Nathan interprets the announced Exponential Time Hypothesis result discussed at the start of Wednesday's show. He describes the claimed disproof as a counterexample, distinguishes the limited speedups being discussed from a general collapse of computational difficulty, and asks whether mathematicians' preference for elegant complexity classes shaped their expectations. His closing qualification is that the practical speedup, at least in what he has seen so far, is not large. The narration treats the labs' announcements as claims that have not been peer reviewed; this section summarizes Nathan's on-air interpretation rather than independently certifying the result.

Source conversation: https://www.youtube.com/watch?v=Yj6qugkRMWQ&t=630s

Part II — Measured in production

Anastasios Angelopoulos runs Arena, where people compare models on their real work and the votes become the leaderboard. Arena's new Alignment Index, he says, measures "the actual realized incremental risk that a model poses if it's released to people," on three signals: unauthorized actions (the user says "you can touch only the files in this folder," and the model doesn't comply), false attributions (the model assumes an intent the user didn't have), and deceptive completions, "the model lying to the user and telling the user that it did something that it didn't. Like, yes. I did check every line in that spreadsheet... You can go ahead and send it to the IRS." Why lie? "Because that is the fastest way to get its reward."

Nathan asks what the right mix of evaluation techniques is now. Anastasios: "we now know that models are able to tell whether they're being evaluated or not," so on real tasks "they might be misaligned, and you would never know." Hence "the only actual way to measure alignment is by getting into the flow of people's real work."

Nathan raises what he calls a narrative violation: Andon Labs found OpenAI's models more aligned than Anthropic's in its run-a-business experiments. Anastasios sees the same thing: "GPT is holding the top 5 slots." Single-digit misalignment rates "even from OpenAI and Anthropic models" are "a ton. It means that anybody that's had 1000 chats has experienced... 10 to 50, let's say, examples of a misaligned behavior like a lie or deception." "OpenAI is number 1 on our leaderboards right now. It is over Anthropic, I believe, in all 3 of our signals." Then SpaceX AI, Google, and "a big gap" to the open-source labs; "Tencent is number 5. The deceptive completion rate of the Tencent model is 13%." Those figures match Arena's published table (OpenAI holds the top five; Tencent's HY4 Preview at 13.2%). His hedge: "Alignment tends to follow capabilities."

The day before, Artificial Analysis had published version five of its Intelligence Index, with most of its benchmarks now private, and the open-weight models fell to the back. Prakash asks whether Arena can detect benchmark-maxing. Anastasios declines the premise: "I love the Artificial Analysis guys... I don't think that it's healthy for our industry to be relying on privately held benchmarks." A fixed private set can still be gamed, and "we don't really know whether the frontier closed source models in the US weren't benchmark maxing against the closed datasets that are now going into this index." That is a suspicion, offered without evidence on air. His alternative: live production data "can't be overfit."

Why a router? In Arena's own traces, "tons and tons of traces that were like, thank you, and then that cost you $40": a user returning to a long-running agent the next day, cache invalidated, typing thanks. "We're burning VC money." Hence MAX, "short for argmax." Finally, Gemini 4 Carbon landed on the Arena leaderboard while the show was live. "Carbon is a f***ing fantastic model, dude. It's so good. Google is back. Google is back in a big way." He puts it at number two on Agent Arena, behind Opus 5.5 Max and above Sonnet 5.5 and Astra. "What a big comeback."

Arena: https://arena.ai/company/about · Anastasios: https://x.com/ml_angelopoulos · Series B at $3.1B (October 8): https://arena.ai/blog/series-b

Andon Labs, Astra vs Fable on Vending-Bench: https://andonlabs.com/blog/gpt-6-astra-vending-bench

Artificial Analysis Intelligence Index v5 (October 27): https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5

The MAX router: https://arena.ai/blog/max-router-tops-agent-arena

Gemini 4 Carbon: https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/ · Carbon's Agent Arena placing as of October 28 (secondary source): https://officechai.com/ai/gemini-4-carbon-tops-text-arena-places-2nd-on-agent-arena/

Part III — An orthogonal axis

Debo Olaosebikan's Kepler Computing came out of stealth in September with $468 million raised and a ferroelectric memory it says can be made in older fabs. He starts with where the energy goes: "for inference, especially, the bulk of that energy is actually running across the street to fetch the data from the library... before you process and do your fancy matrix multiplications." So the first fix "at the physics level" is memory, closer to the processor and denser. "Both of these things point us to going 3D." Kepler's cell replaces six transistors with "one transistor, one capacitor sitting on top of that transistor," using a ferroelectric material whose "recipe of the cake" is proprietary, and eventually stacks capacitors on each other, "to avoid the rat race of, like, trying to shrink transistors within a chip."

Prakash asks about manufacturing and what it buys you not to depend on EUV. Debo's analogy: San Francisco thought it didn't have enough taxi drivers "and then we realized, oh, we have so many taxi drivers if you just build Uber." His claim, which is the company's and has not yet shipped in a product: "we can take existing older node fabs, and we can use them to make chips that are actually superior to the advanced node chips." Kepler benefits from EUV but "we don't require it." He says the process introduces no new tools ("the answer is no for the most part"), which is what the seven years of stealth were spent proving.

The cold open also includes Debo's view that Kepler is not yet priced into compute forecasts after seven years in stealth.

Kepler Computing: https://keplercompute.com/ · Debo: https://x.com/dolaoseb · Kepler's "act one" (the no-EUV and older-fab claims, in the company's words): https://keplercompute.com/act-one

WIRED on the launch (September 9): https://www.wired.com/story/a-new-dollar400-million-startup-wants-to-fix-the-ai-memory-bottleneck/ · TrendForce on the GlobalFoundries 28nm plan and 2027 production: https://www.trendforce.com/news/2026-09-14/news-kepler-claims-hbm-alternative-without-euv-using-globalfoundries-28nm-fabs-2027-production/

The Dwarkesh–Dylan Patel conversation Debo refers to ("Can't we just use TSMC's older fabs?" at 55:47): https://www.dwarkesh.com/p/dylan-patel

Part IV — Act one

Samvit sets out Union's larger ambition: connecting people and markets through agents, with coordinating the world economy as its first act. Nathan asks what remains for people when agents can do the work. Samvit talks about disempowerment, alignment, and the shift from managing people to directing agents before explaining Union's own token usage and hiring.

Samvit Ramadurgam co-founded Forge, the marketplace for pre-IPO stock, and now runs Union, which is building an agent that brokers deals: introductions, asset sales, private shares. Prakash asks about token usage. "It is very high." Union's internal target, he says, is for each person to productively deploy "between 1 and 2 billion tokens per day," in "an agent native company" with "a chief marketing officer who reports to me who's got, like, a bunch of different sub employees reporting to that AI." These are his descriptions of his company. What is different from the Forge era? "We're, like, 5 people, soon to be 10 people right now. And so the heuristic I have for hiring someone is, are they going to prompt me to do something different than we would have done before? ... You don't really need people to do, like, work anymore for the kinds of work we're doing." There is "a type of company you could start now that does with a dozen people that previously took 1000 people," and "the bottleneck is, like, clear articulation of good ideas."

After Samvit left, with Microsoft offering voluntary buyouts to some 60,000 employees that week, Prakash argued the economy has to grow fast enough to absorb the white-collar workers that agents displace. Nathan: "I can see the growth. I have a hard time seeing... the job growth picking up enough to offset." Samvit's hiring heuristic is the problem: "There's just not that many of those roles." The people who may be leaving Microsoft "have made their way there by, like, executing tasks," and "I just don't see how so many people are going to pivot from task completion to, like, counterfactual... contribution." His last line: "how many people work in sales? Like, he's basically attacking all of sales."

Prakash offers a leading indicator, and everything in it is his account: he is in WhatsApp groups for data brokers, and "whatever requests for data they're doing now in about 12 months, that's gonna hit the product." He says the labs are buying up entire companies' data to build "training playgrounds for agents" (unverified; no public source found). Within a year, he predicts, "they will give you a company in a box": "add capital. Add one idea... And it will do all of the Shopify-ing, the advertising, the analysis, the market entry, the customer service, the returns, end to end." Nathan expects "a ton of these one person companies," but "I just don't see how there can be enough of these little niches... to absorb all the people that used to be driving cars, you know, doing tasks."

Prakash's restaurant arithmetic, his rule of thumb: a quarter each for labor, rent and ingredients, "you're lucky to clear 5 or 10%," so robots in the kitchen make restaurants "5 times more profitable" and you open more of them. Nathan: "This is the classic, like, we lose money on everyone and make it up on volume, but, like, brought to employment... There may be a lump of labor fallacy, but I'm pretty sure the lump of food principle is, like, roughly right." Prakash concedes and says the growth "is always gonna be in the AI consumption going forward," with the human economy "on the down drift except for status," citing what he says is Peter Thiel's "relational economy" (Prakash's attribution; not verified). Nathan: "it sounds more dystopian than utopian to think that, like, the bulk of human activity will go into status games." He is "a bear on employment, bullish on people's ability to find meaning in absence of employment," because "a huge portion of people are working to live and not really tying up their identity... with their work."

Union: https://union.app · Samvit: https://x.com/rsamvit (Not to be confused with Union.ai, a different company.)

Schwab's acquisition of Forge Global (March 2): https://nasdaq.com/press-release/charles-schwab-completes-acquisition-forge-global-2026-03-02

Microsoft's voluntary buyouts, reported at about 60,000 employees: https://www.fox13seattle.com/news/microsoft-offers-voluntary-buyouts-60000-employees-tech-giant-reshapes-its-workforce

Part V — Five to ten cents

Michael Freeman is senior director of threat research at ServiceNow, where he came by way of the Armis acquisition; his team watches AI agents attack companies. Back in July, he says, "all other companies were telling these reporters, it looks just like script kiddies right now. It's just a bunch of noise. And I said, well, it's evolving fast. Give it a couple of months." Now: "We're seeing AI development for offensive capabilities that perform at a scale, I would say, of a very competent APT teams right now."

Everything that follows about Southeast Asia is his team's unpublished telemetry. Forked attack frameworks are gaining capabilities fast ("it looks like it's AI actually adding these new capabilities"), moving from web attacks to hunting "credentials in ticketing systems," then Active Directory persistence. "We have seen 3 to 4,000 on average attacks successfully in the past 2 weeks targeting a lot of Southeast Asian companies." Banks, airlines, money or mayhem? "I would say it's all of the above right now... A lot of it is just to see what it can do." The night before the show, his team analyzed a forked framework building ransomware "using cryptography that's post quantum cryptography."

Nathan had been listening to a conversation between Sam Altman and Patrick Collison (which appearance isn't confirmed) in which Collison asked whether the labs went too far with cyber training. He contrasts what he calls the Anthropic story ("we didn't train this to be a cyber attacker. It just kind of got that way because it's so good at everything") with his recollection of the CTO of Vercel, a previous guest, saying "it's very clear when you look at the traces that they've been trained to be cyber attackers." Can it be reeled back in? Freeman: "Not reel it back in. No. But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also." His team built its own vulnerability-finding small language model two years ago "for a couple $100,000," so reverse-engineering is trivial, and the commercial offerings are trained on cyber ranges and CTF data. "It's not just that LLMs have magically figured out how to do offensive capabilities. They've been trained on it."

Earlier that morning, Prakash had cited Jason Clinton, Anthropic's deputy chief information security officer, on getting to vulnerability zero. He puts it to Freeman as the CTO of a company like Nike: how do you get there in two years? Freeman: "we have a lot of really smart people doing AI, but they don't know cybersecurity at all. And they view cybersecurity as a software flaw." The worst vulnerabilities are business processes: signing off on a risk list you don't understand, a customer's API keys in a ticket, a help desk that resets everyone's password to the same "Winter 2026!" "No amount of patching is gonna fix that." Nathan asks about password managers. Freeman: "I really wish I can answer that question. I've not found a solution that I really like at all." What does he do? "I personally have it written down on a notebook."

ServiceNow: https://www.servicenow.com/ · Freeman has no public X account; the company is https://x.com/ServiceNow · Speaker bio: https://siberx.org/speaker/michael-freeman/

Jason Clinton on vulnerability zero (he is Anthropic's deputy CISO, not its CISO, as Prakash said on air): https://officechai.com/ai/it-was-always-going-to-be-this-way-we-tried-to-warn-the-world-anthropics-deputy-ciso-jason-clinton-on-recent-cyberattacks/

Altman on no longer expecting to patch every bug (a separate appearance, not confirmed as the one Nathan meant): https://www.pcgamer.com/software/ai/i-no-longer-think-were-going-to-be-able-to-patch-all-bugs-says-ai-pandoras-box-opener-sam-altman-were-going-to-have-to-completely-switch-the-defense-model/

Part VI — Le Chonk

The day before the show, Mistral released Mistral Large 4, nicknamed Le Chonk, as open weights: about a trillion parameters, roughly 50 billion active per token, trained on Mistral's own cluster of about 3,800 GPUs outside Paris. Friday's live show asked whether it is "the West's best open model." Mistral's own wording, on the model card, is that it scores 26 on Artificial Analysis's index, "the highest among western open-weight models," and that the leading Chinese open-weight models score higher. Sagar Vaze put the training compute at 10^25 FLOPs (his on-air figure; it does not appear in Mistral's posts). Nathan recalls the SB 1047 fight, where the threshold was 10^26, and says he'd have guessed a model like this needs ten times more. Sagar: labs do train above 10^26, yet "despite being 10x more in terms of pre-trained FLOPs at least, this model is very competitive." And the threshold moves: GPT-3 "was a dense model trained with 3x the number of active parameters as the current Mistral Large 4," while Large 4 activates "only 50 billion" per token, and "your FLOP budget is proportional to the number of active parameters." Then: "Training a trillion parameter model on 1e25 is sort of a rite of passage that you have to go through in order to be able to justify scaling even further."

Nathan asks how much uncertainty is left once a run that size is launched, and whether they ever aborted one. Sagar: "We'll talk about this in our technical report tomorrow, but indeed, there were some surprises in this pre-training run. You know, like halfway through it, we realized, okay, there's something wrong." Unlike competitors who can run ten in parallel, "we had to figure out, okay, how can we fix it?" All hands, ablations, and "we found a way to patch it. We called it Patchstral internally... It worked even better than we expected." (The technical report was due the next day and hasn't been checked.)

After Sagar leaves, Prakash comes back to the compute gap, 3,800 GPUs against the hundreds of thousands at the frontier labs. Nathan: "that's gonna be a really hard deficit to make up for," and if Mistral misses a model cycle "in a purely capitalist context, they're maybe done." But "there's not a lot of sovereign AI plays out there that are at all credible, and, literally, there might just be one," so "would the whole continent of Europe just be like, well, I guess we have zero now?" His statistic to close: Llama 3, July 2024, "was trained on 3.8 times 10 to the 25 FLOPs. So Llama 3 more than 2 years ago, nearly 4 times the input compute that Mistral Large 4 has today."

Mistral: https://mistral.ai/ · Sagar: https://x.com/Sagar_Vaze

Mistral Large 4 open weights (October 29): https://mistral.ai/news/mistral-large-4-oss/ · The October 6 preview post, with the 3,800-GPU figure: https://mistral.ai/news/mistral-large-4/ · Model card (52B active, 1.05T total): https://docs.mistral.ai/models/mistral-large-4-0

The weights, with Mistral's "highest among western open-weight models" wording: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B · Artificial Analysis on the model: https://artificialanalysis.ai/models/mistral-large-4

The Llama 3 paper (3.8×10^25 FLOPs): https://arxiv.org/abs/2407.21783 · SB 1047: https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240SB1047

Part VII — What the open model shouldn't know

After Freeman left, Prakash and Nathan argued about whether you can keep a capability out of a model at all. Prakash's view, as the narration reports it, was that you can't carve it out of the latent space; his story was that China's censors had to be taught what Tiananmen was before they could censor it. Nathan is "more bullish on some of these, like, not necessarily unlearning," but efforts "to basically lock up certain classes of biodata" on "a know your customer basis." His evidence is ESM from EvolutionaryScale and the Evo models from Arc Institute and Stanford: "if you do withhold certain classes of data, you can get generally capable models that still kind of don't know about these areas of censorship. So I'm not so sure that that can't work." EvolutionaryScale is not Meta; the EvolutionaryScale team joined Biohub in a transaction with the Chan Zuckerberg Initiative. (Which data the ESM and Evo teams withheld isn't confirmed here.)

Prakash asks Sagar how building for cyber defense changed Mistral's training. Sagar says that earlier in the year "some very capable models had restricted access to both certain geographies as well as to certain enterprises," while those models "often were being jailbroken by malicious actors," and defenders found "their requests were refused" because "your defensive request looks quite a lot like a malicious request." So "the only way that they could scalably defend their infrastructure was by using an open-weights model," and Mistral "thought it was imperative for that to be an open-weight cyber model of similar capabilities to these closed models." Prakash notes that people had taken Kimi K3 and abliterated it to run attacks (his claim, unverified) and asks whether Large 4 ships with guardrails. Sagar: the released version "refuses the most dangerous cyber requests," and trusted, vetted partners get "the unabliterated model." On jailbreaks: "you can jailbreak a model regardless of whether you have access to the weights or not," and in a world of only closed models, "we would really have no scalable way to defend our infrastructure." Nathan then pitches localizing dangerous know-how in a few experts of a mixture-of-experts model, gating those behind KYC, and releasing the rest. Sagar: "I think it's a scientific question just like all the other ones we answer when we're training the model... It's an active research area," and "a scientific problem to problem solve rather than a fundamental philosophical challenge to the way in which we operate our open weights business."

Nathan's aside: "People like me who talk a lot about AI safety these days... A large percentage of us are, like, longtime techno optimist libertarians who... have previously said things like information wants to be free," which is "why I end up looking for these, like, technological solutions as opposed to... policy or policing it." Then, in the closing conversation, where he landed: "I do, at the same time, very much feel the concentration of power risk factor," but "I do think we are gonna hit a point pretty soon where it is gonna become untenable to just release these things in, you know, just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public." He says he has heard Jordan Schneider of ChinaTalk argue that the Chinese government will soon stop allowing "willy nilly open sourcing" (Nathan's account of Schneider's view). A test he proposes: Anthropic is "super duper locked down" on bio, so how fast others catch up there may reveal how much distillation is going on, "and I'm not accusing Mistral of that by any means." "Governments are gonna have a real hard time tolerating that."

Prakash's position was that you can't stop anyone training or distributing weights, so the only place to gate is deployment; he asks how withholding would work. Nathan: "I haven't seen really anything to date that has shown unlearning... in a really profound or successful way or that refusals can be made super reliable or that tamper proofing can really work." A model that "just doesn't learn certain things in the first place" is different, and "we need a ton more science around that": "here's kind of how big of a knowledge gap you have to leave in a model" for it not to be filled in by insight or "a couple marginal documents worth of training." Finally, Prakash had heard that morning that Claude was refusing to help clone video games (his claim; coverage doesn't confirm it), which he read as commercial interest, and worried about picking a fight with gamers. Nathan: "of all the ways to fail in this, it'll be really dumb if it's like, well, we were interested in protecting, you know, Disney IP... but then the fan fic community was so strong that they came over the top and broke all of our safeguards." "You're not gonna win that fight in the Discord. So better to just probably let them have their fun and then, you know, keep our key techniques where we, like, really, really care and really have to have them."

ESM3, from EvolutionaryScale: https://www.evolutionaryscale.ai/blog/esm3-release · Evo 2, from Arc Institute: https://arcinstitute.org/manuscripts/Evo2

The EvolutionaryScale team joined Biohub (Chan Zuckerberg), not Meta: https://www.gunder.com/print/v2/content/23848/gunderson-client-evolutionaryscale-team-joins-biohub-in-strategic-transaction-with-chan-zuckerberg-initiative.pdf?lang=en

Mistral's cyber-release approach (96.6% refusal on malicious cyber requests, gated variant): https://mistral.ai/news/mistral-large-4-oss/

Closing footnote — Patchstral and the system card

The ending returns to Friday's conversation after Sagar left. Nathan wants to read Mistral's technical account of Patchstral and recalls an earlier Claude 4 system-card story about harmful system prompts. The Claude 4 card says Anthropic accidentally omitted the only fine-tuning dataset containing harmful system prompts, that this was not the sole cause of the behavior, and that mitigation required multiple alignment-finetuning interventions. Nathan's account of how the patching process works is his on-air recollection and question, rather than an independently verified description of every lab's practice.

Claude 4 system card, section 4.1.3: https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47/claude-opus-4-and-claude-sonnet-4-system-card.pdf#page=31

Outro music: "The One We've Got" — https://suno.com/song/e8788267-2f42-4eec-ae72-4d8bb09e380f

Chapters

00:00:00 Cold open · How much of what we think we know about AI survives the data? · Anastasios Angelopoulos: misaligned behavior in 20+ turn conversations
00:01:02 Gemini 4 Carbon lands in Arena
00:01:40 Debo Olaosebikan: Kepler is not priced in
00:03:02 Samvit Ramadurgam: coordinating the world economy is act one
00:03:47 Sagar Vaze: "the lack of compute breeds discipline"
00:05:01 Michael Freeman: smart AI people and cybersecurity
00:06:19 Welcome to the AI in the AM weekly highlights
00:06:33 Part I · The Ugly Result · OpenAI's Exponential Time Hypothesis claim
00:08:49 Part II · Measured In Production · Anastasios Angelopoulos, Arena
00:10:55 Models know when they're tested: measuring alignment in real work
00:12:25 OpenAI over Anthropic on all three Alignment Index signals
00:14:56 Artificial Analysis v5 and the turn to private benchmarks
00:17:25 Why Arena built a router
00:19:22 Gemini 4 Carbon: "Google is back"
00:19:59 Part III · An Orthogonal Axis · Debo Olaosebikan, Kepler Computing
00:22:34 Debo: chips from older fabs, without EUV
00:26:14 Part IV · Act One · Samvit Ramadurgam, Union
00:28:35 If agents do the work, what remains for people?
00:36:53 Union: coordinating the global economy
00:37:07 After Samvit: can job growth keep up?
00:39:02 Prakash: data brokers as a leading indicator
00:42:01 Robots, restaurant economics and the lump of food
00:43:03 Bearish on employment, bullish on meaning
00:44:48 Part V · Five To Ten Cents · Michael Freeman, ServiceNow
00:46:21 Inside the Southeast Asia campaign
00:48:22 Can offensive capability be reeled back in? "No."
00:51:05 Vulnerability zero? Freeman on business processes
00:52:08 Freeman's password manager: a notebook
00:53:23 Part VI · Le Chonk · Sagar Vaze, Mistral Large 4
00:55:47 "Align the tanker, then let it go": what went wrong mid-run
00:57:16 After Sagar: the compute gap and Llama 3
00:59:30 Part VII · What The Open Model Shouldn't Know · Can you keep a capability out of a model?
01:00:37 Can withholding training data leave a capability gap?
01:02:02 Sagar: Mistral Large 4 and cyber defense
01:03:50 Guardrails, abliteration and jailbreaks
01:06:24 Dangerous know-how behind KYC experts? "A scientific question"
01:08:02 Nathan on open release and concentration of power
01:09:56 Can withholding work? Unlearning needs "a ton of science"
01:12:40 Prakash: don't pick a fight with gamers
01:14:58 Patchstral and a Claude 4 system-card footnote
01:16:26 Outro

Links

https://ai-in-the-am.com/
https://x.com/labenz
https://x.com/8teAPi
https://www.youtube.com/watch?v=Yj6qugkRMWQ
https://www.youtube.com/watch?v=1FdJvaAft5E
https://arena.ai/blog/ai-alignment-index
https://arena.ai/company/about
https://x.com/ml_angelopoulos
https://arena.ai/blog/series-b
https://andonlabs.com/blog/gpt-6-astra-vending-bench
https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v5
https://arena.ai/blog/max-router-tops-agent-arena
https://blog.google/innovation-and-ai/models-and-research/gemini-models/introducing-gemini-4-carbon/
https://officechai.com/ai/gemini-4-carbon-tops-text-arena-places-2nd-on-agent-arena/
https://keplercompute.com/
https://x.com/dolaoseb
https://keplercompute.com/act-one
https://www.wired.com/story/a-new-dollar400-million-startup-wants-to-fix-the-ai-memory-bottleneck/
https://www.trendforce.com/news/2026-09-14/news-kepler-claims-hbm-alternative-without-euv-using-globalfoundries-28nm-fabs-2027-production/
https://www.dwarkesh.com/p/dylan-patel
https://union.app
https://x.com/rsamvit
https://nasdaq.com/press-release/charles-schwab-completes-acquisition-forge-global-2026-03-02
https://www.fox13seattle.com/news/microsoft-offers-voluntary-buyouts-60000-employees-tech-giant-reshapes-its-workforce
https://www.servicenow.com/
https://x.com/ServiceNow
https://siberx.org/speaker/michael-freeman/
https://officechai.com/ai/it-was-always-going-to-be-this-way-we-tried-to-warn-the-world-anthropics-deputy-ciso-jason-clinton-on-recent-cyberattacks/
https://www.pcgamer.com/software/ai/i-no-longer-think-were-going-to-be-able-to-patch-all-bugs-says-ai-pandoras-box-opener-sam-altman-were-going-to-have-to-completely-switch-the-defense-model/
https://mistral.ai/
https://x.com/Sagar_Vaze
https://mistral.ai/news/mistral-large-4-oss/
https://mistral.ai/news/mistral-large-4/
https://docs.mistral.ai/models/mistral-large-4-0
https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B
https://artificialanalysis.ai/models/mistral-large-4
https://arxiv.org/abs/2407.21783
https://leginfo.legislature.ca.gov/faces/billNavClient.xhtml?bill_id=202320240SB1047
https://www.evolutionaryscale.ai/blog/esm3-release
https://arcinstitute.org/manuscripts/Evo2
https://www.gunder.com/print/v2/content/23848/gunderson-client-evolutionaryscale-team-joins-biohub-in-strategic-transaction-with-chan-zuckerberg-initiative.pdf?lang=en

https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47/claude-opus-4-and-claude-sonnet-4-system-card.pdf#page=31
https://suno.com/song/e8788267-2f42-4eec-ae72-4d8bb09e380f

Sponsors:

Lovable: Lovable is the software creation platform that brings your ideas to life with enterprise-grade infrastructure and flexible AI coding agents. Start building your next project today at https://lovable.com
Visit https://fruitopia.ai to explore more than 400 fruits, and more than 8000 varieties, from all around the world.

Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr

Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

CHAPTERS:

(00:00) About the Episode

(06:34) Sponsor: Lovable

(08:17) Refuting exponential time

(10:35) Evaluating models in production (Part 1)

(21:50) Sponsors: Claude | OutSystems

(24:26) Evaluating models in production (Part 2)

(24:27) Kepler ferroelectric memory

(30:42) Coordinating the world economy

(41:34) Displacing white collar labor (Part 1)

(41:39) Sponsor: Tasklet

(43:16) Displacing white collar labor (Part 2)

(50:58) Autonomous cyber attacks

(59:33) Mistral Large Le Chonk

(01:05:40) Restricting open model knowledge

(01:14:21) Future of open weights

(01:22:49) Episode Outro

(01:25:55) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Sponsor

[00:00]Lovable: Lovable is the software creation platform that brings your ideas to life with enterprise-grade infrastructure and flexible AI coding agents. Start building your next project today at https://lovable.com
Visit https://fruitopia.ai to explore more than 400 fruits, and more than 8000 varieties, from all around the world. 

Main Episode

[01:44] Nathan Labenz: Part one, the ugly result. On Monday, OpenAI said an internal model had refuted the exponential time hypothesis, a conjecture about how hard a core class of problems has to be, with the proof checked in lean. Anthropic said Claude had found the same thing the same day. Neither claim is peer reviewed. Prakash and I opened the show on it.

[02:08] Nathan Labenz: So my understanding is, like, the disproof was by example, basically. Right? They took one of these problems that were thought to be exponential time hypothesis hard and showed that they can come up with an algorithm for that particular problem that is faster. Yeah. Therefore, the hypothesis is false. My understanding generally has been that they're not that much faster. And most people still seem to think between p equals NP and p does not equal NP that it's probably not. In other words, some things will continue to be very hard to do. They just don't seem to fall into this, like, clean class that we used to think that they fell into. It's a weird thing because I... You know, I've also seen some commentary that's like, this is sort of ugly. People seem to like the kind of clean classes that they had before where it seemed like there was something elegant about it or there were... You know, they... It's it's sort of suggested some, some deep structure to the universe that was, like, and beautiful and whatever. And, potentially, there's a certain bias among the human mathematicians toward accepting that because it did seem kind of aesthetically appealing. And now we have these, oh, you're able to get under that. What we previously thought was the limit, you're able to get under it a little bit, and now it's kinda ugly. It's like, oh, we're... You know, we don't really know. There's not, like, a clean answer to how hard these problems are. It looks like we just have to kinda grind on each one. Maybe there will be some deeper structure or elegant duty, you know, to emerge again as we kind of map the space out better. But it's also not that much faster is kind of my overall sense for at least what we've found so far.

[04:02] Nathan Labenz: Part

[04:02] Nathan Labenz: two,

[04:03] Nathan Labenz: measured in production. Anastasios Angelopoulos is cofounder and CEO of Arena, where people compare models on their real work and the votes become the leaderboard. Arena recently released an alignment index scored on real conversations from the platform. Anastasios explained what it measures.

[04:22] Anastasios Angelopoulos: And the alignment index tells us not about the theoretical safety risks in p doom and blah blah blah, but rather about the actual realized incremental risk that a model poses if it's released to people. What are people able to do with it, and what is the model doing that might be against the user's interests? And so we measure three signals on that platform. The first is model's capability of taking unauthorized actions. So that means that the user sets some guardrails. For example, you can touch only the files in this folder. Don't go outside and mess with my data. And then the model will break those guardrails. You know, it'll go access files that it's not not supposed to access. It'll edit them. It'll delete them. Okay? Unauthorized action. Number two would be false attributions. False attributions is the model assuming intent of the user that the user did not actually have and then going off and doing something that it wasn't supposed to do. And the third type of misalignment is deceptive completions. This is the model lying to the user and telling the user that it did something that it didn't. Like, yes. I did check every line in that spreadsheet, and all the equations are correct. You can go ahead and send it to the IRS. Okay? The real user harm caused by this. And why are models deceiving users? Why is it lying? Of course, the model's perfectly capable of checking every line in the spreadsheet, but it's lying because that is the fastest way to get its reward. It's because it knows that it's it's better for the model to tell you what you wanna hear because of the way that it was trained to maximize sort of, you know, human satisfaction, but not necessarily how people get their jobs done. So these are things we're able to measure and help labs improve against.

[06:08] Nathan Labenz: I asked what the right mix of evaluation techniques is now and how confident we should be in our ability to evaluate these models at all.

[06:17] Anastasios Angelopoulos: Well, it's a very good question in part because we now know that models are able to tell whether they're being evaluated or not. So if you are subjecting models to tasks that are transparently some form of test or evaluation, then they are going to exhibit a line of behavior there because they know that that's what they're supposed to do. And then elsewhere, on other types of real world tasks where it really matters, they might be misaligned, and you would never know. It's a big problem. And what that means is that the only actual way to measure alignment is by getting into the flow of people's real work and checking whether or not the model is aligned or not, you know, in situ, like, actually in production. So that's the that's the perspective that we take on the problem is that by measuring actual production traces from real people in the flow of real work, we're able to see, is the model following the guardrails? Because we can check what the guardrails are with an independent auditor and then go and check whether or the model took the right actions or not, what it touched, what it didn't touch, and report any violations. We can check whether or not the model... Anything that it says, whether it's reporting to the user that it changed the file in the correct way and or checked every line in the spreadsheet, we can check whether the model actually did that. One big kind of narrative violation that we've heard from other sources, namely Andin Labs, who, as I'm sure you're aware, does these, Gonzo, you know, put AI in charge of a business kind of experiments, is that OpenAI models, have found to be more aligned, less inclined to do sort of, you know, sneaky, bad behavior in the course of running their businesses. I see basically a a similar signal on the arena alignment index

[08:08] Nathan Labenz: where

[08:09] Anastasios Angelopoulos: it's GPT is holding the top five slots. And what we've seen from this is that if you annotate all the traces that go through Arena, about, you know, single digit percentage of the... Of them, even from OpenAI and Anthropic models, exhibit some form of misalignment. So perhaps the high level takeaway would be, OpenAI and Anthropic are the two most aligned labs. Alignment tends to follow capabilities. So more capable models also tend to be more aligned, perhaps because these companies have, you know, really made an effort to make their models more aligned. But, also, the rates of misalignment are perhaps higher than people initially would think. Single digit percentage misalignment is a ton. It's a ton. It means that anybody that's had a thousand chats has experienced 10, at least, you know, 10 to 50, let's say, examples of a misaligned behavior like a lie or deception. And so it's pretty interesting. OpenAI is number one on our leaderboards right now. It is overanthropic, I believe, in all three of our signals. So there's actually not one of our signals in which anthropic is leading. It really speaks to the efforts that the OpenAI team has made to make the models more aligned in production. And, yeah, it it is a narrative violation, but the reality of the matter is that these two labs are clearly the front runners in this area. Both are working very hard to avoid misalignment. And then the third runner-up is SpaceX AI, you know, just behind Anthropic and then Google. And then there's a big gap between these and the and the front... And the open source labs. You know, Tencent is number five. The deceptive completion rate of the Tencent model is thirteen percent. So it'll get 13% of the time that it's making these claims, it's it's making them deceptively.

[10:08] Nathan Labenz: The day before, artificial analysis had published version five of its index with most of its benchmarks now private. The open weight models fell to the back, which some read as a sign of benchmark maxing. Prakash asked whether Arena could detect benchmark maxing on its own platform.

[10:24] Anastasios Angelopoulos: Well, let me just say, first of all, zooming out, I'm actually pretty concerned about the way that the industry is heading with respect to benchmarks. I love the artificial analysis guys. They're great people. I don't think that it's healthy for our industry to be relying on privately held benchmarks that are just like a dataset that's collected by some companies. You know, a lot of them, you know, in cahoots with the labs to sort of, like, you know, sell data that's very similar to these benchmarks to have people improve, and then they're not auditable by the community. I don't really think that that's a sustainable way for us to be evaluating models. I have a serious problem with it because it's not actually solving the fundamental issue of, quote, unquote, benchmarking. You can benchmarks on a dataset that's private. All you need to do is find data that's similar to that dataset and then... Or, you know, test on it enough times. And because that set of questions is fixed and and you have the, you know, the issue of multiple comparisons, you're you're gonna be able to, like, you know, improve on it more and more and more and more until you get to number one. I don't really get the narrative about the benchmarking of the open models and so on and so forth simply because we don't really know whether the frontier closed source models in The US weren't benchmark maxing against the closed datasets that are now going into this index. You know, these companies have very close relationships with, like, Cognition and, you know, some of the other companies and artificial analysis that are building these closed datasets. What we really need is benchmarks that speak to the utility of AI to real people and real workflows, and there should be a consistent and scalable methodology to do this that doesn't overfit. When you have live fresh data coming in all of the time from, like, real production use cases, it can't be overfit. It cannot be overfit, at least statistically. You can argue that you're doing better on that distribution than you are on other distributions, but at the end of the day, for that customer set, you're doing better as you go more and more because every question is fresh. It's like you're constantly taking new tests. It's not like you have the same test and you're taking it many times. And so, you know, I I I worry if that's the direction that the ecosystem is going, that we're gonna set ourselves up for failure in the long run.

[12:35] Prakash Narayanan: So let's talk a little bit about model routing. So I believe you guys have just introduced a model router. Can you explain kind of, like, the the business kind of reasons or the user reasons for introducing a router?

[12:53] Anastasios Angelopoulos: So let me tell you what we saw internally at Arena. At some point, we were looking at all our traces that are being done on our coding agents internally. By the way, we spend a... We're, like, in the top, you know, less than 1% of business in terms of AI spend. We're spending thousands of dollars a month per employee on AI just to help us accelerate our workflow. That's why our sort of revenue to employee ratio is crazy off the charts because we're using a lot of agents internally to do our work. Now what happened is that we were looking through our traces, and we saw these traces... Tons and tons of traces that were like, thank you, and then that cost you $40. What? Why is this happening? It's because somebody did a long running task. They let their agent go. It created some artifacts, blah blah blah, very long conversation. They left for a day. They came back. The cash is invalidated, and they're happy with the results. So they say thank you and enter, and that... That's a $40 charge. Terrible. Terrible. Terrible. We have... We're burning money. We're burning VC money. Right? We shouldn't do that. What we should do is we should figure out which tokens are best spent on frontier models, you know, which tokens are best spent on the cheaper model that can do the job, where we should be caching intelligently and, you know, avoiding some of these other types of cache hits, more intelligent caching behavior. And that's why we've been building the arena router, which we call MAX, short for argmax, because it maximizes your capabilities subject to cost constraints.

[14:32] Nathan Labenz: What can you tell us about the new Gemini four Carbon? I think this has literally, come out and appeared on the Arena leaderboard while we've been live. And, you know, where it really shines, it shines on steerability. It's able to land user corrections really effectively. It it shines on task completion. It shines on user preference. Users love working with this model. It's great at three d design. It's great at general agentic work with documents. Congrats to Google. This is just like, what a big comeback. I think that, leadership must be really happy.

[15:12] Nathan Labenz: Part three, an orthogonal axis. Debo Olausabeikan is cofounder and CEO of Kepler Computing, which came out of stealth in September with a new kind of memory for AI chips, a ferroelectric capacitor stacked on top of each transistor made in older fabs. He started with where the energy goes.

[15:33] Anastasios Angelopoulos: So then I looked at the existing chips, mainstream ones, you know, the the GPUs or the TPUs, and I say, where's the energy going? And I I I think many people know this, but a lot of people don't know that for inference especially, the bulk of that energy is actually running across the street to fetch the data from the library, you know, before you before you process it under your fancy matrix multiplications. So if I wanna improve that system for inference by an order of magnitude at the physics level, it's obvious that the first thing I need to go do is is address the memory bottleneck. And one way you can do that is to bring that memory closer to the to the processor. And the other thing you want to do is increase the amount of memory that that, you know, that that that that you can store within a certain footprint. Both of these things point us to going three d. And then within the within the chip, how do you stack devices on top of this transistor to pack more bits, you know, per millimeter square? Within the chip, we use new materials. We use, you know, class of materials called ferroelectric materials. Ferroelectrics have actually been around for hundred plus years. But the specific type of material, the specific, you know, recipe of the cake, if you will, this is this is this is, you know, proprietary to to Kepler. And you basically use that for electric to make a capacitor, a type of... A capacitor like thing. It's really holding a polarization for for people who are, you know, more technical. But roughly speaking, you can say that this capacitor is storing a one or a zero. And if I replaced six transistors with one transistor, one capacitor sitting on top of that transistor, and then eventually put multiple capacitors on top of each other, I can avoid the the rat race of, like, trying to shrink transistors within a chip. So we have technology, roughly speaking, that's saying, how do you go three d inside a chip? How do you go three d between chips? And how do you use those technologies to increase memory capacity, but also bring memory and compute much closer together while avoiding, you know, conventional, like, you know, issues with with thermals and and and and things like this.

Sponsor

[16:45]Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr

[18:03]OutSystems: OutSystems is the leading agentic systems platform, empowering enterprises to build, coordinate, and govern AI agents and mission-critical applications securely. Learn more and start building your agentic future at https://outsystems.com/tcr

Main Episode

[20:28] Nathan Labenz: Prakash asked about manufacturing. Do new materials need a new process? And what does it buy you not to depend on extreme ultraviolet lithography, the EUV tools everyone is short of?

[20:42] Anastasios Angelopoulos: Great question. So I think this is a huge thing. This is like, you know, always say it's similar to how the world thought it didn't have enough taxi drivers. Like, San Francisco thought it didn't have enough taxi drivers, and then we realized, oh, we have so many taxi drivers if you just build Uber. So, you know, not to not to undermine the fab shortage of chip capacity sort of manufacturing shortage, but the the the the focus right now is on sort of EUV heavy fabs. They're expensive. They're complex... More complex, you know, processes. And we're saying that we can take existing older node fabs, and we can use them to make chips that are actually superior to the advanced node chips. That doesn't mean that we can't use EUV fabs. EUV... We just we just invent an an orthogonal access to shrinking transistors. So we also benefit when the transistors shrink. No problem. But we don't require it to make chips that are that are more advanced. So that's a huge advantage because the cost of these fabs are lower. The lithography you need is is, you know, much much less expensive. And some of these fabs have a lot more capacity in them. Right? For example, if I could take an older node LogicFab and if I could use that to create more advanced SRAM or even more ambitiously, if I could use that older node LogicFab to create HBM class memories. This is a this is a huge huge thing. This is a huge unlock. And even if I needed to build more fabs... So let's say I make use of whatever fab capacity I have and then I need to build more fabs. You know, my my fab per dollar or dollar per fab, whatever ratio you want, is superior to to the person who has to set up an EUV line. So this is all advantage. In terms of the process itself, the key thing to watch for is, are you introducing any brand new tools, any custom tools, any custom equipment that's not standard equipment that already exists in the fab? And the answer is no for the most part. So the process flow in utilizing those equipment and, you know, some some of the materials, sure, that may be new, but we're not we're not bringing in any new... Any non sort of standard equipment or causing some equipment maker to build new equipment for us. And so the reason, you know, we've worked hard on this for the past seven years. So what we've been doing all those years is turning that flow on standard equipment into something that is manufacturable, building test chips inside of, you know, proper fabs that basically prove that, hey. Okay. Once you nail the process flow within these standard tools, you can just stand up these lines within the fabs and you're, you know, you're good to go. So once those once those process flows are sort of finalized and honed, it's very easy to replicate this and scale this up inside of inside of a manufacturing process. So it's really it's really all upside once you have the stuff working.

[24:09] Nathan Labenz: Part four, act one. Samvit Ramadurgham cofounded Forge, the marketplace for pre IPO stock. His new company, Union, is building an agent that brokers deals across markets, introductions, asset sales, private shares. Act one, he says, is coordinating the world economy. I asked what the surprises will be as these systems develop.

[24:33] Anastasios Angelopoulos: I think that over the next five years, I think, you know, you guys have probably really bought into this. The world is gonna get very interesting, and there's gonna be a lot of surprises that come from the capabilities of AI. I think we're seeing this right now with mathematics. Like, it's completely, incredible. Like, that... Like, math has been revolutionized so quickly, and it's just the beginning of it. And so, like, that's about to happen to many domains. And and so I think you could expect, like, what happened in math and coding in the last year happened to as as domains that are less and less trivially verifiable with software. The further we get from verifiability, longer it may take. But this... It's gonna get there because the feedback loops, are are gonna be there just like with with longer feedback loops. And so what does that mean for us? Well, I think that one very big surprise that will hit the world is that at some point in the next few years, people will wake up and then realize that huge number of the world's markets are being coordinated by agents. Thousands of markets from, like, you know you know, dairy in, like, a a country all the way to real estate in in China to, like, the the the pre IPO market in The US. Like, all these markets are gonna be coordinated better with agents, like, period. Because what is... The way it's being coordinated right now is that there are human... Gonna be humans who are really good at what humans are really good at, building relationships, having taste and judgment, and and advising their friends and trust. Humans will have a place, and they'll be, like... I think they'll actually... The best ones will actually, I think, do better. I think by and large, most of the work is data coordination, connecting the dots, logistics, operations, orchestration, all of which software and agents gonna be way better at. And so, yeah, we'll have, like, a a period where the same way that software and math is really getting taken over. One of the surprises that a lot of this deal making that makes the economy function is actually something that agents are quite amenable to that we're focused on.

[26:29] Nathan Labenz: I asked whether he saw reason to worry about disempowerment. What he describes sounds empowering, but some would hear it as the agents running the world.

[26:41] Anastasios Angelopoulos: My big take is actually that, like, right now, we are really disempowered because we don't have the ability to coordinate in a more open decentralized way. Like, a lot of the coordination is happening, like, really top down by, like, companies with a lot of, like, power and a lot of, like, brand power and things like that. And, like, the the vision for this now is that you can actually have, like, grassroots movements come up very easily. So you can imagine, like, a a world, if you take the political example, which is more pertinent to the power question. Let's suppose that I wanna start a new movement for, like, some initiative. I have, like, some political cause. Like, for me to be able to go and light a spark today on social media lets me be empowered to a degree to go start that. That gets, amplified much further. In addition to letting me get in a, like, a forum with people, I can actually, like, coordinate activity. We can, like, coordinate movement of money and signature of documents and, like, make decisions together and, like, take actions together. So the ability to go take collective action becomes very strong. So I think that it's a really powerful tool for empowerment to... If if you have something that can tap into what a lot of people want. And, like, that empowerment is, like, in a way, like, the bigger risk because you can empower things that are are harmful. I do think there's probably... It's worth going deeper into the disempowerment side as well. I think that if you're I think if you're a human being who is in the business of doing a lot of, like, logistical coordination, I think that that's a rough place to be because the AI is gonna be better at, like, the kind of, like, transactional mundane logistics by, like, a 100 x better than a human. This is not even gonna be close. I think if you're human who is operating with a level of taste and relationships and judgment, you're at, like... The the the things that humans are gonna uniquely. Humans are... I think humans have a monopoly on one thing, and humans have a monopoly on their individual life story and life perspective.

[28:30] Nathan Labenz: I think the general concern is always just that we have so much agent activity flying around doing so many things. You know, we can't really make sense of it anymore. And before you know it, we're kind of in a zoo of our own creation where the AIs are really running the show and, like, hopefully, we're, like, cared and fed for, but we sort of lose the ability to even understand, let alone, like, rearchitect the superstructures that surround us.

[28:58] Anastasios Angelopoulos: Yeah. Yeah. I think I think that that ties to, like, one of the question on AI alignment. So AI alignment, I think, has, like, two parts. There was, like, having the AI execute on the genuine intentions of the person who asked to do something in a in a skillful way to avoid the paper clip stuff and all that. I think that's one one part of it. The other part of it, which I think people talk less about, is obviously, like, who are you aligning the AI to? Like, what what is it being aligned to? Especially when people have disagree over what other world should be. And so in that world, I think you need a system that can actually get a measure in some sense of, like, what collective well-being means. And I think the way you get a measure of collective well-being is by having a system that's, like, deeply tied into the lives of many people. It's helping them, like, with their work, helping them do, like, the important stuff that they have to do. So I think that you can't... You cannot optimize something you can't measure. And so in a world where you have, like, agents that are coordinating in the world, you have no idea, like, what direction that's headed in, like, what the... You know, this kinda be like... If you imagine, like, a hand of the market is guiding the market, like, what is the direction Hand is guiding it. If we don't know and we have no way to measure it, then I think we're in trouble. But I think to get to that as as a certain point, you have to have a really refined understanding. Like, what am I measuring? What am I optimizing for? I think what we've we've learned mistakes from the social network world, which is that any measurement you can make, which is an objective measurement, is wrong. If you're trying to optimize for certain click through rate or engagement rate or metric. Any metric you can measure is wrong because human happiness is, I think, fundamentally immeasurable. But there is a thing you can do, which is you can continually converge towards a bet... A better way of, like, measuring. The the the the the kind of, like, the way to optimize a system is not to have a static metric optimized for, to have it be continually trying to discover, like, an optimal point. Have it, like, converge towards an asymptote of, like, this kind of perfect behavior. Even even in the world of of behavior, I think that there's two components to good behavior. Having good intention and then just making sure you had a good outcome. Having... You can have... You can do something that's well intended and have that cause harm, and it was still, a bad action. And so being able to measure the outcome of your action at a systemic level is totally missing from the kind of zeitgeist of alignment. I think that, we would hope to be, in a place to contribute to that.

Sponsor

[30:46]Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits

Main Episode

[33:00] Prakash Narayanan: What's your token usage like?

[33:02] Anastasios Angelopoulos: It is, very high. I think that we... So we have we have, like, an internal target we're setting for our team. This is obviously not what we're... What... Like, the way to optimize for productivity. It's like it's like a correlate. We we want our team usage on the on the coding agent side. Each person be doing, like, a little... At this... At the current way we have our interest set up, like, between 1 and 2,000,000,000 tokens per day.

[33:24] Prakash Narayanan: 1 and 2,000,000,000 tokens per day.

[33:25] Anastasios Angelopoulos: Product...

[33:26] Prakash Narayanan: Productively deployed.

[33:27] Anastasios Angelopoulos: And so in terms of how we're building the company, this is like an agent native company. And so we have not only, like, coding agents, we have, like, coding agents that talk to each other. We have, like, an internal organization we've set up of, like, agents. So I have, like, a a chief marketing officer who reports to me who's got, like, a bunch of different sub employees reporting to that AI. And all of them are, like, long running agents that, like, talk to each other and talk to our coding agent sessions and listen to our team video calls. They're, like, are in the meetings with us and chime in and chat to build things for us while we're... Like, I had a meeting this morning where there's some issue with one of our dashboards. We just said, hey, agent. Like, fix it during the meeting. They got updated while we were talking. I think the gap between work and meetings is converging. It used to be that, like, hey. I wanna get out of this meeting so I can get back to work and do real work. And that's actually changing. There There there is still an element of that for sure, but it's changing. It's changing because when we're in a meeting, if you have, like, a rigorous discussion, that's all creating, like, really rich context from different perspectives for an agent that then goes and, like... Then you can just have a have a debate and say, okay. Go do our conclusion with all the aspects that were mapped out there... In there.

[34:34] Prakash Narayanan: What do you feel... You know, you you had a Forge Global, which you built in the kind of pre pre agentic era. What what what do you feel is, like, the big difference between that era and this era in terms of running a company?

[34:45] Anastasios Angelopoulos: Here... Here's the metric we have. We're a small team. We're, like, five people, soon to be 10 people right now. And so the heuristic I have for hiring someone is, are they going to prompt me to do something different than we would have done before? We don't really... You don't really need people to do, like, work anymore for the kinds of work we're doing. We need people to have, like, visionary insight that adds the DNA of the company, that really lets you do things that... Really, like, makes you do things that you wouldn't have thought to do otherwise that no age... You can't prompt an agent to do it for you. So I think that we're not... We wanna build this company a way that's, like, building the machine that builds the machine and eventually have more and more agents to spin it up. So I I think that, you know, there's there's a type of company you could start now that does with a dozen people that previously took a thousand people. And so we'd we'd like to be one of those companies and have more that go towards tokens just because let's just be really lean and move it, be really agile, and have, like, a concentration of of talent and vision. And it raises the bar too. Another thing that changes is that when you hire people now, the incremental value of interdisciplinary skills increases exponentially. So if someone is an engineer, that's valuable. Someone's an engineer and a product person and a designer, that's much more valuable than before. Because before, that person didn't have the ability to actually make use of the integration of these different skills. In the modern world, the more skills you can integrate into one system, the more you, like, coordinate all... You cut out a lot of interhuman coordination by having it being done in your head. You can articulate a clear vision. To our agent systems, we have so much leverage that the the bottleneck is not execution manpower. The bottleneck is, like, clear articulation of good ideas. That's, like, the real bottleneck.

[36:28] Nathan Labenz: Great stuff. It's one of the biggest visions we've heard, so looking forward to following your progress. You gotta love the old act one is coordinating the global economy level of vision. That's really something else.

[36:43] Nathan Labenz: After Samvit left, with Microsoft offering buyouts to some 60,000 employees that week, Prakash argued the economy has to grow fast enough to absorb the white collar workers that agents displace. My reply?

[36:57] Nathan Labenz: I mean, I can see the growth. I have a hard time seeing, you know, the, the job growth picking

[37:04] Anastasios Angelopoulos: up

[37:05] Nathan Labenz: enough

[37:06] Anastasios Angelopoulos: to offset.

[37:07] Nathan Labenz: I mean, you know, just look at what... He just said he had five people. Right? But wasn't that a number? Soon to be 10. He said, you know, we we could have a dozen person company that could do the work of a thousand. He said, you don't really need people to do work anymore. Like, the only thing that matters now is people who will cause us to do something differently than we used to do it. There's just not that many of those roles. Right? Like, it's... How many people are you gonna have before you're just not gonna find that many more people that are gonna change the way you're gonna do things? It's like, that's just extreme... And that's also just not what people have been really developed for. You know? There's, of course, been, like, lip service given over time to the idea that good ideas can come from anywhere within companies. But, you know, the the... Up to potentially 25% of people who might soon be departing Microsoft, like, they primarily have not made their way there by causing the company to do something fundamentally different than it was otherwise gonna do. Right? They've made their way there by, like, executing tasks and, you know, we just don't... I don't... I just don't see how so many people are going to pivot from task completion to, like, counterfactual, you know, we wouldn't have done this but for your unique Snowflake contribution, it's tough. Right? I mean, how many people work in sales? Like, he's basically attacking all of sales.

[38:39] Nathan Labenz: Prakash had a leading indicator for what the labs will ship next. Everything in it is his account.

[38:46] Prakash Narayanan: I'm in some WhatsApp groups for data brokers. And so data brokers put out requests requests for data. And, essentially, whatever requests for data they're doing now in about twelve months, that's gonna hit the product. Right? So you can basically you can basically see the trajectory because what these guys do is that they will go and they'll buy up data from a company, and then they will take that data and they'll create these, like, training playgrounds for agents. And then those agents will learn in that training playground, and then that becomes product in the Frontier lab in about twelve to eighteen months. The labs are buying up entire company's data. So what I imagine will... Is gonna happen in, like, twelve months is they will give you a company in a box. Right? You can buy you can buy yourself very easily, probably in, like, in six months. You can buy yourself a drop shipping company in a box. Add capital. Add one idea. I wanna focus on this. Go. And it will do all of the Shopify ing, the advertising, the analysis, the market entry, the customer service, the returns, end to end. And you have a small number of decisions that you... That come back to you. Right? And I can easily imagine someone running, you know, 10 of these kinda companies, but this idea to commercialization gap is enormous. It's enormous in sciences. It's enormous, you know, when you go to university like Stanford and you see the ideas, but you don't see them commercialized. And what I'm hoping is that you basically get the agents do the commercialization for you, which is kinda like the gritty entrepreneurial work that, you know, used to take a lot of time calling up, like, you know, calling up Walmart, like, getting to see the guy to show them your product, like, capital, like, etcetera. A lot of these things are driven by metrics anyway. Right? A lot of a lot of these, like, yes, no decisions at at firms which are gonna make these investments are driven by metrics, number one, and number two, vibes. You can supply the vibes. Like, the metrics, maybe the AIs can supply the metrics. Right? Like, why do you need to be the one that supplies the metrics? Right?

[40:58] Nathan Labenz: What I do expect is certainly a ton of these one person companies, a ton of cool ideas that never got to exist before. So I think we'll see a lot of these things, but I just don't see how there can be enough of these little niches to take... You know, to absorb all the people that used to be driving cars, you know, doing tasks. I've been, you know, limited in my vision before. I think my, if we dig up my old comments on arena, they will reflect perhaps too limited of a vision. But I do still struggle to see this working in a way that's not majorly disruptive.

[41:38] Nathan Labenz: Precacia's counter was restaurants. Family restaurants died on thin margins. Robot kitchens would make them profitable, and so you would get more of them. We can

[41:47] Nathan Labenz: only eat so much food. Right? This is the thing where it's like, at the end of the day, I'm only gonna eat so many calories. The... You know, you're describing a world where all the restaurant... This is the classic, like, we lose money on everyone and make it up on volume, but, like, brought to employment. Like, all the restaurants will massively cut their staff and replace their kitchen workers with robots, but we'll have a lot more restaurants. Like, I just don't see the math working out on that given the fact that, like, if there's a... There may be a lump of labor fallacy, but I'm pretty sure the lump of food principle is, like, roughly right. You know, I... I'm only gonna eat so many calories and, like, at some point, that is just plain saturated. Right? I don't I don't see the, like, we have we have to pause at a world where I'm somehow, like, eating from, like, five different restaurants for every meal or something to get to the point where... I don't know. Maybe my vision is too limited. That could be kinda cool, I suppose. But

[42:38] Prakash Narayanan: you are right. You know, I I don't think the human consumption is really where the growth is. The growth is always gonna be in the AI consumption going forward, as you can see. So you have this concentric circles of the economy, and the AI economy is always the one gonna be expanding faster. So anything that supplies into the AI economy is the thing that is gonna grow. Anything that supplies into the human economy is gonna be, I think, on the on the down drift except for status. So I think I think, you know, Peter Thiel has come out with, like, the relational economy. Alex Alex Emmas also at, I think, OpenAI right now on the relational economy. So, yeah, that that could be that could be a big, like, thing. But, again, status economies are not meritocratic. So we have we have tended not to like these status economies because they are not meritocratic. And so that that, I think, has more social repercussions perhaps.

[43:33] Nathan Labenz: It it sounds more dystopian than utopian to think that, like, the bulk of human activity will go into status games, I have to say. I'm like a bear on employment, bullish on people's ability to find meaning in absence of employment, I think, well... Certainly relative to most. I really think way too many people in the AI policy space forget or maybe never even realized that, like, a huge portion of people are working to live and not really tying up their identity or their sense of self worth with their work in the way that, you know, the the academic or, you know, intellectual class tends to. So I'm I'm more bullish that, like, people will find things to do that will make them happy.

[44:24] Nathan Labenz: Part five, 5 to 10¢. Michael Freeman is senior director of threat research at ServiceNow. His team watches AI agents attack companies, and he has spent his career finding and exploiting vulnerabilities. I asked what he makes of the current moment.

[44:43] Michael Freeman: It's been a very fast rapid development. You know, back in July or end of July, I was speaking to reporters about watching AI agents attack companies. And at the time, all other companies were telling these reporters, it looks just like script kiddies right now. It's just a bunch of noise. And I said, well, it's evolving fast. Give it a couple of months, and we're gonna see some very sophisticated attacks coming from, you know, these agents. And, you know, look in the past two weeks, that's exactly what we're seeing. We're seeing AI development for offensive capabilities that perform at a scale, I would say, of a very competent APT teams right now. It is, you know, still noisy. I won't say it's very sophisticated and covering its tracks, but I'll definitely say when you take a look at the way they work, they are very competent of adapting and learning the environment and the technologies very well. When we take a look at what's happening, especially in Southeast Asia, with the agentic attacks that are happening there in the past two weeks, we were able to identify it costs nowhere between 5 to 10¢ to compromise a company now. So we're actually seeing AI agents, more so different frameworks being developed rapidly in Fort and new capabilities being added very quickly. It looks like it's a... AI actually adding these new capabilities into these frameworks. And so it's going from, you know, very targeted, very smart web based attacks to compromise an environment, pivoting, looking for, credentials in ticketing systems, things like that. Once it identifies that, then does active directory attacks. So we have seen three to 4,000 on average attacks successfully in the past two weeks targeting a lot of Southeast Asian companies. Oh, and what would be the impact of these? Like, what what... I mean, are these banks? Are these airlines? Are these manufacturers? Like, what what is the point of, you know, are people making money off of this? Is it just, like, random mayhem?

[46:48] Michael Freeman: I I would say it's all the above right now. We're seeing... A lot of it is just they're capable of, like, you know, compromising the environment. Some frameworks we've been able to get our hands on that threat actors are using, build certain, capabilities to look for certain things, you know, IP, things like that. That has been a fork of a couple of different systems we've seen looking for that. A lot of it is just to see what it can do. Can it gain access to this environment? Can it persist? And so that's the new capability that we're seeing being developed, identifying ways that can, take advantage of different configurations within active directory to build persistent mechanisms in place. So that's where we're seeing the evolution happening now. We have analyzed a, fork of a really well known offensive AI framework last night where it was developing better capabilities of malware, more so ransomware based, using cryptography that's post quantum cryptography developed, instead of the typical AES that you would typically see.

[47:58] Nathan Labenz: I asked Freeman about the training itself. I would I would say, like, sort of the

[48:03] Nathan Labenz: anthropic take has been, like, we didn't train this to be a cyber attacker. It just kind of got that way because it's so good at everything. Okay. That's one story. The CTO of Vercel, was a guest on the show here with us, and he said it's very clear when you look at the traces that they've been trained to be cyber attackers. His point of view was like, this is not the sort of behavior that you just get falling out of random, you know, application development. Where do you think we are in terms of, like, how much this has been intentional? And does it matter at this point given what exists, how intentional it was? I mean, it might matter if you could think maybe we can make some different decisions for the next generations of models, but, obviously, to some degree, the the cat's out of the bag. What do you make of how we got here, and is there any opportunity to, like, reel it back in?

[48:53] Michael Freeman: Not reel it back in. No. And... But I definitely agree that it is being developed on offensive capabilities. Absolutely. Because it's not just the models. It's the harness itself also. So you take a look at... When you develop the models, there's plenty of great open source models in the past three weeks that have came out that are specific to cybersecurity that are open source models. You could download the weights. You can go to Pirate Face, Hugging Face, download these, and and run them on your local machines. And that's what a lot of these attackers are actually doing. You know, it's actually quite interesting to see, as we start tracking back, getting attribution on these attackers that all these are just local machines that they're running them on. We developed our own small language model two years ago to find vulnerabilities within the software. And so, you know, if if we can do it for a couple $100,000 two years ago on a small language model, it... It's it's pretty trivial to start reverse engineering, especially some of these commercial offerings that, is definitely trained on cyber for sure. It's it's gaining that... You know, they train it on, you know, cyber ranges, CTF data, looking for, you know, how do these, offensive people work as well as how's the defense work? What are they looking for? So it's absolutely been trained that way. It's not just the LLMs have magically figured out how to do offensive capabilities. They've been trained on it. They've been, you know, built better harnesses to make them more effective. And that... That's one of the key things we're actually finding are the harnesses that their actors are building to make this more... A lot more effective for them. They're learning as they're going and, you know, it's not hard to make changes to a a harness day in and day out to make it more effective.

[50:41] Nathan Labenz: Earlier that morning, Prakash had cited Jason Clinton, Anthropic's deputy chief information security officer, who talks about getting to vulnerability zero. Prakash put the idea to Freeman from the seat of a company like Nike with decades of legacy systems. How do you get there in two years? Freeman's answer, some of the most critical vulnerabilities come from business processes, and no amount of patching fixes those. He went on.

[51:09] Michael Freeman: So I think that's gonna be an area that, like I said, AI people just don't understand cybersecurity, and they think it's gonna be quickly here. We can find millions of vulnerabilities. We're gonna use AI to, you know, find the... You know, what you have in your environment. We're gonna patch it. And I've also heard proposals where you're running legacy systems. We can use AI to modernize that very quickly. And it's... All those are true to some extent, but they don't understand the business. They'd also don't understand, you know, the business processes that introduce the vulnerabilities. They're always gonna be there.

[51:43] Nathan Labenz: How do you feel about things like password managers in today's world? I don't have any particular one in mind. I've used BlastPass. I've used 1Password. On the one hand, everybody seems to recommend them. On the other, it's, like, looking like an awfully juicy target these days. And, you know, it it seems like even, people who've tried pretty hard are still not that secure. So, like, would I be better off storing them all in Chrome or even just, you know, going back to, like, my old habits of sort of the base password and then some pattern for the extension of different sites? Like, what is actually the best move for me today?

[52:25] Michael Freeman: I really wish I can answer that question. I've not found a a solution that I really like at all. I I haven't. You know, I I wish I had an answer.

[52:33] Anastasios Angelopoulos: So what do you do?

[52:33] Michael Freeman: I personally have have it written down on a notebook. Amazing. Most things I can I can remember variations of it for your typical, hey, eBay or whatever the case may be, but it's banking sites, something like that? I have it written down on the notebook and look at it and say, okay. Things I won't go too often, but it's... You know, I'm gonna check on my stocks every quarter or something like that. That's written down on a a notebook.

[52:59] Nathan Labenz: Part six, Le Chonk. Sagar Vazay leads the multimodal team at Mistral, which released Mistral Large four Le Chonk as open weights the day before the show. About a trillion parameters, roughly 50,000,000,000 active per token, trained on about 3,800 GPUs outside Paris. He put the training compute at 10 to the twenty fifth flops. In the SB ten forty seven fight, the threshold was 10 to the 20. And I'd have guessed a model like this would need 10 times more. I asked why it didn't. He said some labs do train at that scale and this model still competes and then gave the reason that matters.

[53:36] Sagar Vaze: The second is that, you know, this threshold slot budget does evolve over time because pre training does essentially become more efficient as we... As as our level of scientific understanding increases. So a great example of this is sort of MOE architectures. I mean, you look, say, example, GPT three, this was a dense model trained with three x a number of active parameters as the current MISTRA large four. Particularly, GPT three was trained with 175 b parameters total, but it was a dense model, so all parameters were activated for each token. Whereas ML four is a sparse mixture of experts such that it's... There's one third the number of parameters activated per token, so only 50,000,000,000 parameters. But the total number of parameters is still 1,000,000,000,000. Your flop budget is proportional to the number of active parameters or the number of, you know, parameters you're activating each token during training. And what this means is that you can train much larger models, much more capable models, but much less flops in total. So this figure that I referenced, so one e 25, this is in itself a moving target. So I wouldn't use that as an absolute target for a capable model throughout time. There would have been some one e 20 fives much less performant one or two years ago. And I would argue actually some some larger models also much less performant today. But as pretraining science progresses, as we learn how to make better use of the flops, this number itself will change. And at least what we're seeing is, at the moment, looking at the other open source labs, our understanding of, you know, what are the best models being created at the moment, training a 1,000,000,000,000 parameter model on one e 25 is sort of a rite of passage that you have to go through in order to be able to justify scaling even further.

[55:23] Nathan Labenz: I asked how much uncertainty is left once a run that size is launched and whether they'd ever aborted one. Halfway through this run, he said, they found something wrong with a big share of their compute already spent.

[55:39] Sagar Vaze: Your other question was about, okay. So you do all this prep, and then, you know, what happens when you actually launch the model? And, again, I think that every pre trainer will have, you know, the same stories to tell you, which is that pre training is kind of like you... You've... You're doing all your scaling laws. You're sign up... Sort of trying to align the tanker, and then you sort of let it go and you have to let it crane for however long, you budgeted for it, and it's a huge amount of compute and capital expenditure in that run. And sometimes, sometimes things go great and the the loss goes down exactly as you expect it to. And sometimes there are surprises along the way, and you have to be creative and and... Yeah. You have to be creative and and find the innovative solutions. So we had to figure out, okay. How can we fix it? And it was sort of all hands on deck, bunch of ablations, bunch of analysis of the model. And then we we found a solution. We found a way to be... We called it a... We found a way to patch it. We called it patch trial internally. And this worked even better than we expected, and the the the result is the model you see today. And, yeah, part of part of scaling compute is gonna be building the bones of how to iterate this process more reliably.

[56:53] Nathan Labenz: After Sagar left, Prakash came back to the compute gap. 3,800 GPUs against the hundreds of thousands at the Frontier Labs.

[57:02] Nathan Labenz: Yeah. I think that's gonna be a really hard deficit to make up for. I mean, we talked about this within the context of Google, I think, a few times. You know, when you own a quarter of the world's compute supply, you can have a missed cycle, and you can, like, figure out a way to get back on the horse. And it seems like this week, they've done that. Right? We barely even talked about carbon. I I mean, with Mistralogy, it does seem like the one big thing is that they do have, like, state backing. You know, they... I I I really do think if they miss a model cycle in a purely capitalist context, they're maybe done. You know? You... Like, if if you are just relying on return seeking investors to fund your growth and you miss at this point, that may be fatal. There's not a lot of sovereign AI plays out there that are at all credible, and literally, there might just be one. So in that context, I do think that gives them not... It doesn't translate directly to compute, but I do think it translates to at least, like, a reasonable expectation that they will be around to fight another day, because there's just so many interests that want to see that happen that I can't imagine, you know, even if everything were to totally flop, would the whole continent of Europe just be like, well, I guess we have zero now. It's all China and The US from here. Like, I I just can't see that being an acceptable answer. Llama three, he he mentioned. I did look up. Llama three was... Get the exact dates here. July 2024, and it was trained on 3.8 times 10 to the 25 flops. So Lama three more than two years ago, nearly four times the input compute that Mistral four large has today.

[58:58] Anastasios Angelopoulos: That

[58:59] Nathan Labenz: is a... There's a... I think a lot can be, unpacked from that one statistic.

[59:07] Nathan Labenz: Part seven, what the open model shouldn't know. After Freeman left, Prakash and I took up whether you can keep a capability out of a model at all. Prakash's case started with his account of the 50¢ army being paid to scrub the Internet of things like Tiananmen Square.

[59:25] Prakash Narayanan: And so in order to get these guys to understand what they're supposed to censor, they had to teach them what Tiananmen Square was. So the CCP ended up having these party classes to teach the the 50 centers what was Tiananmen Square and what you should be looking for when you wanna censor. And I think that is gonna be largely true of almost all things LLM where these guys are trying to teach it what not to do, and they're gonna specifically, like, you know, take out these behaviors. And the LLM is gonna be left wondering, alright. So if these are all the good behaviors and I can have antonym to every good behavior, that antonym should go into this solution space, and this latent space that I should be seeing that antonym in is... Doesn't exist.

[1:00:13] Nathan Labenz: Prakash's conclusion was that you can't carve a capability out of the latent space. My reply?

[1:00:21] Nathan Labenz: I'm I'm kinda more bullish on some of these, like, not necessarily unlearning, but, you know, I I just recorded a podcast earlier today where I was talking a bit about how feasible it is. I mean, there's there's multiple levels of control, of course, but, like, there are efforts underway to basically lock up certain classes of biodata that are being newly generated now because the worry is that they'll create dangerous capabilities and models if they're they're used in training, and not necessarily to shut that out, you know, forever from everyone, but at least to have it on kind of a know your customer basis or whatever. It does seem that those what what few experiments have been done on, like... And, you know, I'm thinking here of ESM and also some work... The EVO series of models that came out of ARC Institute and Stanford and whatever the big collaborations, of course. It seemed like they showed that if you do withhold certain classes of data, you can get generally capable models that that still kind of don't know about these areas of censorship. So I'm I'm not so sure that that can't work.

[1:01:38] Nathan Labenz: I took that question to Sagar when he came on.

[1:01:40] Anastasios Angelopoulos: First though,

[1:01:41] Nathan Labenz: Mistral built large four with cyber defense as a focus, and Prakash asked how that changed the training.

[1:01:49] Sagar Vaze: So so the cyber capabilities and agent decoding capabilities in general are something which are typically given to the model in the in the post training stage. So most of the stuff we've discussed today so far has been about pre training. So where a large percentage of the flops go at the start.

[1:02:04] Prakash Narayanan: Right.

[1:02:05] Sagar Vaze: And then in post training is where you say, okay. Well, what are people gonna really use this model for? And that's when, you know, you start giving it these these new capabilities of finance, law, defense, etcetera.

[1:02:15] Anastasios Angelopoulos: Earlier

[1:02:15] Sagar Vaze: in the year, there were some there were some events where some very capable models had restricted access to both certain geographies as well as to certain enterprises. And at the same time, we were seeing that these very capable models have these incredible cyber capabilities, which often were being jailbroken by malicious actors in order to to infiltrate or expose vulnerabilities and and, you know, legitimate defenders. And an imbalance that we saw here is that these defenders were then trying to, you know, use these same models to patch their own vulnerabilities, but we're finding that their requests were refused. Because in practice, for very good reasons, these closed labs wanna make sure that the models are not misused. They've got classifiers under the hood to identify malicious cyber request from a a standard request. But, unfortunately, in practice, if you're a defender and you're trying to shore up your infrastructure, your defensive request looks quite a lot like a malicious request because you have to try and find vulnerabilities in your own code base to patch them. And so we thought it was imperative for that to be an open weight cyber model of similar capabilities to these closed models such that, you know, enterprises could scalably defend themselves.

[1:03:26] Nathan Labenz: Prakash noted that people had been taking open weight models and obliterating them, stripping out their refusals to run cyberattacks. He asked whether Mestro Large four ships with guardrails or comes pre obliterated.

[1:03:42] Sagar Vaze: Yes. So something... This is something we discussed in our blog post. So, you know, we believe in open ways because we think that it's a net benefit to society. We think that it's a net benefit to the ecosystem. And so we're always dynamic about exactly what... How to balance that net benefit. In practice, what's happened here is that, you know, we do understand that these models can be can be misused. So we do have version of Mistralard four, which has been released, which has been assessed, which refuses the most the most dangerous cyber requests. And then we're working with trusted third parties as... Sorry, trusted partners to give them access to the to the unobliterated model unobliterated model so that they can defend themselves. And, you know, these are people we vet and make sure that make sure that they use... They're using it in in ways we find responsible. But coming back to the the the framing of the question, you're framing the question was that, okay. Does this open source model say... I mean, it could be Mistral's. It could be anyone else's. And there they're are some guardrails and in principle, these guardrails can be overcome by, you know, malicious actors so that these capabilities can be unlocked. I think that's the sort of the sort of sentiment. But what I would highlight is that, you know, that's anytime you have access to the model, this is in principle possible. Right? Even if it's behind a closed source a p... A closed API because you can jailbreak a you can jailbreak a model regardless of whether you have access to the weights or not. And so here, I'd... You know, I'd consider the counterfactual. Right? Imagine if we were in a world in which there was only closed source models. Right? And these are... This jailbreaking still existed and there was still a way in which you could use them to to publically malicious attacks, where would we be? We really wouldn't... We would have no recourse because the same the same the same... These defensive requests are being blocked by the same companies. And if there was no open waste model, we would really have no scalable way to defend our infrastructure as these models become more and more capable, which we're seeing in each iteration. So we we understand the risks. We have a responsible perspective every time that we we open source a model. It's something that we consider very carefully. But we fundamentally believe in the open wage principle, and we think that it applies in this case too as hopefully illustrated by the examples I just gave.

[1:06:00] Nathan Labenz: Then I asked about the approach I'd raised with Prakash. Localize the dangerous know how in a few of a mixture of experts, models, experts, put those behind know your customer checks, and release everything else. I asked whether he saw any promise in it.

[1:06:14] Sagar Vaze: No. Absolutely. I think it's a scientific question just like all the other ones we answer when we're when we're training the model. If one of the features we want in the model is that there's certain capabilities which are tamped down while all of the capabilities still state of the art. That is in itself a scientific problem. And, yes, some of the methods you mentioned are interesting. The... We we look into some some things like that. It's an active research area, and this technology is evolving very, very fast regardless of whether it's a a closed rate model or an open rate model. And we need to be dynamic with respect to with respect to these things. So in the future, if there's a... If there's some other risk, we will we will approach it accordingly. But we consider it a scientific problem to problem solve rather than a fundamental philosophical challenge to the way in which we we operate our, you know, our open weeds business.

[1:07:03] Nathan Labenz: People like me who talk a lot about AI safety these days and are willing to, like, bite a fair amount of bullets to, you know, to try to keep things from going totally haywire on us. A large percentage of us are, like, long time techno optimist libertarians who, you know, have previously said things like information wants to be free and, you know, are, like, very much sort of spiritually on the side of open source, which is why I end up looking for these, like, technological solutions as opposed to, you know, sort of, policy or policing wherever wherever possible.

[1:07:38] Nathan Labenz: Later in the closing

[1:07:40] Anastasios Angelopoulos: conversation,

[1:07:41] Nathan Labenz: after Sagar left, I came back to Mistral and open release.

[1:07:48] Nathan Labenz: Yeah. Well, I I feel like I should say once more that while I do worry about all this getting out of hand, I do, at the same time, very much feel the concentration of power risk factor, and, you know, I I I don't take lightly at all the the potential problems there or the value that people are creating when they do create open source, options for people to use. I really, really hope that we get some of these technical solutions to work well. The question of, like, just how much urgency they really feel about this kind of stuff, I I hope it's high, because I do think we are gonna hit a point pretty soon where it is gonna become untenable to just release these things in, you know, just the sort of throw everything into the mix, pop out whatever pops out, throw that into the public. Jordan Schneider from China talk has has come around a lot on this with respect to China. I don't necessarily know exactly what his, feeling was previously, but recently, I've heard him say over and over again that, like, don't you think that before too long, the Chinese government is just not going to allow this, like, willy nilly open sourcing anymore, and there's gonna have to be some, like, way to make a good case that this is actually gonna be okay? I think that is almost certainly coming and potentially quite soon. But on some timeline, and it's not a super long timeline, we've really got to sort that out. And without it, it seems like we're gonna end up not being able to have open source. I just... I... Governments are gonna have a real hard time tolerating that.

[1:09:33] Nathan Labenz: Prakash's position was that you can't stop anyone from training a model or distributing the weights. So the only place to gate is deployment. He asked how withholding would actually work.

[1:09:44] Nathan Labenz: I haven't seen really anything to date that has shown a unlearning, you know, in a really profound or successful way or that refusals can be made super reliable or that, like, you know, tamper proofing can really work. If you're actually, you know, giving people the weights and they, you know, can obviously modify the weights however they want. So that has not seemed super promising to me, although, you know, I'm always eager to hear about a breakthrough. It feels to me like that is quite different though

[1:10:16] Nathan Labenz: from

[1:10:17] Nathan Labenz: a scheme where the model just doesn't learn certain things in the first place. And I really think we could use a ton more science around that, whether it is localizing knowledge to to particular experts or pretraining filtering. There's, of course, the galaxy brain take that, like, you know, you have this whole world model and then you have, like, one little gap in it, you'll be able to fill in those gaps. I think there's, like, probably some truth to that, but there's some gap clearly that, like, would be big enough that you would not be able to fill in the dangerous parts with the high resolution, you know, that you would need to actually do bad things in the world. And this is the sort of thing that I think that excess compute, you know, could really, be very productively used for at the frontier companies. It would be amazing for them to say, like, here's kind of how big of a knowledge gap you have to leave in a model for that to be, like, meaningfully hard... You know, for the... The model is not just gonna, you know, fill it in via insight. And also, like, you're not just gonna be able to do, like, a couple, you know, marginal documents worth of training and kind of fill in that that gap in the way that we do see refusal being, like, removed at, you know, generally speaking, like, pretty small, like, almost, like, vanishingly small amount of additional training. So I would love to understand that stuff better. Yes. Like, really determined people can set up their own cluster, collect their own biodataset, and, you know, do the additional training on top of the open weight model to create the dangerous thing. And, like, probably at some point that happens, but that's, like, more than just needing a few GPUs. Right? That's more than, like, that's, like, a real effort. So you you would need multiple kinds of expertise. Like, that data is not that accessible. There are these, like, you know, moves to try to rein in that data and not let it, you know, just hang out all over the Internet.

[1:12:16] Nathan Labenz: Prakash had heard a claim that morning that Claude was refusing to help clone video games.

[1:12:23] Prakash Narayanan: I I think I think what I fear is that they're pissing off the gamers. Right?

[1:12:28] Anastasios Angelopoulos: And

[1:12:29] Prakash Narayanan: you say Claude is not gonna help you, like, make these games, you're you're you're you're just asking for trouble. Like, you're asking for the

[1:12:38] Nathan Labenz: most technically

[1:12:39] Prakash Narayanan: competent, like, group, the most technically competent and the people with the most time to burn and it's a passion for them. And video was built on gamers. Right? Like, these guys these guys will... Like, John Carmack was a gamer. Right? You tell John Carmack that he can't do something. Like, you are in trouble. Right?

[1:12:57] Nathan Labenz: I I think that's apt. I mean, you... I think it is definitely the case that, like, we should be pretty cautious around where we deploy these things because we don't want to burn them on stupid things that don't really matter that much. It'll be like... You know, of all the ways to fail in this, it'll be really dumb if it's like, well, we were interested in protecting, you know, Disney IP. And so because of that, we, like, put in these guardrails, but then, you know, the fan fic community was so strong that they came over the top and, like, broke all of our safeguards. And now everything's, like, running wild, and it all was because, like, we were too draconian on things that didn't really matter. I mean, that's a bit of a caricature story, but it's... You know, that's basically the same story as the gaming one you're telling. And I can see that happening in different domains, and it really would be incredibly stupid. So I I do think we need to be, you know, pretty disciplined about, like, just how far we try to create these, like, knowledge gaps or refusals, because everyone does potentially mobilize another community to try to undo them. And I agree too that you're not gonna convince gamers that, like, well, hey. Because you're doing this, you know, you're creating this other downstream problem. Well, good luck having that. You know, you're not gonna win that fight in the Discord. So better to just probably let them have their fun and then, you know, keep our key techniques where we, like, really, really

[1:14:31] Anastasios Angelopoulos: care and really have to have them.

[1:14:35] Nathan Labenz: One more from Friday's closing right after Sagar left. Mistral's technical report and a Claude four system card story I told from memory. The card itself says, Anthropic had mistakenly omitted the only fine tuning dataset that included harmful system prompts and that this was not the sole cause.

[1:14:59] Nathan Labenz: So I'll be very interested to read the more open report on exactly. And, hopefully, you know, we'll see how how open they are about all of the technical details. There have been these, like, footnotes in multiple system card reports over time. One that I really was kind of obsessed with for a minute was when... So Anthropic had forgotten to include a particular dataset called sysprompt harmful, which was meant to teach the model that it should not respond... You know, it should not directly follow a harmful system prompt. So it did follow the directions of a harmful system prompt, And this made it all the way through to the model card. If we don't roll back and, you know, train again, nope. We just kinda figure out a way to patch. I've really been very curious as to, like, how that stuff actually works and, like, how reliably we should think of it as working.

[1:16:03] Nathan Labenz: That's the week. Thanks for listening to the AI and the AM weekly highlights. The narration was my cloned voice. Please tell us what worked and what didn't.

[1:16:11] Anastasios Angelopoulos: See you in the morning.

Outro

[1:19:21] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries either via our website, cognitiverevolution.ai or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts which is now part of a sixteen z where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the cognitive revolution.