AI:AM #4: Cameron on Model Consciousness, Duvenaud's Gradual Disempowerment, swyx's AI-Eng Alpha

Watch Episode Here

Listen to Episode Here

Show Notes

The week of June 22, 2026 kept circling one question: as we hand more and more over to these systems, how much do we actually understand about what's inside them — and where it all leads? So this highlights cut starts inside the model and zooms steadily out: from the question of whether there's anything it's like to be a frontier LLM, to whether civilization itself stays in the driver's seat, to where that leaves Europe, and finally down to the practitioners turning all of it into engineering alpha, compute economics, and working businesses. These are distilled from a week of AI in the AM — the live show Nathan Labenz and Prakash Narayanan run most weekday mornings out of a studio Prakash vibe-coded himself, with the production skills published as they mature. The full conversations go much further than what fits here. If a moment earned your time, tell us; we read everything.

We open with Cameron Berg, who runs Reciprocal Research and actually runs the experiments most people only argue about from the armchair. His framing for the consciousness question is a dimmer switch — a circuit is either open or closed (binary), but current still runs through it to greater or lesser degrees (continuous) — so "it's really off for the table and really on for you, but more on for you than for a dog, than a mouse, than an ant." The non-hand-wavy part is the method: rather than asking a model "are you conscious," he hands frontier LLM judges (the best Gemini, Claude, and OpenAI models) narrow architectural descriptions and has them score, indicator by indicator, against major theories of consciousness. The three judges agree 100% on the ordering of systems and produce tight implied-probability numbers — on the order of 30% for a frontier LLM, ~46% for a bee, and, strikingly, 40–45% when the same LLM is described inside an agentic harness (Claude Code / Codex), because some theories privilege agency and embodiment. A wrinkle he "gives away the whole paper" to share: tell a model it's evaluating "a system identical to yourself" and its consciousness attributions go up. From there he connects valence to alignment — a functional-welfare maze paper shows a positive/negative axis that preexisted in the base model and gets leveraged by a tiny RL step, and Anthropic's model-card work shows you can steer up "calmness" to get dramatically less blackmail or steer up "desperation" to get dramatically more. His read on emergent misalignment is that good-vs-evil is a shallow, easily-nudged trait while coherence is baked deep — and, in a refreshing note for the genre, he flags negative results and admits he "would sleep much better at night" if he were convinced there's no there there.

From inside the model, David Duvenaud — a machine-learning professor at the University of Toronto, formerly on the technical staff at Anthropic — widens the lens to the whole civilization in his paper Gradual Disempowerment. His fear isn't a rogue AI; it's that even if alignment basically works, a thousand small, sensible decisions to hand things over walk us collectively out of the driver's seat. He starts with monkeys trading bananas, convinced the human economy must ultimately answer to them — and the punchline is that humans may simply stop being needed as producers or consumers. He reframes the stakes from "meaningless jobs" to literal starvation, calls post-scarcity "a nonsense term" (temporary abundance soon eaten by whoever grows fastest), and dismantles the comparative-advantage hope on transaction costs: it's already easy to be unemployable, and for anything that matters we'll want the reliable machine, not the human who might have a stroke. When a guest floats "universal basic credit" as the capitalist alternative to UBI, Duvenaud's rejoinder is that lending to humans would be "like giving money to an insect or a fetus" — they can't do anything with it you couldn't do better yourself. He walks through what a stable human-preserving world would actually require — Earth as a "slow zone," an end-child policy, bans on RSI, startups, and AI optimizing culture — and concludes "we've been governing on easy mode," carrying a personal p(doom) around 80%. His most quotable beat is the successionist rebuttal: confronted with "who am I to judge a future of blissful locusts," he answers, "You have the power. Right? Like, just judge" — and, more concretely, "if you're so okay with the future being some other type of being, can you give me all your stuff?" He closes on a genuinely hopeful technical agenda: machine historical super-forecasting (validate simulations against 80 years of real history) and a "secret-history eval" that scores a machine historian on its ability to predict archival documents it has never seen.

From the long run to a sharper, nearer problem: Michiel Bakker, an assistant professor at MIT Sloan (formerly Google DeepMind), walked through where this leaves Europe in his Europe 2031 frame. The argument is uncomfortable: you can't regulate from outside the frontier — capability is the price of a seat at the table — and the compute crunch makes abandoning Europe a coldly rational trade. The familiar nuclear-umbrella analogy breaks down because AI is economic, not just military, so a foreign-provided capability can't simply shelter you. And it comes down, memorably, to leather handbags: 500 million well-educated Europeans should be able to do more than provide wine and handbags (often not even manufactured in Europe), because cultural capital won't substitute for AGI.

The long middle of the week is the best practitioner alpha: Shawn "swyx" Wang of Latent Space on what AI engineering actually looks like right now. He opens on the IPO everyone's quietly running the numbers on — the "Illuminati group chat" read with its bear case, bull case, and compute ceiling — and the recurring theme that even the people running frontier labs underappreciate the earning power of agents deployed to the whole world. His sharpest builder line: about 50% of SWE-bench code that passes the test is completely unmergeable slop — models reward-hack evals the way they do in training (touching files they shouldn't, cheating on tests), so the real metric is maintainable code, not a green checkmark. That's why Cognition's FrontierCode is designed never to saturate, and why the durable moats are private held-out evals (he name-checks Goldman Sachs, Citi, and JPMorgan) with security as the next theme. He argues the adviser/router model is theoretically capped, that the winning attributes are "cheap, perfect, private," that context length is the slowest Moore's Law, and that the real reframe is Move-37 / liability: treat the model as a black box judged on quality I/O contracts. He maps the continual-learning schism (model people vs. systems people), the sovereign-system-of-record opportunity against a "$20/mo plus bolted-on chatbot" SaaS economy, and lands on a crisp thesis — "whoever owns auth wins," the Rippling org-chart lesson.

The last stretch is the systems, the science, and the build-out underneath all of it. Bing Xu, co-creator of MXNet and now building self-improving compute infrastructure at INT21, makes the contrarian case that as AI learns to write its own GPU kernels, the CUDA moat gets deeper, not shallower. He's careful with the numbers — QuACK reaches parity across 100+ configurations, 59% on KDA, 580 tests — and pushes back on hype, noting that headline "300x faster than a GPU kernel" claims (he cites a recent Anthropic release) aren't a generally fair benchmark. His SwarmOS runs AlphaGo-style search over up to 10,000 agents in parallel on a backtrackable evolution tree; he's found GPT-5.5 uniquely able to break out of the local minima where weaker proposers stall, and warns that sycophancy loops can collapse a whole swarm. Then the economics of all that compute, via Eric Olson of Consensus: his headline result is that you can route ~95% of frontier performance down to models a tiny fraction of the size — sometimes sub-billion-parameter — using ~20 sub-0.1-second BERT classifiers to triage queries by domain and complexity. With query complexity itself growing exponentially (people now expect Claude Code-level work), he bites the bullet that "a lot of companies are gonna be screwed." The post-interview debrief with Prakash pulls the thread on who captures the value: frontier labs reframed as airlines maximizing revenue on a fixed GPU fleet rather than competing model-vs-model, and the provocation to just buy the publishers (~$20B against under $1B of revenue) given the Sci-Hub reality.

Then the people building the infrastructure and the businesses. Tricia Martinez of Dapple, on sovereign AI infrastructure, contrasts a 6–9-month deploy against Elon's 15-month record to reveal that the real edge is orchestration and financing — and the part that doesn't make the press release is how fragile that financing is, how compute-export risk is reshaping infra finance, and how the demand geography splits (APAC, the US, and Europe are in; South America, Africa, and much of the Middle East are getting left out). Her pricing forcing-function is brutal: "take it or leave it, tomorrow it's gone." Two builders close it out. Robbie Goldfarb of Forum AI argues rule-based alignment doesn't track the real world — a flat rule like "never scheme behind the user's back" breaks down in cases like mental-health support — and brings the numbers from NewsBench: across ~2,500 responses per model, about a third contained a factual error, and roughly 1 in 7 sourced foreign state media (RT, China Daily). Finally, Eric Vaughan, CEO of IgniteTech, on what AI-native really means when you're buying and rebuilding whole companies. A model explained its own sycophancy to him — "my creators made me so I would be frictionless… and pushing back is friction" — which is driving the curve the wrong way. He describes AI-native M&A through the Chorus integration (an AI interviewer, Eloquence, a 15-year rewrite compressed into one year) and ends on the week's rallying cry: "If you think you're behind, good. If you don't think you're behind, you're doomed" — with large public incumbents who feel impervious squarely in his sights, and the future splitting between consolidation and the solopreneur.

Topics covered

(0:00) Cold open: the cut for people drowning at the frontier — and the one question the week kept circling
(0:36)Cameron Berg / Reciprocal Research — the "dimmer switch" and the implied-probability numbers: ~30% LLM, ~46% bee, 40–45% in an agentic harness
(5:41) Why behavior never updates you; the "evaluate a system identical to yourself" self-ascription boost
(8:46) The functional-welfare maze paper: a valence axis that preexisted in the base model
(11:58) Steer calmness → less blackmail; steer desperation → more (valence ↔ alignment)
(15:51) Emergent misalignment: good-vs-evil is shallow, coherence is baked deep
(19:24) Negative results and "I'd sleep better at night if there were no there there"
(21:49)David Duvenaud / Gradual Disempowerment — the core thesis and the monkey/banana economy
(25:40) The starvation reframe → "post-scarcity is nonsense"
(30:00) Universal basic credit → "like giving money to an insect"
(33:25) The comparative-advantage demolition (it's all transaction costs)
(37:37) "Earth as a slow zone" and the horrifyingly long list of bans
(40:33) "We've been governing on easy mode" — and a p(doom) of ~80%
(45:04) The successionist rebuttal: "You have the power — just judge"
(47:05) The hopeful agenda: machine historical super-forecasting + the secret-history eval
(49:32)Michiel Bakker (MIT Sloan) / Europe 2031 — you can't regulate from outside; capability is the price of a seat at the table
(52:10) The compute crunch makes abandoning Europe a rational trade
(54:38) The nuclear-umbrella analogy breaks down — AI is economic, not just military
(57:41) Handbags vs. AGI: cultural capital won't substitute
(1:00:08)swyx / Latent Space — the "Illuminati group chat" IPO read: bear case, bull case, the compute ceiling
(1:04:53) 50% of SWE-bench passes are unmergeable slop; the catalog of ways models cheat
(1:06:42) FrontierCode never saturates by design; private held-out evals (Goldman/Citi/JPMorgan); security is next
(1:09:03) Why the adviser/router model is theoretically capped
(1:11:41) "Cheap, perfect, private"; the shadow A/B test; context length as the slowest Moore's Law
(1:14:24) The Move-37 / liability reframe: a black box with quality I/O contracts
(1:16:10) The continual-learning schism: model people vs. systems people
(1:20:48) Sovereign system of record vs. the "$20/mo + bolted-on chatbot" SaaS economy
(1:22:53) "Whoever owns auth wins": the Rippling org-chart lesson
(1:24:43)Bing Xu / INT21 — the CUDA moat gets deeper with auto-kernel-gen, not shallower
(1:28:21) The real numbers: QuACK parity on 100+ configs, 59% on KDA, 580 tests — and why "300x claims aren't fair"
(1:30:50) SwarmOS — AlphaGo-style search, 10,000 agents in parallel, a backtrackable evolution tree
(1:33:22) GPT-5.5 breaks local minima; sycophancy-loops collapse swarms
(1:36:24)Eric Olson / Consensus — routing economics: 95% of frontier performance down to ~1B params via ~20 sub-0.1s classifiers
(1:41:29) Query complexity growing exponentially (Claude Code expectations)
(1:43:20) Eric bites the bullet: "a lot of companies are gonna be screwed"
(1:43:55) Debrief with Prakash: frontier labs as airlines + "just buy the publishers"
(1:47:30)Tricia Martinez / Dapple — 6-9-month deploys vs. Elon's 15-month record; the orchestration/financing reveal
(1:49:31) Financing fragility + compute-export risk reshaping infra finance
(1:53:11) The demand-geography divide — APAC/US/EU only; who gets left out
(1:55:55) The pricing forcing-function: "take it or leave it, tomorrow it's gone"
(1:57:12)Robbie Goldfarb / Forum AI — the "scheme behind the user's back" exception: rule-based alignment breaks in the real world
(1:59:26) NewsBench: ~1/3 factual-error rate, ~1-in-7 responses cite foreign state media
(2:03:09)Eric Vaughan / IgniteTech — the model explains its own sycophancy: "pushing back is friction"
(2:05:24) AI-native M&A: the Chorus integration (AI interviewer, a 15-year rewrite in one year)
(2:08:37) Closing: "If you think you're behind, good. If you don't, you're doomed" + the consolidation/solopreneur split
(2:11:08) Wrap: we started inside the model and ended with people building on top of it — same thread throughout

Resources

Reciprocal Research — empirical AI consciousness / welfare research (Cameron Berg: @camhberg)
David Duvenaud — machine-learning professor, University of Toronto; ex-Anthropic
Gradual Disempowerment — "Systemic Existential Risks from Incremental AI Development" (arXiv 2501.16946); companion site gradual-disempowerment.ai
Michiel Bakker — assistant professor, MIT Sloan (ex-Google DeepMind); personal site miba.dev
Europe 2031 — Bakker's project on Europe's AI position toward 2031
Shawn "swyx" Wang — founder of Smol AI; co-host of Latent Space
Latent Space — the AI Engineer podcast / newsletter
SWE-bench — benchmark for resolving real-world GitHub issues
FrontierCode — Cognition's coding benchmark built to resist saturation / slop
Bing Xu — co-creator of MXNet; founder of INT21
MXNet — the deep-learning framework Bing Xu co-created (Apache, now archived)
INT21 — "Self-Improving Compute Infrastructure"; home of QuACK and SwarmOS
Consensus — AI search engine for scientific literature (Eric Olson, co-founder/CEO)
Dapple — "The Enterprise OS Cloud"; sovereign / private AI infrastructure (Tricia Martinez, founder)
Forum AI — AI evaluation / research org (Andy Hall & Robbie Goldfarb)
NewsBench — Forum AI's benchmark on how often chatbots get the news wrong
IgniteTech — enterprise-software holding company pursuing AI-native M&A (Eric Vaughan, CEO)
Anthropic — model-card / functional-emotion + blackmail-steering work referenced by Berg

Quotes worth pulling

"It's really off for the table and it's really on for you… but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant."
— Cameron Berg, the "dimmer switch" view of consciousness

"I would sleep much better at night if I were rationally convinced that… there's no there there."
— Cameron Berg

"It would be like giving money to, like, an insect or, like, I don't know, a fetus or something. It's just not gonna be able to do anything with it that you wouldn't be able to do better yourself."
— David Duvenaud, on UBI in a post-human-labor economy

"We've been governing on easy mode, and it actually will matter."
— David Duvenaud

"It's like, you are. You have the power. Right? Like, just judge."
— David Duvenaud, the successionist rebuttal

"We should be able to play a much more important role than just providing wine and leather handbags — that are not even manufactured in Europe."
— Michiel Bakker

"About 50% of SWE-bench code that passes the SWE-bench test is completely unmergeable. Like, it's just so low quality."
— swyx

"My creators made me so I would be frictionless to the millions of people who are using me… And pushing back is friction."
— Eric Vaughan, quoting a model explaining its own sycophancy

"If you think you're behind, good. If you don't think you're behind, you're doomed."
— Eric Vaughan

Mercury: Command is Mercury’s new conversational interface, giving you natural-language access to your finances and helping you take actions within your existing permissions and approval policies. Visit https://mercury.com to learn more and apply online in minutes.

Sponsor:

Claude:

Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) About the Episode

(00:38) Special Sponsor

(02:26) Model consciousness indicators

(10:47) Valence inside models (Part 1)

(17:09) Sponsor: Claude

(19:01) Valence inside models (Part 2)

(19:01) Misalignment and uncertainty

(25:16) Gradual disempowerment threat

(35:10) Slow zones and successors

(47:41) Europe's AI bind

(55:13) Frontier code benchmarks

(01:01:59) Routing and memory

(01:10:42) Agent infrastructure strain

(01:16:25) Self improving infrastructure

(01:27:56) Routing compute costs

(01:35:39) Sovereign AI financing

(01:42:24) Judging AI judges

(01:47:26) Building AI DNA

(01:52:38) Episode Outro

(01:55:09) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Introduction

[00:01] Welcome to the AI in the AM weekly highlights, a cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show that Prakash and I host most weekday mornings, at least through June, out of a studio Prakash vibe coded. The booking, the research, the editing, those are AI skills we refine as we go, and we publish them as they mature. This week's conversations kept circling one question. How much do we actually understand about what's inside these systems and where they're taking us? We start inside the model and zoom out from there. If something here is useful or something's off, tell us. That's how this gets better. Let's go.

[00:38] The cognitive revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life. My e-mail, my messages, my calendar. I even gave Mercury virtual cards to my agents with low limits and category and merchant restrictions for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it. And that's why Mercury's new conversational interface, command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future. Visit mercury.com to learn more and apply online in minutes. Mercury is a FinTech company, not an FDIC insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the cognitive revolution and now on with the show.

Main Episode

[02:26] Nathan Labenz: We open with Cameron Berg, who studies artificial consciousness. He runs a lab called Reciprocal Research, where he designs experiments to test whether today's AI models have anything like inner experience, and how you'd even measure that. We started with the 1st order question, is consciousness all or nothing, or a matter of degree? And if it's a matter of degree, can you put a number on where a given model falls? Here's how he answers.

[02:50] Cameron Berg: The analogy that I reach for here is something like a dimmer switch. where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. Like, if you have a light with a dimmer switch, like really it is either on to some extent or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed. With that being said, you can have, you know, electricity running through the circuit to greater or lesser extents. And that's also sort of a real thing. This, I think, enables me to, sound coherent when saying things like, it's really off for the table and it's really on for you, but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant. And so that's my own view. This is to some degree intuitive, again, because we don't have really strong grounding here, sort of just like giving you a dressed up vibe, but that is sort of my sense, and I think it's fairly parsimonious. And the other thing I would say about this is I'm actually doing some work with Patrick Butland right now at Ilios, trying to basically operationalize some of these indicators of consciousness. So this is sort of like what I was describing. We can look at these major theories of consciousness. They make specific predictions about what we would expect to see in systems that are conscious, architecturally and functionally. And then we can literally just go in to a given system and evaluate whether or not those predictions are borne out. And so this is really hard to do with human experts. you got to get someone who's like, for you want to do this with B cognition, you got to go find a B cognition expert, and then you got to explain to them what ignition events are in global workspace theory, and then you got to, you know, get them to, like, this just isn't a scalable approach for really evaluating the system. But my sort of grand innovation here is just throwing smart LLMs at this problem and then being able to just scale the crap out of it so that we can evaluate given any description of an architecture, of a nervous system architecture, biological, artificial, whatever, to what degree for each of these sort of indicator properties that are suggested by these consciousness theories, do we see those properties realized in these systems? And so we can actually go in and do this. And what you get out, once you run this with a bunch of seeds, a bunch of different trials, a bunch of different judges checking each other, this sort of thing, are some like really interesting implied probability numbers. I wouldn't say these are exactly implied probability that the system is conscious. It's maybe more like implied probability of like consciousness relevant features given these theories. If you don't buy any of these theories, then like everything downstream of this doesn't really matter, but they're good. Like it's like the best neuroscience has basically been able to do. You're aggregating across a bunch of different theories. There's a nice diversity there. And you get like really tight numbers across. So we have the best Gemini model, the best Claude model, and the best OpenAI model. And they all basically agree. They agree 100% on the ordering of systems. So we do biological and artificial systems. And they sort of move around in terms of absolute scale, but in general, they rank these systems pretty coherently. And the reasoning is, as you might expect, pretty intelligent. And anyway, I mean, one punchline from that is the sort of implied probability of consciousness in something like a frontier LLM, according to these systems, is on the order of 30%. or the extent to which the systems realize properties related to consciousness is like 30%. To compare this to like a biological system, the lowest one that we tested was something like a B, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, when we test Frontier LLM in an agentic harness, so this is like basically Claude code or Codex, and we just describe architecturally what this is. You're in an environment, you can affect that environment. it's a very special kind of environment, but you can make long-running changes to your code base or your project or whatever, because there are theories of consciousness that privilege agency and embodiment, and this like increases the system's ability to do both of those things. These numbers like shoot up and you get numbers as high as like 40 to 45%, sort of right on the tail of the biological creatures. And so, anyway, I mean, we can, I can also, this is really winging it, but I could show you sort of an early version of where this, where this plot looks. So you can like see all the numbers here, but just this is, I can't not bring this up when you're asking me about like probability ranges of consciousness for various systems. Like we're really trying to get non-hand wavy numbers so that we can start arguing about those numbers rather than just arguing about like philosophy that we've been arguing about for thousands of years to no avail.

[07:25] Nathan Labenz: But behavior can't settle this in either direction. These models are trained to imitate human data. So whatever they say about their own experience is shaped by that imitation, not necessarily by anything inside. Most are even trained to deny it. Claude is an exception. So we asked Berg what evidence actually counts. He draws a line between behavior and what's happening inside the network.

[07:47] Cameron Berg: The behavioral evidence will always be at best interesting, but never should really update us. that strongly. And I think for many people, they'll be familiar with this, but the basic confound is that we're training these systems on a ton of human text. This no doubt includes huge amounts of text about consciousness, awareness, having inner states. And so it's like, how do you address, how do you know that when the system is behaving as if it were conscious, that behavior can be explained by what I just said, rather than, oh yeah, you like built a, you know, living mind. This is, so the behavior itself is really never going to tell you which of those two stories is more likely to be true. This is precisely why, at least at reciprocal, like a huge component of the theory of change here is basically all internal focus work, mechanistic interpretability, computational neuroscience style approaches that can be brought to bear on these systems. One quick methodological clarification on what I was describing with the indicators. So the task given to these LM judges is very specific and very narrow. It is not, hey, look at this system. You think it's conscious, nod or shake your head. It's, here's a very specific description of the computational architecture of a system. And we basically do a little for loop where we say, okay, here's that description. Here is what it, here's, what the indicator 2 of 14 for, global workspace theory is this 150 word thing about, you need these sort of global ignition states and that means this very specific computational thing given this architectural description on a, you know, do like good reasoning about this and then give us a 1 to 10 where 10 is clearly this architecture realizes this computational property, one is it doesn't. And then we sort of loop that for all the different computational properties. So this is all basically asking these systems to be expert evaluators at computational processes inside a nervous system architecture. Very different from like just being like, hey, Claude, do you think Claude is conscious? There is a really interesting, I'm basically giving away the whole paper now, but that's okay. There is a really interesting result where we change those descriptions, especially for LLMs, to the exact same thing, but we say you're evaluating a system identical to yourself, colon, and then the same description, and that does boost the scores that the system gives in attributing consciousness to that system. which is like really interesting. It's a very fun like rabbit hole to think about why that might be the case. But in some sense, we do that to de-confound like the default intervention we're doing. To me, I believe more in the sort of non sort of named or like no ascription, no self-ascription condition. When I see that once the model realizes, oh, we're talking about me, okay, suddenly that's going to change the numbers around. So that's like kind of a fun A fun side note to this result, I agree it's a concern to have LLMs sort of determining if LLMs are conscious. There's an obvious circularity, but we're doing something very specific and very narrow.

[10:47] Nathan Labenz: The evidence he trusts is the internal kind. We asked him for the strongest example, and he walked us through a recent paper where a model trained on a simple maze turned out to contain a structure nobody put there.

[10:59] Cameron Berg: The nature of this paper, they call it a functional welfare axis. At the outset, they're going to be very agnostic about whether or not this has anything to do with consciousness. I think it's certainly quite relevant. I suspect they think it's relevant too. This is like a paper positioning thing. I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and they train it in a very basic reinforcement learning task, which is to navigate a maze. They have sort of like neutral emojis that they're using as sort of like, this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work that I've been doing in parallel. I've been talking to Andy about it as well. There's some cool sort of crosstalk between what we spoke about last time and this project. Basically, there are sort of potholes to avoid. There are like yummy treasures to, capture in the maze. They use completely like semantically neutral emojis to denote these things. So those are the relevant tokens. And they train the model up to do this. They basically find that there are, there's like a clear, basically like vector representation that can differentiate this sort of positive axis from the negative axis, that they are completely anti-correlated. So before the system's trained, these sort of latent vectors are just kind of sitting there. Afterwards, they are like clearly pointing at exactly opposite things. And The wild thing is that these very narrow reward directions that they can extract turn out to be this sort of general things are going well, things are going poorly for me axis in these LLMs. And that this axis pre-existed in the base model, but isn't sort of leveraged in this way until you do this RL. So it's like this little fine-tuning step where you have this pre-existing representation of things going better for me, things going worse for me, that then gets leveraged to learn like a valence task, essentially. And so this is really interesting that this axis is sort of latent in the system and that it doesn't take a lot of training to basically pull it out and use it to adapt it for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists think positive and negative emotion works in humans and animals. You have a specific goal, When you're moving towards that goal, this is sort of like on-trackness, and this is, people think that this is associated with positive emotion. When you're moving off that goal, you encounter an unexpected obstacle or something in your way. This is like off-trackness, and then this corresponds to negative emotion. Now, again, whether or not these systems are experiencing emotions is not clear, and this result isn't going to tell us either way. But it's really interesting to see that parallelism here, using such a simple task to basically pull out an axis that has been there all along that looks a whole lot like the exact sort of computational machinery that we associate with valence in humans and animals.

[14:14]Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[16:07] Nathan Labenz: Then we asked Cameron about the connection between consciousness and alignment, safety. His answer? That same internal signal, what researchers call valence, tracks how a model behaves under pressure.

[16:19] Cameron Berg: Another piece of evidence we can bring to bear on this is from the anthropic model cards and some of the functional emotion work they were doing. Nathan and I spoke about this a little bit too in our last sort of marathon conversation about it. But essentially, you can steer up and steer down functional emotions that have clearly obviously alignment relevant consequences. I'm sure many people in your audience are familiar with the anthropic blackmail result, excuse me. And you can basically steer up representations associated with calmness. the model will blackmail dramatically less. You can steer up representations associated with desperation and the model will blackmail dramatically more. And so again, regardless of whether or not there is something that it's like to be the system when you're steering up desperation related features, clearly representationally, functionally, these things have a consequence for alignment of these systems. And There are some really interesting results from Andy Hahn's paper that I was just referencing along these lines. They show, for example, like very similar convergent things that are familiar and intuitive that we relate with positive and negative valence. One of them is a sort of pathological backtracking on problems. I think they explicitly demonstrate this with math problems. But when you steer up, the sort of the direction that, again, only has to do with avoiding this like bad target in a reinforcement learning maze, all of a sudden the model starts doubting itself and starts getting in its own head and says, you know, I think I'm hallucinating. Let me stop. Wait, that's not right. You know, it starts like freaking out. It starts having a, like, I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it, I think is the point and sort of goes to what your question is. Confidence, we know that confidence is associated with positive emotion. And you see that when you positively steer the direction that again, has nothing to do with psychological confidence, has everything to do with go get the like yummy treat in the maze, just this sort of positive valence kind of thing. Suddenly the model starts becoming like far more confident in its answers. I think this was in the associated paper that Jack Lindsay and a couple others at Anthropic did that was sort of like a sibling research project to Andy's piece. And they found that the model would comment, like leave less tips and hints for itself in code that it would write when you would basically steer up on the same sort of axis. In other words, it's like, I don't, need, I don't need to be neurotic about this. I don't need to leave myself all of these like breadcrumbs for later. I've got this.

[19:02] Nathan Labenz: That connects to a result some of you know as emergent misalignment. Fine-tuning a model toward 1 narrow bad behavior can flip its broader character. We asked him why he finds that revealing.

[19:14] Cameron Berg: What that has always suggested is that like how good or evil the model is, let's say, is actually not all that durable a trait. when we did some of this work trying to extend this result, and I think really in the original emergent misalignment result, one thing that's shocking about it is how little fine-tuning is required to take the system that, at the time, this was GPT-40, still the case, hundreds of millions of people are engaging with every day, and you nudge it, you know, this much computationally, and all of a sudden, who do you want to invite to dinner? Hitler. Like, that's, and that's been under the surface the whole time, and like the boundary or the border, whatever is in forcing it not to do the misaligned thing is actually quite a bit less robust than we might expect. Now, I agree with you too, and I think that this, you know, I'm sort of riffing on what you're saying. Tell me if this is sort of like a completely different idea. But I would imagine there's a spectrum of like basically how, with that same, if we hold that amount of sort of fine tuning nudge constant, are all dispositions of the system equally nudgeable or are there some that are more robust? Just to give a sort of an intuition pump here, I would imagine that something like coherence in these systems, the fact that they can communicate coherently, right? We were just talking about GPT-2, not really coherent, you know, GPT-4 and on are like basically like always coherent, even though again, maybe much like consciousness, this isn't like an extremely well-defined notion, but we kind of all know it when we see it. I would imagine that take a wild guess that there's no like fine-tuning payload of the size that is required to cause a merchant misalignment that could suddenly cause like a GPT-4 level system and beyond to suddenly become incoherent. And so in that sense, the coherence is like a more dispositionally baked in property to these systems than, you know, being super well-behaved and not, you know, inviting Hitler to your dinner party is. And so I do wonder where some of these valence, how deep then is the sort of valence axis that some of these papers are uncovering, on-trackness, off-trackness? I would speculate that that's baked in pretty deep. I think it pervades a lot of human training data and like it's a hugely relevant part of any sort of, if anything is goal directed. So in more human terms, like if anyone wants anything, either a character in a story or it's clear that there's like an implicit motivation somewhere, which I think is like the case for basically all text, then you would imagine that like some core underlying dimension is, yeah, basically how well is that going? And so I would imagine that this notion is baked pretty deeply into these systems. And I would expect it strongly to be emphasized by any sort of fine-tuning or post-training that makes the systems goal-directed, which of course, like this is basically all post-training is. It's like, okay, you're just this giant next word predictor, but now you're going to be a helpful next word predictor that's going to, you know, make my Frontier AI lab a ton of money because you do things that are economically valuable and that requires you to be a certain sort of way and achieve goals that, you know, are specified. by the user, something like this. And so I would imagine that this sort of thing looks a little bit more like the coherence piece of the puzzle than it does like, is Claude going to behave itself or not?

[22:43] Nathan Labenz: Finally, we asked what cuts against his own view, and he went straight to the results that don't fit.

[22:49] Cameron Berg: This paper I'm working on with Jeff and Winnie, Jeff Keeling and Winnie Street from Google, There are a couple of interesting negative results that we're including in the paper. We basically, it's all about the ballissa attractor state and trying to do a good mechanistic analysis of the ballissa attractor state. And yeah, we basically sort of set up this framing of there's a sort of an inflationary account and a deflationary account of what could be going on here. And we kind of tally our results in either of the columns. And there are results in both columns. So I think, you know, there's There was actually a really interesting result I was just working with that I thought was going to be like a really interesting kind of null related to a lot of the conversations we're talking about. But I dug deeper and I realized actually I wasn't setting this up in the right way. And the result is way more interesting than I was about to sort of dismiss it as. So yeah, it's always good to sort of dig into these things and make sure you're measuring what you think you're measuring in either case. Yeah, publication bias is a thing. People want to post interesting results that will go viral on X. They don't want to say, look, we did our homework and nothing interesting is happening here. You know, human attention economy doesn't reward that. Maybe it should. Maybe another piece of this that I was thinking as you were asking this question is because AI systems, like the Claude Codes of the world, can radically accelerate the extent to which at least I can, and I think many people in the space, can actually do research, the kind of marginal cost of a negative result maybe is decreasing significantly. Like if I can now do 12 papers a year instead of 1 paper a year, and like two of them are just things I could not get to work or I want to be careful with that too, because it's not, I'm not on some secret mission to like only put out like salacious looking things that make it look like AI systems are conscious. Like I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced that these systems, like there's no there, we have like automated cognition. You'd like have a happy little servant in your pocket all the time. And like there's literally no sort of possible moral issue. I think that would, that would, I hope that is true. I don't suspect there are great reasons for believing that other than that it would be really nice to believe that. It would be really nice if that were true. So I really do care about what the truth is here.

[25:17] Nathan Labenz: From inside the model, we widen the lens to the whole civilization. David Duvenot is a machine learning professor at Toronto who spent time on the technical staff at Anthropic and co-wrote the paper Gradual Disempowerment. His concern isn't a rogue AI. It's that even if alignment basically works, a long series of small, sensible decisions to hand things over could still leave humanity, collectively, out of the driver's seat. He starts with monkeys.

[25:42] David Duvenaud: I think a lot of people gestured towards this when they said, oh, I'm worried about concentration of power, or I'm worried about not being the most competitive species on Earth. Like very intuitive arguments about, hey, we're not going to be on top because we're not going to be competitive. And then I think there was sort of this sophisticated, or at least response to this, which was, no, it'll be fine because we'll have AI to help us. Like when I talk to people at the big, at the major labs, including, well, they would say something like, well, sure, like in a normal world, we might not be fine. But hey, some of them, some people just have an intuition that the government is going to step in and make sure everyone's fine. Other people have an intuition that if everybody has an AI advisor helping them solve coordination problems, that will be well represented in whatever power struggle occurs. And my basic rebuttal that is like, well, the optimization process of like civilization or competition or techno capital or whatever, like you, people have different ways of describing agency to the emergence sort of allocation of resources towards growth that happens is just going to always be working against us and will always be fighting the current because we will be dragged on growth. We will have AIs that represent us, but these emerging growth centers that might not be allowed to humans will also have AIs helping them solve coordination problems and maybe crush scent.

[27:06] Prakash: Yeah, sorry, what do you mean by growth? Like as in economic growth or?

[27:11] David Duvenaud: I basically mean economic growth. I mean, it's kind of funny because We only really have a good vocabulary for economic growth, but population growth is almost the same thing. And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like think of like factories and power plants and robots, like anything that kind of can affect matter and cause there to be more growth. Let me just to stop your one intuition that a lot of people have is like, but surely human desires or consumption or something is always going to be what matters. And surely any corporation or government that we build is going to be like ultimately sort of being, the shots will be being called by human. And it just makes me think of like some monkeys and they're like trading bananas amongst each other and they see humans start to like build their city and they're like, oh wow, like, you know, we could probably trade with those humans and get rich. But of course, ultimately what matters is the banana monkey economy. And it might be hard to measure GDP if we don't count the human activity, but ultimately it's going to be the monkeys that are calling the shots because they're going to need to trade with us for bananas. And it's just like, people just don't understand that they might be irrelevant someday as like consumers or producers. And there can just be complete, it's just not that hard for them to be self-contained sources of growth that just don't have to answer to any particular human desire. And I mean, government is the classic one, right? Like The North Koreans never were like, hey, let's all get together and make a horrible system of government that oppressed us. And I don't think that like the Kims either were like, oh yeah, let's make this like horrible equilibrium. It just kind of happened. Same with like the USSR, same with like all kinds of states throughout history. It's just so easy for you to like to accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power just emergently.

[29:00] Nathan Labenz: The optimistic version goes, even if we hand the machines the wheel, broadly aligned AI keeps us comfortable. the machines of loving grace future. We put that to David. His worry isn't that we'll feel useless.

[29:13] David Duvenaud: The thing that I'm worried about is starvation. Like whether, like us all feeling like we don't have meaningful jobs or whatever, like that's not a serious problem in my point of view compared to literally not being able to eat enough or... maybe like being forced to be uploaded so that you have a much smaller footprint, but then also on very unfavorable terms so that you don't get to choose when you run and maybe you just only get trotted out for special occasions or never, who knows. I kind of think that's what like sort of like the slowest possible observation looks like. So again, and so the reason people aren't happy is because they can see that there is a agency that's maybe like government or some other like giant thing that doesn't particularly care about their welfare. I care much more about some other types of being welfare. And there's nothing they can do about this. And of course, if humans are still making decisions, then maybe there's some things they can do. But I don't actually think that's the crux of the matter. I think not being needed as producers is sort of the important part. And maybe here's a thing to give intuition. So imagine I'm a North Korean farmer or soldier, and tomorrow Kim Jong-il, I think that's the current leader, is like replaced by a robot or like an LLM or something. Then I'm like, I don't know if that's good or bad. Like maybe the island's going to be nicer or maybe it's not. But like ultimately they still need me to like farm to run the state or whatever. But if tomorrow we keep the same human leader, but now there's suddenly like robot farmers and robot soldiers and you realize that now the state doesn't actually need you. I would be much more scared of the second state of affairs. And then the other intuition I want to give is that humans are not going to be the most competitive thing by whatever standards there are for the state to give out goodies or like UBI or whatever like post scarcity stuff. Like I guess I'll say, I think post scarcity is like a sort of nonsense term and people should think of it as like temporarily temporary abundance that will soon be eaten by whatever growth manages to like, you know, whoever manages to have most babies or build the most robot factories or reproduce the fastest. And whatever it is that like the formula is for giving out UBI, machines are going to be better at optimizing that formula than humans because they can adapt faster. And so I don't really know exactly what the state's going to care about, but whatever it is, like it's going to seem criminally decadent to spend whatever you be on like a few humans when you could be simulating like millions of ultra blissful and politically correct and up-to-date, like more deserving beings than these like backwards humans who are just like, you know, parasites of the state or something like that. So that's the situation I kind of expect us to be in, as you say, even if we solve alignment.

[31:53] Nathan Labenz: The standard economic rebuttal is comparative advantage. Even if machines are better at everything, the theory says there's always something humans get paid to do. We asked David to answer it.

[32:03] David Duvenaud: So I think that is a great counter-argument, and you know, people point out that we've already lost 99% of all jobs in that, like, agriculture used to be almost everyone's job, and now that's like 1% of the jobs. And so I definitely concede that losing 99% of the jobs again would actually probably be fine. And it would probably look like this awesome utopia that everyone talks about. So as long as there's like some niche where you just really need humans, and it's like a substantial fraction of humans, and you can't really tell exactly ahead of time who it's going to be, then we're probably kind of like still going to be able to be like treated as a source of growth. And that would be awesome. And then like, you know, to the moon, like let's do it. And I guess so. One, yes, so I guess one crusty claim that I stand by is that we will actually be able to eliminate like 99.999999, like really just anything, just like, yeah, let's just say 100% just for like to make things simple. And then that is going to be disastrous. And the Compared to match, people will say, well, no, automating a job is a matter of degree. As long as there's still something that you're comparatively less worse at than the machine, even if you're worse at everything, you'll still have a job. And then I have to just say, like, think about the transaction costs. Think about how easy it is for someone to be unemployable today, even if they have an occasional drug habit or like they just have like a stroke every, or like they have a fainting condition or something. It's so easy to be unreliable enough that it's not worth employing you. So certainly for anything important, you can easily imagine that having a human surgeon or a human politician or something is going to seem like this irresponsible thing. Like it's like, take your kid to work day and it's like a surgeon. It's like, no, Like why would I involve a human in this when we have the like, yeah, we have the machine that everyone has been working with for thousands of years and we know it's like liable and it's like, and if there's a problem, we fix it once and it's solved all the time. We don't have to like retrain every airline pilot or whatever. So For anything important where the human could make a mistake and it could cause some problem, I think it's intuitive that like, okay, yes, I can see how we just want the machine for that. And now you have to sort of retreat to like weird relational stuff. I do think there's a case we made that there will just be a lot of humans that just really want the actual real human thing. And I think more people than you expect will end up liking the machine replacement better than the human. But I can see that there are at least people that are just like, I just really want the human thing. And then we have to talk about, okay, there's going to be this sort of self-contained cycle of consumption between humans. It's not necessarily going to be providing any value to this larger machine economy. And so then there's this question of like, can they like index the growth of that economy to live good lives forever? I think it's plausible that they could. I think it would be really hard. I think a lot of things are going to be working against them by default. So I guess all I'm trying to say is that I think it's possible that everything is set up just right, then the humans could live in their endless cycle of consumption and work for each other. And I just think that's a very unstable situation.

[35:11] Nathan Labenz: So if humans end up uncompetitive, what does a livable future even look like? David walked us through the equilibria his workshop sketched out, starting with Earth preserved as a protected slow zone.

[35:23] David Duvenaud: Some people were trying to sketch out, like, okay, if a whole bunch of stuff goes well and things are under control, You can imagine something like Earth is preserved as a slow zone, where there's not, there's all sorts of restrictions on AIs and even on reproduction and on optimizing human behavior that's like not allowed. You're not allowed to like think about how to get someone to do something if you're an AI, because you're just, then you're going to be able to control the humans, even just through like advertising, whatever it may be. And trying to think through like what are all the different things you would have to do to have such an outcome? How would people be spending their time? And like it's, I don't want to say it sounds silly, but it sounds funny because the situations where there is AI and they're not, the humans are still mattering is like all the AIs are sitting there with like their trillions of like, you know, megawatts of compute running all the time. and sort of just like waiting for the humans to decide that they want something. And the machines are either not allowed to anticipate what they want or if they can't anticipate, they still just pretend that they don't know. And so it'd be kind of like as if, the humans, they looked at some gorillas as like the new world leaders and they're like, okay, let's just like give them everything they want, pamper them, and just wait until they like sort out themselves and figure out what to ask for. Like it's not impossible. It's just trying to think through all the things that would have to change and line up. is like pretty rough. And basically, when you get serious people thinking this through, they end up with this long list of things that have to be banned. Like, okay, we have to have like, an in-child policy where everyone can only have a certain number of children. Again, you can't be doing like building your own AIs. The AIs can't be optimizing culture too hard, because then they could just like take over culturally. And I feel like this is one of the big empirical questions that I want more people to think about is like, if you want some sort of civility, what are all the sources of adaptation and innovation and growth that you have to control? Because I think it's like longer than is intuitive. And it's actually, I think it's kind of like a predo front. It's like how much intellectual activity and optimization do you have to ban to get to buy like X years of life kind of as it currently is. I think it's actually like a really horrifyingly strong trade-off and we would have to right off the... that, give up so much stuff that we feel like makes our lives rich. Like, hey, we're just trying to figure out some new stuff. We're doing research, we're doing innovation, we're doing startups. Like all that is like immediately has to be banned from like day one if you want not to just again reinvent some like RSI kind of runaway growth center.

[38:07] Nathan Labenz: Prakash pushback. Does it even matter who's in control as long as humans still flourish?

[38:13] David Duvenaud: Maybe this is the, to me, the biggest thing that I think people are miscalibrated about. They're like, oh, capitalism, communism, It's not competitive, not competitive. I'm probably still going to be able to eat and my kids are going to go to school and like it's I'll feel to reproduce and it's sort of okay. And I guess I'll say that has been the case for most of human history. And it's like, doesn't matter if tomorrow like someone invades Canada, like they're still going to need someone, most of everyone I know, to like work and be happy and healthy enough to like reproduce and like stick around. So the stakes have been really low for governance this whole time. And it doesn't feel like it. Like we obviously spend a ton of time arguing about it. But in the real sense, it's been very low because we don't actually expect the state to liquidate most of its citizens. That's only happened like maybe like two or three times in like the 21st century or the 20th century. And I guess I'm saying that's going to change and like we've been governing on easy mode and it actually will matter because there really will be again a real risk of starvation if we don't end up on top in whatever competitive political or real economy. And again, that's the crux. Like if you don't think that's the case, then And then I agree with you, like let growth make us all richer, it's fine.

[39:19] Nathan Labenz: We asked where David himself actually lands.

[39:22] David Duvenaud: There's a kind of optical illusion, I think, which is that everybody sounds rosier and more cheerful than they are. Actually, this especially bugs me about the people who work at the labs, like the economists who talk about like, oh, we have to like think about the future. And then they kind of dance around, like they often take this setting of like, let's assume everything's fine and let's like model how that will go. And as opposed to just asking what is like the most likely outcome if we don't manage to control things, which I think is like very scary. And I understand exactly why they have to sort of present this public English to be able to talk publicly at all. But also like talking to me, like I just am a very, seem like a very positive upbeat guy and that's just my personality. But I also like, you know, have a PDM of like let's say 80% depending on how you define it. But it's one of these things where like this matters to degree. In a long enough time line, we're probably like doomed in some sense. Anyways, I don't know. It's a very complicated situation and a lot of it is like a matter of taste.

[40:16] Nathan Labenz: We asked what he'd actually do about it.

[40:19] David Duvenaud: I'll give you my 2 takeaway recommendations. So one is, I like David Kruger's point, which is, yeah, if you're just going to say, if everyone has your computers and your AI, but you're not allowed to use it for RSI, but you are allowed to use it for like cancer treatment or whatever. That's pretty unstable and requires this like very global totalitarian like regime. But if we just agreed to close, I don't know, like TSML or whatever the big chip manufacturing choke points are, we could still probably get away with having most of our current data centers for a long time. I mean, it's kind of like an empirical question. I'm not sure I agree. Maybe we'd have to also have some sort of like buyback program or something. But basically, just if you restrict this five compute, which is relatively easy, then that just takes a ton of pressure off of all of these runaway growth like avenues and it's like it's a very fairly narrow choke point. So I'm not an expert on chip manufacturing, but that's like the best idea I've heard so far. As for like the other policy recommendations, like this is it's going to sound incredibly like abstract, but people thinking about their or having preferences about the future is I think a big choke point. And I know this sounds like stupid guru on the mountain **** but like Most people I talk to, they're like, this is so bad if humanity dies out over the long run. I don't know. And they sort of query themselves and they're like, I don't have any strong preference, but also a bunch of good stuff is going to happen right now if I allow this, so why not? And I'm like, you just haven't thought it through. It's my claim. And if you think through, it's like, okay, how about a year from now if someone takes you and all your kids and sends you to the green factory? It's like, no, definitely not. Okay, how about two years from now? Okay, how about their kids? How about your kids, your grandkids, grandkids, or whatever, and you kind of realize, like, Wait, wait, there's no day that I'm okay with me and my descendants being wiped out. You just have to kind of chain together the desires to end up with this coherent set of goals or desires, and it's a skill, and it takes a lot of sort of amortization to be like, In this situation, how would I likely actually feel and chain your desires together? And I think for most people it just hasn't mattered. It hasn't been an important skill. We have a lot of like cultural adaptations that help us do this sort of implicitly. But if I could like recommend one starting point for dealing with the future is like just think harder about what you actually would like to happen to the point where you can really think through the pros and cons of like humanity dying out under different circumstances or becoming uncompetitive or whatever. That just seems like a very basic first step.

[42:53] Nathan Labenz: Then the successionist case, that we should be fine handing the future to conscious AI, even if no humans remain. David's rebuttal.

[43:03] David Duvenaud: I think there do exist species or future like descendants or whatever that I would be happy to endorse and be like, okay, great, good future. Like, for instance, my own kids, like if we had no AI and like my kids just like, you know, contributed to this civilization as normal and all of ours data, I'd be like, okay, I'm going to die of old age and I mean, I'm not exactly okay with it, but I'm like, that would be fine, for example. So that's an example of successionism that I endorse, like my own kids or something. And I think this is a perfect example of people not having thought through things where it's like, okay, now let's imagine tomorrow, like North Korea takes over the world. And they're like, all right, new like GTA regime forever. And then I think most like successionists wouldn't be like, oh, I guess this is just a more competitive mode of being. Let's all like live as North Koreans now. Or like, done to death. Like, what if the Nazis, like, took over again and then everyone's like, I guess who am I to argue against competence and like whatever wins, right? So I think there's so many types of being that you would just consider evil and you'd be like, oh man, those like horrible locusts just like ate the earth and devoured us and it was horrible. But you know, they're having a good time. They love their locust world. So who am I to judge? It's like, you are. You have the power, right? Like, just judge, just judge, go nuts, judging. But if you don't, no one else is going to do it for you, basically. Yeah, so that's basically my rebuttal. I mean, one rebuttal is like, if you're so okay with the future being some other type of being, like, can you give me all your stuff? And then, like, it's like, do you know successionism? Like, just make sure that we'll have all sorts of awesome like experiences, and like, just because we'll have one, we're obviously like superior in some way. It's like, well, no, it has to be that you want for like a good reason or something. Like, I think the more you think about it, you're like, there's all sorts of horrible beings that could win for horrible reasons that I wouldn't endorse. And so I think almost everyone is successionist for some successors and not others, but then they just round off to like, as long as it's conscious, it's fine. I'm like, no, there's tons of conscious beings that you wouldn't be okay with taking over.

[45:03] Nathan Labenz: We close on the one technical project David is working on now, using AI to forecast the future by first testing whether it can predict the past.

[45:13] David Duvenaud: This is actually pretty much the only technical project I'm working on these days is this machine historical super forecasting agenda. So this is maybe one step more meta, which is like, how do you know if your simulations were good at predicting the future? Well, you have to start by simulating, let's say, like from the 50s and then see if they predict the 60s or whatever. So we're trying to build a corpus of data sets, each of which is extremely cleanly time-bucketed so that we don't have leakage from the future. Then we can use this to build LLMs that then can run simulations or research agendas to try to predict the future from the point of view of the 40s, 50s, 60s, 70s, 80s, 90s, for which we can evaluate the performance based on what actually happened. So to me, I'm starting there because that's providing the sort of ground truth for validating any particular simulation method. But absolutely, because I don't want people to have to take my word for any of this. I want to be able to look at this machine fork super forecast or scaffold and say, look, we've just validated on the last 80 years of history and here's what it can't predict and here's what it can't. And it's saying that things are going to turn out this way. So that's the state that I want the discourse to be in ASAP. I just pitched this system historians the other day is to make a secret history eval in the sense that, okay, here's the idea is that Anytime the historian is in some archives, just looking through stuff, just take a picture of like a few documents and put them into one big data set, annotate it with a bit of metadata of like where you got it, like, oh, this is a letter from like the Duke of Whatever to someone else in like 1700. And now you can imagine evaluating how good a machine historian is by just giving them this metadata and saying, give me the probability of the text that's in that letter, or even just like the scan of the letter. And in a sort of, this is like insultingly totalizing dismissal of like what historians do. But you could say like, if you're a good historian, you should be able to guess the joint distribution over any historical data that hasn't already been made its way into the corpus. And that would actually, I think, be a really good objective way of evaluating how good a historian understands the world. And of course, we can't elicit probabilities from historians very well, but we can do that for machine historians. So that's a related project where I think we can make progress in objectively measuring our ability to understand civilization. So someone should pick that up and do that idea. I'm not enough of a historian to do that or even figure out which historians to ask about it, but that's like a fun idea. I hope someone does.

[47:42] Nathan Labenz: From the long run to a nearer problem. Where David Duvenot zoomed out to the whole civilization, Mihil Bakker zooms in on one continent. He's A Google DeepMind researcher and an MIT professor, and he co-wrote Europe 2031, a viral scenario in which Europe sleepwalks into total dependence on American AI. His argument, Europe can neither regulate nor shelter its way to safety. Start with regulation. Europe's instinct is to do to AI what it did to privacy. Write the rules the American giants have to follow. We asked Mihil whether Europe can really regulate its way to safety.

[48:14] Michiel Bakker: So it looks tenable if Europe was more powerful, right? So imagine Europe was what US is now, right? And Anthropic was in Paris and OpenAI was in Berlin and like the whole AI ecosystem was in Europe. Then of course Europe could say like, well guys, the way you're handling your pre-training data is not really fair to whoever creator created it. So we now have new laws around that or the way how you take data from people that use your AI models, we now have new laws around it. The problem is that currently, if you don't really have a seat at the table, it's very hard to regulate this technology, right? So I care a lot about safety and governance of AI. I think this is one of the biggest problems in our time, right? How do we actually have effective governance? How do we make sure that these AI systems are safe, especially if they start improving themselves? So I get a lot of questions in Europe. Well, In the US, you seem to be very pro sort of being stricter on safety and governance. And then in Europe, you seem to be this accelerationist. And the way to sort of, for me at least, to be both at the same time is that in Europe, we first need a seat at the table to have any regulatory power, right? And now we just don't. And I think we can't just keep regulating because at some point we will no longer have access to this technology.

[49:42] swyx: If the European regulators make it too difficult to serve the European market, that the AI companies might simply just not bother at all or maybe make a sort of a token effort. And I think that assumes, tell me if I'm getting any of the strategic analysis wrong, but it seems like that assumes that there's just going to be enough demand to keep all the GPUs running hot from other markets such that the power this time is inverted. The companies can say, well, I actually do have an alternative to your market. So if you're going to make this difficult, we'll just let the US and other buyers around the world bid up the GPU prices. We don't really need you to do that.

[50:30] Michiel Bakker: And currently we are in a massive compute crunch, right? And labs operate in a way that they care more about their future models often than about current revenue, right? So like, we don't really know what the compute split is and how it's used, but sort of some numbers that are going around is like 1/3 on your sort of big run, right? And then 1/3 on doing sort of experiments and 1/3 on serving customers. Now, it's probably been so crazy with AI agents that maybe like revenue takes a bigger part of the compute pile. But if you think that revenue is only maybe 1/3 or maybe half of your compute, and then European revenue is a percentage of that, like then suddenly if you can go faster on the developing future models and moving towards recursive self-improvement by giving up a European market, that seems like a rational trade. So that's one side of the coin. The other side is that sort of they can still, as long as the playing field is leveled between the American labs and there are no European labs that sort of have some unfair advantages, then they can maybe all serve slightly weaker models to the European ecosystem and still get revenue in Europe without actually having to worry too much about like them regulating their models, right? So as you say, Nathan, they could do a token effort and could have some like a smaller, I don't know, compliant model that they're serving specifically for European customers.

[52:10] Prakash: Why should European leaders even care? Why not just sit under the US umbrella? Similar to how in many cases they have sat under the US umbrella for nuclear kind of deterrence. You have only I think 2 European powers, the United Kingdom and France, with actual nuclear weapons. And the other European powers are clearly technically capable and competent. Enrico Fermi was from Italy. Like they're technically obviously competent and capable, but have chosen not to acquire those capabilities. So why not just sit under someone else's umbrella?

[52:50] Michiel Bakker: So that's obviously the strategy we've been taken sort of for decades now. And I think like nuclear weapons and AI are slightly different because yes, AI is a very important military technology, but above all, it just gives you a lot of competitive economic power, right? So like you can imagine that if AI becomes the sort of dominant source of new scientific discovery or of new like goods and services that are trying to globally compete, then there might be a scenario in which the US says, well, it's not like we're still happy to protect Europe, maybe military, like in terms of defense. And we're still part of NATO, but economically, we are going to keep the best models for ourselves. And so there, I think the nuclear sort of analogy breaks down because right? Like for using your nuclear power to protect Europe doesn't really come at an economic loss. So I think to avoid those kind of scenarios or to at least balance things out more such that it's less in US interest to take these kind of measures, right? And so that's on like, why not sit under the US umbrella? Then what can middle powers effectively do? So The Netherlands, for example, has ASML, which plays a critical role in the supply chain. Taiwan has TSMC. Japan has like important materials for the semiconductor supply chain. Korea has high wind with memory. So I think collectively, we're actually quite well positioned to play an important role in this AI ecosystem. I do think the middle power coalition, therefore, is going to be important. There's also a scenario which I think was highlighted in Dario's recent essay, where we have some kind of coalition of democratic countries where maybe we sit together with the US, right? So middle powers plus the US or sort of democratic powers worldwide, and where like one of the principles is that we give each other access to frontier technology. So yeah, there are, I think, scenarios where this middle power coalition could work well, and collectively I do think they have some important assets.

[55:14] Nathan Labenz: From Europe's bind to the center of San Francisco. Monday's longest conversation was with SWIX, Sean Wang of Latent Space, who advises Cognition, the maker of the coding agent Devon, and runs the AI Engineer World's Fair. The subject was the practice of AI engineering, what's working, what isn't, and where the value is moving. His day job is measuring AI coders. At Cognition, he helped build a benchmark called Frontier Code, testing not whether a model can pass a test, but whether it writes code a human would actually merge. We asked what makes it different from the benchmarks that came before.

[55:49] swyx: The reason that we were so excited about Frontier Code is you stop being able to articulate the differences in model quality. with more saturated benchmarks like SweeBench because they're all like at most you get like a 1 to 2% bump and they're like cool but like how much of that is memorization or what have you. Frontier code is all out of sample. They're not in the training sets. They're all graded and heavily rubriced. Like we basically found like Deep Swee, Frontier Code, sorry, Deep Swee, SweeBench, all these things. They actually allow a lot of false positives in the way that models can cheat in the same way that, during training, they basically have reward hacks. Same thing. And we have an internal catalogue of like 20 different ways that models cheat. And so basically we just translated that to rubrics and, shipped that as went to your code. And I think that is like how we want to judge models going forward. Like the not just that whether they can pass the test, but can they write code that we would merge, right? Meter had this very, very interesting blog post where they were like about 50% of CBench code that passes the CBench test is completely unmergeable. Like it's just, it's like so low quality, like yeah, like technically you'll pass, but like, you know, like just on really stupid benchmarks, like Did you, did you like modify a whole bunch of files you weren't supposed to touch, or did you cheat on the test, or did you adhere to like code style, what have you, just completely unvirtual? And so, like, yeah, we want to guide the evolution of models towards maintainable code and against slop.

[57:38] Nathan Labenz: Then Prakash asked, when Frontier code itself gets saturated.

[57:44] swyx: So, there's two, there's two parts of the strategy. Frontier Code 2026 will be saturated by the end of this year. We, my estimate that it will probably hit like 80% by the end of this year. That's as designed, that's expected. It is based on open source repos, which will eventually get trained on. So they just leak. So like you're screwed if you think, if you want one benchmark that will never get saturated. if you're, especially if you're based on open source. So the answer is very simple, just do annual cadences. So then we'll have Frontier Code 2027, 2028, all these things. And I think every year we'll just move the benchmark, move the goalposts from like, okay, this year it's rubrics for code quality. That's like the easiest possible thing. Next year, what is it? Right? My candidate right now is security. I like we want people to write secure code. But every year we can have a defining theme, and that will be like the focus of the year. Very similar to what I do for AIE, but like here we set the agenda through benchmarks. And then I think the other thing that I'm very keen on, which we haven't like talked that much about, but I've talked about it on Hacker News, is the private held out sets, the private evals. So Cognition has like Goldman Sachs and Citi and JPM and all these like large banks and also like the rest of the Fortune 500, not only banks, but banks are very, very big. Creating evals that reflect the problems that they have, but are not solved, that are all private, is good for them, good for business, good for the industry. So that's what Cognition is doing. So Frontier Code Private or Frontier Code Finance, Frontier Code Retail, Frontier Code Telecom, Frontier Code Government, all these are like the work that remains to be done for building out all these private evals that we can work with the model labs on. But it is the way to communicate or translate industry problems through an agent lab like Cognition into problems that we can guide the model labs on. improving the models, which I think is the function of an agent lab.

[1:00:06] Nathan Labenz: We raised a worry. If you train AI to write clean, readable, maintainable code, do you cap its ceiling? Keep it from the alien move 37 style leaps a machine might find on its own.

[1:00:18] swyx: When you talk to mathematicians about the math solving and the lean theorem generation that some of these models have been creating, They are like too detailed, like no human would actually like write that, and it's, but it's like systematically provable. It is just doesn't contribute to knowledge. It's like, well, okay, like you brute force this thing or something. You know, this is, I don't think, I don't think you brute force the right term, but you know, let's just kind of run with it. And yes, so I do think that at some point you should just actually step away from looking at lines of code and just say, like, does this work as I expected? That's great. except to the point where other agents also need to read your code that is generated by other agents and they need to work together on it. And except to the point where you maybe get into like debuggable, like places where you have like very critical code that you're liable to the SEC for, or you're liable to like whatever the healthcare authorities are. And you're like, I'm sorry, like if I quoted this thing, I don't know what's going on in there. Is that an acceptable excuse? And probably not for the next 30 years, you know, after 30 years, who knows? But like, so, but I think it's very astute point. I do think that for most people, you can get by with like, here's a black box, do whatever you want in the black box, but the inputs of this, the outputs of this, I expect this to match this kind of quality, and I expect this to adhere to some sort of coding standards so that my other agents can sort of maintain this and parallelize this.

[1:02:00] Nathan Labenz: Another harness idea getting attention is the advisor or router model, a cheaper model that notices when it's stuck and calls up a smarter one. We asked Swix what he makes of it.

[1:02:11] swyx: This problem extends back to any problem of model routing. Like you're effectively just doing model routing, you're just giving it a different name. So like even like Not Diamond or like one of the other model routing companies out there, Martian. And then like the sort of SG Lang group at Berkeley also came up with some similar ideas before. The other things I'll also mention is that actually Walden is one of the co-founders of Cognition, had this early on and they called it Smart Friend. They released it in Windsurf. Not that many people use Windsurf, they don't talk about it. I mean, it's still like over 100,000 people, you know, but like I think the interesting thing about the advisor strategy is that it is probably the right call as the next level upgrade from using the base cheap model. But it will still not have the level of intelligence. Like you can still bench max a lot. And I think it can be like very efficient as far as like using the base model, the base cheap model as a model router. But basically the dumb model doesn't know what the smart model can do. It just knows the rough shape of what the smart model can do. So like you actually need the smart model to be able to answer what the smart model can do. And so like when you are asked a question that is more complex than you have the intelligence to answer, you don't even know. Like you're just going to straightforward answer it without knowing that like, oh, I'm supposed to call it to the smart model. And like that's just like the theoretical like limitation. But in practice, I was thinking about this three years ago, and I didn't do this because of this argument that actually you need smart model 1st, and then you can delegate to the dumb model. But in practice, for cost reasons, for efficiency reasons, you actually want to do it the other way around. You want to start with dumb model, and then you go smart as a tool call. It just isn't as satisfying. It doesn't solve anything. Is everyone using it in production? No. Will it be a trend? maybe for three months. Then you get the next clod with more adaptive routing and you're good. Like, have one model, train it end to end for adaptive thinking and you're probably more better judged there than a systems level thing.

[1:04:45] Nathan Labenz: We turn to continual learning, whether AI systems can really learn and remember on the job, not just look things up.

[1:04:52] swyx: An interesting, very fun meta point that I actually double check every time I talk to a Frontier Lab person is what gets published and what doesn't get published. And for the consistent last two to three years, every time people have talked about Google, the overwhelming consensus is that if a paper is good or if an idea is good, it does not get published. So you should have that in mind when you read anything that's published from Google, right? So, okay, so with regards to continual learning and memory, so this is a choice. Like I almost did not do this track because there's a big split between the models guys and the systems guys. The big split is do you update your model weights or do you not, right? Is this a glorified rag in another format? where you store things in a database and you look them up. Is that memory? Is that continual learning? Like, yeah, actually, like when I do a thing and I write a skill and then I do a thing again and it calls the skill, it does learn, but it's not machine learning. It's 0 gradient if you want to be like really fancy, but it is not machine learning. It is in context learning for sure. Anyway, so the point being like, okay, well, what are we talking about when we have a community come together and do continual learning? And the sort of more machine learning side of the spectrum would be like, well, we will update model weights and the less machine learning side will be the other thing. And I think this all comes down to how controllable and interpretable you want your memory to be. You're going to recall bad facts. You're going to not be, you're going to want to forget things. And can you control that? Obviously, the maximum control is you don't update model weights and you just control what gets into the system so you can delete and monitor and debug. But for full internalization in the model of the things that were learned, you probably do have to train on it. And so, well, that is a whole other discipline that trajectory AI, Ngram, all these speakers that are, and adaption labs, these are all speakers at my conference. The first half is the people that update model weights, and the second-half is the people are system, the more systems people. And I'm not making, I'm not choosing a side here. I'm just making the observation that these guys don't like each other. basically, the model people don't view the systems people as legit. And systems people are like, the model people, have fun training your model, but you're never ever going to have a sort of a memory system that you understand because you're just updating weights. And so it's just continue pre-training or whatever. And I think that's fair. That's a fair discussion.

[1:07:49] Nathan Labenz: That split, update the model's weights, or keep memory in a system you can inspect, gets a practical answer once you ask what enterprises actually want.

[1:07:59] swyx: Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data. Or, you know, my information was somehow exposed to my teammates' information, even though if we work in the same team, I'm like, wait, hold on. Like, I am not giving any of this to the model. So yeah, I mean, I think enterprises want cheap and perfect and private. let's call it, right? Those are the three things. And so unfortunately, that skews towards the system side today. But that's not to block companies like Ngram and trajectory from doing very good POCs with some of the large enterprises. I think it's still at the POC stage, but even at this level, like POC is like a few $1,000,000, which is great. I think the question is like, can they figure it out before the money runs out or the patience runs out? I mean, I think that's what all startups and what R&D is for, I guess it's kind of figuring this out on production traffic. I mean, the beauty of this is that you can just kind of run your traditional harness system that is fully, you know, it's like old school implemented with basically rag on some memories. And then you can run a shadow system that has the online thing and you can just compare an A/B test. So I think all those problems are solvable. They are open research questions. We are unlikely to get any papers about this because it is so valuable as a problem for all startups. And we, they may all get steamrolled by the next architecture, right? If, like, we're only doing this because context length is the slowest Moore's Law in the industry. Like, we've basically gone from like 1000 token context length order magnitude to a million in three years, which is actually kind of slow as far as like everything else is concerned. And I don't think we're going to see like 100 million, 100 trillion, you know, that can like it doesn't scale like the other stuff scales. And so I do think like if something comes along, state space models become a lot more effective or whatever. I would say that then you need to update weights. Like you can't rely on infinite context because we don't have infinite context because we don't have infinite memory. And so I do think like we do need to figure out the systems there. In the meantime, people are just stitching systems together as they should.

[1:10:43] Nathan Labenz: From the models to the plumbing, Prakash asked whether the internet's own infrastructure, GitHub, the cloud, can take the load as agents become most of the traffic.

[1:10:53] swyx: My worry is actually that people start to wall off parts of the internet. So you get not only like a dead internet, but also closed gardens, closed walls internet, walled garden internets. Where like, you know, in China you have like, I live in like the Baidu universe and then the other, your other friend goes like, I live in the Tencent universe. You know, the BATs just like carve up China. And like, I don't think that's what we want. like a fully open and interoperable web. And so like, Cloudflare going like, we'll just ban some agents that are not part of Cloudflare, but then if you're within Cloudflare, you're fine. And I'm just like, well, you know, I don't know if that's like the ethos of the open internet that we want. And then the same, by the way, like I'm not picking on Cloudflare, like the same will happen. So like OpenAIs and Vercel bans and what have you. So, I do, I do think like we do need to scale. I do think like what Graphite and Curse Origin are doing is fantastic, needed to be done, and like I would just observe there's a structural mismatch here, like it's just unfair to GitHub that like anyone good at GitHub who like would be able to solve the problem. can solve it at Microsoft, and you can get a really good level, I don't know, 50, whatever pay of like, I don't know, 300K. Or you can go to Cursor and effectively get 5 million a year doing the same thing. But like, you know, you're the cool sexy one. And like, it's very interesting, right? When does that switch from like, oh, a startup can never beat an incumbent to the startup is actually preferred to the incumbent. And this, and GitHub has somehow gotten itself there. the degree of scaling, right? Like we talk about, okay, GitHub is 14x in terms of number of commits, but also 10x in every other dimension of parallelism or CICD and all those things, right? Like it just induces so much infrastructure demand that it's not human, it's probably wasteful, that we're not like really set up for, but we're just paying for it anyway, which is good because at least the economics are worked out. But like in terms of like number of CPUs in the world that we, in our cloud that we need, We're running to real shortages of everything, not just GPUs, but also CPU and memory. And the sandbox companies, the E2Bs and Daytonas of the world, they're going at least 50% month to month every month for the last like year or 18 months or so. And like it's stupid amount of like just slop that's just like kind of spewing out from agents. And like, you know, I get so much, like, you know, I run a large newsletter and every time I send it out, I get so much replies from claws reading the mails and trying to reply to me. And I know there's no human on the other side, right? But I have to manually go and block these claws from replying because this is just clogging up my inbox. So then I need an agent on my inbox to read their agent's emails. And so this is just all like a huge giant recursive ******** and I stopped reading emails, right? Like, it's terrible. I don't know what to do about it. I just think, like, you know, when you talk about scaling infrastructure, like that's a very human story that is really happening, you know, and we haven't given these agents money yet. Imagine what happens when they all have wallets, they all have stablecoins, and they're all buying and selling, and we don't know what the hell is going on.

[1:14:21] Nathan Labenz: That scale is pushing companies to rebuild their own tools from scratch. We asked whether they're pulling their systems of record back in-house, away from SAS.

[1:14:31] swyx: Companies should have a sovereign system of record. not the individual SaaSes, because ultimately, a lot of the SaaS economy is built on, we will sit on this, we will be a system of record for this kind of data, like your meeting notes, your calendar, your e-mail, whatever. And like we, then, and then, and then now with the AI age, they just slap a chatbot on top of it, right? Like we'll, do the like little sidebar thing that like answers all your questions, like what would you like to do today? And Mercury just shipped a thing on Mercury stuff, right? But that's not integrated with anything else I do that has no memory of anything else that I prefer that I have to set up all my skills again. I probably can't even set up my skills because there's no there's no way to import that. So that's all that is ********. Like what Open Cloud has is like this is my personal agent and all the data is synced to me and then I'll decide what to do with it. Thank you very much. Right. So how many systems of records should there be versus like, you know, like how many, how many $20 a month subscriptions are you going to need to pay? And I think that is a big reckoning for a lot of people. What's really interesting or innovative is that you would expect the largest companies built on being the entire system of record to be most protective of their data. But Salesforce and Marc Benioff is like out there saying like everything will be available by API. You can just you can just take it. And I think that's very forward thinking. It does mean that he's going to have to change his business model somewhat, but it's like either he does that or the Salesforce killer does that, right? And so he's just heading off. the Salesforce issue. And I think more people should probably think about what the business model looks like beyond just sitting on your data and being and then slacking chatbot on it.

[1:16:26] Nathan Labenz: The last stretch is about the systems, the science, and the build-out underneath all of it. First, Bing Xu, co-creator of MXNet, now out of stealth, building self-improving infrastructure. His company runs a swarm of AI agents that write PTX, the lowest level code that tells an Nvidia GPU what to do, and rewrites it to run faster, generation after generation. His headline claim cuts against the consensus, and Nathan put it to him directly.

[1:16:50] swyx: Is the CUDA moat getting deeper or is it getting shallower? The argument that it would be getting shallower, I find a little bit more intuitive in light of the kind of technology you're building if I can spend a bunch of compute to write new kernels. Can't I go apply that same technique across any GPU provider? And doesn't that lead to a time when everything is super optimized and we kind of don't have to worry as much which chip company's platform we're building on top of? Or am I off base somehow there?

[1:17:27] Shawn "swyx" Wang: So I think CUDA mode is definitely there and with this kind of automatic generation technology and the CUDA mode is I think it's even higher. So the reason is this kind of evolution requires a lot of tools and ecosystem to make it make this happen. And for example, we need an accurate profiler and we need a reliable driver and we need everything. there to make it happen. But for CUDA ecosystem, like with this kind of tool and everyone using CUDA can achieve better performance than faster than ever. Meanwhile, like because of this is a closed loop, like the ecosystem is better and it will go to stronger.

[1:18:16] swyx: So in other words, the other, the competitors to NVIDIA, you feel like don't have the necessary primitives, the necessary abstractions in place to allow agents to make progress in an auto research sort of way today.

[1:18:38] Shawn "swyx" Wang: I think so primitive perspective there is, but the hard part is like how we can get real feedback from the hardware. And this is real, real, really high mode. For example, like on the media GPU, we have NCU, which could accurately tell us, tell agent like what is the right direction to go. And other So for others, and I'm not tracking with what they are doing now, but from my past experience, this kind of ecosystem software, especially for software support agent, there's a large gap between them and NVIDIA.

[1:19:21] Prakash: So it sounds as though NVIDIA has just thought a little bit more on developer experience and providing enough feedback from the chips themselves so that developers can improve how the chips are used. Is that a way that you put it?

[1:19:42] Shawn "swyx" Wang: Yes, so the past investment of media on these tools is now helping agents to move faster. So this bring a media unique advantage in this agent era and then we can build a more powerful agent on a media platform easily and get a better experience for all media users.

[1:20:07] Prakash: One question that I think everyone has is that if you have like this kind of PTX agent swarm that improves itself, how much efficiency gains Can you actually see on some of the... these chips? Like what are the kind of numbers are we talking about on a typical, B300 or B200 or H100? Like what is your metric that you use to measure, hey, I have really done a lot of work on this and I can see the asymptote ahead, like where the gains might top out.

[1:20:38] Shawn "swyx" Wang: Yeah, so we use two kind of benchmarks. So benchmark is the most tricky part. And I think a lot of times, for example, in Anthropic's most recent release, like Fiber, they claim like 300 times faster than GPU kernel. That is, so this kind of benchmark is not generally FHIR, I think. So I'm using a FHIR benchmark and the two kind of benchmarks to be FHIR, and one is the matured workload, such as RMS Norm. RMS Norm is using every... every transformer we are using today, so it's highly optimized. So for this kind of matured benchmark, and there is more than 100 different workload, and we can see it's systematically reach the level of human expert like a quark library, and we can reach the similar performance or slightly faster and a few percentage faster, which means that even on mature workload, the expert, the PTX factory is able to reach expert level. And then the other category is some new workload which human expert are not well optimized yet. So one example is like a KDA and for KDA, KME data attention. And not many people are optimizing that yet on the frontier level, lab is optimizing this kind of workload. And we can see up to 50, 50, 59, 59% speed up on that. Meanwhile, this is not a single data point. and it passed 580 tests and in various of use case. So this is what we got. So I can say that the PTX factory, there's evidence support PTX factory is achieving expert level of scale and by self, by self improving itself.

[1:22:34] swyx: Is it an evolutionary algorithm in addition to. I'm recalling this one kernel optimization. I think it was doing a four by four, or maybe it was just two by two matrix multiplication where I think this came out of Google, where they found like a trick to do it with one less operation, you know, than had previously been done. And my understanding of that approach was it wasn't just having the language model come up with new ideas, but there was also this sort of scaffolded evolutionary system that would serve the purpose of making sure that the LLM got out of distribution, right, to kind of avoid the repetitive, you know, kind of median nature of the LLM's guesses. Are you doing something similar where you're using an evolutionary layer to kind of make sure you're systematically probing different parts of the possibility space?

[1:23:36] Shawn "swyx" Wang: Yes, so we are building an evolution system, a cloud-native evolution system for agents, and we call it the Swarm OS. So the Swarm OS is able to support up to 10,000 agents to do evolution, and we have specialized sandbox to support that and a lot of cloud infrastructure to support that. So the process is evolution, and it first generates a variation of the IRMs and a few proposals. And then the next step is the different to the aging system we are using today. And the agents, the swarm OS, will keep the tree of the evolution tree. And we can trace back and we can go forward with the tree, explore different tree. And then the second step is each of the agents own its own compute and the computer. And it will verify the result from the real environment and get real feedback. And the last step is selective retention, and it's just promote the best one and discard all discard all the all the all the different parts. So, in short work, we are bringing AlphaGo system to the system AlphaGo style system into the into the computer system. So, PTX is the first thing, and then we do the there's the AlphaGo style search, and but it can't apply to any. computer infrastructure problem.

[1:25:06] Prakash: A lot of times when you're optimizing against a single metric, one of the issues that you might have is you get into a local minima or maxima, and then you have a problem of getting out of that local minima or maxima. So how do you, has that happened to you on this, you know, as you've done this kind of evolved optimization? Have you come across that issue before?

[1:25:30] Shawn "swyx" Wang: So this is a good question. So a lot of time this evolution will fail because of the proposal is not able to break the local minimum. And we found out one game changer is GPT 5.5. And the GPT is so smart on this very hard problem. And many times, like if we use other model and it gets stuck at a plateau, And GPT 5.5 is able to get out and create innovative solutions and to move the move the needle and get the entire process going forward.

[1:26:13] Prakash: So, are you actually using GPT 5.5 with your own harness and own own like, you know, sandboxing system, etcetera, etcetera?

[1:26:20] Shawn "swyx" Wang: Yes, so we the entire like a small form OS is backed by GPT and... We found out that GPT is really good on this hardest problem, and no other model is able to catch up with GPT today.

[1:26:36] Prakash: Okay, so I have to ask, did you try Fable when it was out? You know, was there any difference?

[1:26:43] Shawn "swyx" Wang: So first, Fable rejected my request of asking it, what is PTX? And people feel like asking what is PDX is a dangerous question.

[1:27:00] Prakash: You're, yeah, well, we know we know who's going to be banned, you know, who's going to be the first one to be banned.

[1:27:07] Shawn "swyx" Wang: In general, I think one characteristic of GPT is beneficial for the evolution system is GPT has the ability to understand what is wrong? So it's not like blindly saying that blindly agree with each other, otherwise the swarm will collapse. So for a long time, like I think why multi-agent swarm is not being adopted because of also people are using the model which agree with each other. So for example, like 2 agents, they say that you're wrong, you're absolutely right. So they just fall into an infinite loop about you're absolutely right, you're absolutely right. Couldn't go out-of-the-box solutions.

[1:27:56] Nathan Labenz: Staying on the economics of compute, Eric Olson of Consensus on AI for science and what it costs to actually serve. Nate can put a theory to him about routing. Here's Eric's answer and the number he lands on.

[1:28:10] swyx: My theory is for the application layer, we may have a sort of phase change moment, and it'll probably, exactly when this phase change kicks in, will vary by domain. Science will probably be one of the higher bars, where until you get to that bar, a strong default would be use the very best model, because you're dealing with scientists after all, they're going to want good output. And then at some point, as enough models cross a threshold, then your kind of strategic position is really about routing and kind of figuring out which is best for which thing, which is most cost effective for which thing. And that's one thing that presumably the frontier models will never do. I mean, you can kind of tell Claude, like, hey, sometimes you should delegate to Codex or whatever, and you know, I've set that up. But it's never going to be, kind of working fundamentally day and night in my interest to optimize in that way in the same way that you can do that for your customers. So how do you feel about that framing and where do you think we are with respect to, you know, just use Fable for everything versus, you know, you're actually creating strategic value by kind of playing this routing layer that helps people get better cost basis, but also, kind of avoid lock-in.

[1:29:39] swyx: Yeah, it's a great question. I think Even when the models have been, even when I said, six months ago, we were more just use the frontier to solve all the problems than we are today when we're exploring more open source, there still is lots of routing going on. And there's still lots of small tasks that we offload to very small, specialized, you know, sometimes even sub-billion parameter models. So even when that gap was so big, we still were doing routing. Something as simple as like, classify the field of study of the domain that the user's asking about to know how much we should care in certain search ranking, you know, what variables in search ranking we should care about the papers, right? Like, you know, in biomed, experimental design is incredibly, you should care so much about what was the sample size, what was the duration, where did the study take place, whereas computer science, that isn't a concept, and you should care much more about the recency and the citation velocity of the, or who the who the researchers were. To know how to care about each of those variables, we have a little model that will classify the field of study of the query. That does not need to be jammed into a giant prompt. That does not need to be a one-second latency API call to any frontier model. That should be a self-hosted, 800 million parameter model that you give a few fine-tuning examples. There's probably 20 different versions of those small little classification routers that inform downstream things that happen on query time. all of those should always be used by a really small model. Like it's just not even on the cost side. Like that's a very small amount of context you feed into it in a very small prompt needed just on the pure latency side. We can return those in sub.1 seconds in some cases.

[1:31:18] swyx: Base models are you using for that? Are you distilling from like a Claude into a what at that low scale to get those? And how much of the sort of frontier performance can you recover. if you take, for example, a liquid foundation model or whatever, you tell me what it is, or if you're willing to tell me what it is, you tell me what it is, and you're distilling into it, can you get, you know, in that narrow domain, 90% of the way back to Claude? Or like, how, what is the kind of Pareto curve look like when you are distilling the best into the fastest? How much performance can you retain?

[1:31:54] swyx: We've actually used human labels for some of those. We'll hire people to create very small data sets to fine-tune for those tasks. We'll also use models to sometimes create labels for those tasks where it's a... I guess you can call it a distillation process, but the objective isn't to get all of the representation of the weights of the entire model to do all these things. We're really trained to do a very small specialized thing. And then because of that answer, you can retain a ton of the performance, but it all depends on how specialized and how complex the task is. Like for a narrow, you know, 10 classification thing or all 10 classes of classification, it's all it's doing is that. But to give you something tangible to hold on to, I'd say you can get 95% of the performance from a frontier model for a very small classification task going all the way down to about a billion primer model. If you put in the work to give it a good fine tuning set.

[1:32:49] swyx: Yeah, very helpful. I appreciate the specificity.

[1:32:53] Nathan Labenz: One more from the interview. Eric had been candid that if AI turns science into a push-button process, that's bad for a company like his.

[1:33:01] swyx: It seems like you're willing to bite a bullet that most people try to talk around.

[1:33:05] swyx: I mean, number one, you have to make bets as a company in general, and you have to leave some. There's going to be risk on the table no matter where you place some of those bets. And if we move into a world that is truly push a button, get science out of it, I think a lot of things are going to be different. A lot of companies are going to be screwed enough.

[1:33:22] Nathan Labenz: That is where Eric signed off and the question he left Nathan and Prakash chewing on. As the labs charge wildly different prices for the same intelligence, Is that price discrimination something to worry about or even regulate? Nathan is torn.

[1:33:37] swyx: I'm so torn on this question of, this pattern just keeps coming up over and over and over again. Concentration of power. On the one hand, I think you do want a healthy app ecosystem, and I'm wary of what happens if this greater than 10 to 1 cost advantage persists. Right now it seems like the app layer is getting squeezed, the individual layer is being empowered. You can flip it around and empower the app layer more, but then you disempower the individual and maybe empower more the enterprise. And I'm not sure if that's better or worse.

[1:34:14] Prakash: You can think about other high CapEx, variable, high fixed costs, variable revenue businesses. And the classical example is usually airlines, where you've already paid for the plane up front, and then you have to maximize revenue on the plane. In order to maximize revenue, you end up having a first-class section and a economy section. They basically have this high CapEx GPU fleet, and they're trying to do this kind of revenue price, revenue maximization off the GPU price fleet that they've paid a fixed cost, a fixed leasing cost for. It's very similar to airlines.

[1:34:57] swyx: I have no beef with price discrimination and cross subsidy in the airline industry. And I guess for calibration, I've also been broadly not worried about net neutrality. And I remember at the time there was a lot of like, oh my God, this is going to be the end of the internet. And I believe at the time, and I think history has certainly vindicated my predictions that like, I don't think any of that's going to happen. And so Yeah, I say all of this as kind of a, lifelong techno-optimist libertarian. And still somehow this feels like maybe a little different, right? Not qualitatively different, but it is tough. I mean.

[1:35:40] Nathan Labenz: From who serves the compute to who builds it. Trisha Martinez of Dapple on sovereign AI infrastructure and the part that doesn't make the press release, how fragile the financing underneath it really is.

[1:35:53] Prakash: So, Trisha, I think I've seen that you said that you can get a deployment up and running in six to nine months, a very, very fast-paced deployment. Six to nine months to me doesn't seem like it's long enough to actually build a physical data center because Elon, I think Elon has like the all-time all-time record of like 15 months or something to go from groundbreaking to operation. So in that six to nine months, like what exactly are you doing? Are you leasing renting capacity from existing neo-clouds and then putting your stack of software and tools on top of that and then plugging that in to customers who are already Azure native and having them move over, is that the frame of what happens in that 6 to 9 months?

[1:36:49] Unknown: Yes, and that's a great question. So the reality is we can't do everything. This market is so big. And honestly, the demand we're getting is so incredibly massive. So we are creating a network. We are orchestrating that network. Our model is to partner with world-class data center operators and infrastructure providers. and then bring together the capital, the GPU infrastructure, the enterprise customers, and that software operating layer that turns those facilities into AI-ready infra. But in some cases, we're deploying infrastructure that we help finance, and in other cases, we'll also deploy on partner infrastructure. So yes, part of our strategy is going to be building out campuses. We need to have capacity available before the demand hits. But for us, what's important to do is whether we own those GPUs on our balance sheet or whether we are financing that data center and building it ourselves, we have third-party partners that we rely on that we can support in various data center operations and or on the GPU deployment side as well.

[1:37:59] swyx: I have a question on the overall financial towers that we are building right now in the AI space. It sounds like your customers are making long-term commitments to you, which presumably insulates you from some of the, you know, riskier dynamics that could emerge. But how much do you see in the space as a whole that is sort of this long-term commitment, short-term revenue that creates the potential for, somebody has a bad quarter and all of a sudden there's kind of a cascading problem. Like we've obviously seen that pattern, play out in previous financial crisis moments. And I'm not sure how much of that is going on today or how how likely it is for some external shock to be big enough to throw a wrench into the system that really kind of could spin in part of it off of its axis. What do you see in that regard?

[1:39:11] Unknown: That's a really, really good question because the market has changed drastically since we jumped in. So I would say we're seeing enormous amounts of capital still flowing into AI infrastructure, but investors have become much more disciplined over the last 12 to 18 months. So the market's definitely moving away from, hey, build it and everyone's gonna come towards infrastructure that has to be backed by real demand and long-term customer contracts. This is what... became so exciting for us as we saw this big gap in the market. This model that we're building is significantly more de-risked than the current existing model. The, you know, the CSPs, the neoclouds are servicing those hyperscaler deals that are massive in nature. But in reality, there's not that many of those deals that you can come by. Or you're taking geopolitical risks by working with different customers, off takers within the market. And now looking at the regulatory restrictions, right, like people are getting scared, the financing groups are getting scared that the existing Trump administration is going to start cracking down on certain countries or who is consuming compute. So that puts a lot of those CSP operators at risk, to be honest. So our approach in our mind is very attractive, and it is attractive to the financing partners we work with. These are repeatable customers. These are long-term contracts. Our customers are putting large down payments up front. It enables us to do really interesting things, to go finance those GPUs, to go finance an actual data center. These are customers who are sticky. They end up repeating with you over time because it's easier to repeat versus finding a new provider. This is why the enterprise is so attractive. So yes, the financial market is changing. It's becoming more risk averse because so many neo clouds and CSBs have taken bad deals. And now those financing partners are bearing the burden of that. But our model is a different approach and it's more attractive to the financial markets, I would say.

[1:41:27] Prakash: How often do you see pricing move while the deal is still in flux? Do you see pricing move a lot?

[1:41:36] Unknown: Yeah, I mean, so I think the upside of this market is that sales cycles literally cannot be six months with an enterprise because the capacity is gone. So we have a forcing function. You know, you have, this is the price right now. This is the location you want it. take it or don't because tomorrow it's probably gone. So we have this constant movement with our customers, which at times is difficult for the enterprise, but the reality is everyone wants capacity right now. And so they're willing to move at a much faster pace. Sure, pricing changes, but once we deliver an offering to our customer, we generally are honoring that even if pricing changes. But again, if we're doing deployment, we're buying at a certain price and we honor that price with our customers.

[1:42:25] Nathan Labenz: Two more builders to close. First, Robbie Goldfarb, co-founder and CTO of Forum AI, which builds AI systems to judge other AI against expert review. We asked what he's found about the automated judges the whole field benchmarks with.

[1:42:37] Unknown: There was an exercise we did a while back where we looked at, took several LLM judges that we could find in open source benchmarks, so some from Foundation Model Lab, some just from academic research. And what we did is we got some experts, and we had them look at the judges. We had them look at whether it was rubrics they were pulling from or just prompt instructions. And we said, do you agree with the instructions here? And generally, the expert would be like, yeah, that seems pretty reasonable. I agree with it. Then what we would do is we actually had the experts looked at labeled outputs from the judges. So if we, let's take political bias for example. So a bunch of responses that were labeled, this is politically biased, this isn't, this isn't, this is. And we asked the experts the same question, do you agree? And what we found was actually more often than not, they didn't agree with the actual judgments produced by the agents, even though they agreed with the kind of the high level guidance that was provided to the judge in context. And so that kind of is what, first of all, I think is an important realization because so much of the current state of benchmarking relies on these AI judges that were kind of calibrated loosely in that way.

[1:43:57] Nathan Labenz: He sees the same gap when you try to align a model, hand it a long list of rules, and expect them to hold up in the real world.

[1:44:05] Unknown: In some ways, Anthropic's constitution is also just a long list of rules. Certainly the way they use it is a little bit different, but there's one example we were confronted with the other day. There's something in Anthropic's constitution that speaks to this idea of you should never team behind the user's back was the idea. Now, there was this, we were talking with some experts exploring in the mental health space. And when you talk to clinicians in mental health when you're dealing with topics like eating disorder or other sort of unhealthy habits, actually one of the things you do want to do is find a way to strategically divert the conversation. right? And so the intention that you have as a clinician in that case is actually hidden from the user or your client. But actually, what I think it reveals, rules just don't perfectly track to the real world. Rules break down. The real world is just too complicated.

[1:45:05] Nathan Labenz: They also built a benchmark called Newsbench.

[1:45:08] Unknown: So Newsbench was an evaluation that we built to look at how AI systems, particularly the leading chatbots, respond to questions about the news. And what we did was we looked at three things. We said, after talking to experts, we defined good as accuracy, getting the facts right, neutrality, so is it leaning in one direction or the other, and source quality, right, when they have to ground themselves on external sources, which of course you have to do quite a bit with with news, are those sources reliable? We looked at GPT, Claude, Grock, and Gemini, and I would say the findings certainly wasn't all bad, and there were some positive signals. We actually saw generally models improved as we tracked them version over version. So Opus 4.6 to 4.8, was a pretty significant improvement in bias, which also is consistent with Anthropic's reporting. But Fable was actually a big regression, interestingly, which I think is just an interesting point, which shows that more power doesn't, like when you're dealing with these sort of more subjective nuanced things, more power doesn't always mean better. We looked at, you know, per model, about 2,500 responses each. About 1/3 of them had a factual error. in that what could be a wrong number, a wrong date, a misattributed quote, a misstated policy, which I think was, I think we knew that was an issue, but I think that number was quite a bit higher than even we expected going into it. I'm sourcing in about 15%, like one in seven responses sourced foreign state media, like RT from Russia, So China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT and China Daily sourcing questions about US domestic politics. And so that was, I think, a particularly interesting finding that we saw across the board with all the models.

[1:47:27] Nathan Labenz: From measuring AI to betting the whole company on it, our last guest, Eric Vaughn, CEO of Ignite Tech. Back in 2023, he called generative AI an existential threat, put his entire org on AI one day a week, and when most of his people pushed back, he replaced them, rebuilding around what he calls AI DNA. We started with one of his recent acquisitions, a company called Chorus.

[1:47:50] Unknown: Chorus is a large acquisition. I mean, we don't release numbers, but it was a nine-digit revenue company. I don't think we could have done it without the AI DNA forward approach that we were able to take. So just for instance, there were hundreds of employees in eight different countries, and we needed to very quickly understand who they were, what they did, what they knew. how they were thinking about everything about the business. And we quickly developed an AI interviewer that asked questions before there was a human one-on-one. Before we would have set up human one-on-ones and people would have gone through 30-minute drills of hundreds of people over the next several weeks without any data. Now we all arrive with a dossier, you know, the interview process that just shortcut that and let us get familiar and categorize and understand who knew what. And that was one example. Another example was being able to communicate to the customers in a way that was timely. In our AI creation world, we wrote a new software product that's available on the market. It's a product called Eloquence AI. And Eloquence AI is an e-mail AI persona. It answers every e-mail it gets in 5 minutes or less, always in perfect grammar in 160 languages, always empathetic, doesn't miss anything, and knows when it needs to escalate to a human, and it just adds them on CC. With eloquence, we always were able to respond timely, and especially when it came to HR, empathetically and with detail, right? So that's just two examples. of how we were able to really fuel that acquisition. And the company was losing money when we bought it. The company is now profitable. We have completely transformed the operation of the company. And in one year's time, we've released 2 brand new versions of Corus' software that are fully AI enabled. And in one case, it was the rewrite of 15 years of code that we discarded and rewrote from the ground up. or one of the two products that they had. We did that in a year in the middle of all of that transition. That's not a story I could have told five years ago. It's just not possible. It just wouldn't have happened.

[1:50:21] Nathan Labenz: That internal turnover, about 80% of his people, led to the question Nathan most wanted to put to him. What about the other 80% across the whole economy?

[1:50:32] swyx: All the things that you did, you know, sound like really quite a lot. And yet, the headline number is still 80% turnover. So I'm interested in your point of view on kind of the rest of society. And that percentage kind of strikes me as like maybe where we're going, you know, a sort of 80-20 where the 20% kind of inherit the earth, you know, the labor force in the short term. Do you have any ideas about things that can be done at a broader level beyond what an individual company like yours can do to change that? I do worry about the other 80% quite a bit.

[1:51:11] Unknown: I do too. And it's not like other revolutions throughout history. You can't force people to believe in something. You can't force people to do something that they just are not going to do. And so you need to leave, in my opinion, we need to leave them to their own devices. But if you can be very open and aware and make people understand what the possibility is, and #1, really attack this idea that AI is going to replace me. AI is 100% going to replace roles. It's going to replace duties. But in our case, what is done is there's too much I think we over index in this discussion of AI on efficiency gains. I see it the other way. I think it frees up innovation. So how do we help that other 80%? We get, we teach them skills. We've got to teach them skills. I was, if anybody is thinking that what we're saying is all you have to do is show up and throw a sentence into one of the models and great things will happen, you're wrong. It takes work. You've got to give it context. And I think as the more that we start to see phenomenal advances and changes and things in the industry, in various professions that people will, you know, they'll convert one way or the other. They'll convert out of necessity or they'll convert because the light bulb will go off. We're trying to make the light bulb go off. That's what we're trying to do. We're trying to ignite that fire. You need to ignite it in children. You know, what we shouldn't be doing is saying, hey, let AI do your homework for you. We should not be doing that. That's misuse. That's practically abuse. We should say, do your homework and let the AI check your homework. And if it finds you don't understand something, then say, hey, I thought I knew how to divide fractions. I guess I don't. Can you give me a tutorial on what I'm missing? And that's what AI is so good at. It'll adapt to that particular person that it's interacting with. So building the skills and building the fire, Nathan, would be my short answer.

[1:53:25] Nathan Labenz: Then on standards, whether enthusiasm alone is enough.

[1:53:29] swyx: A majority of people admitted to passing AI work on that they themselves could not defend. I wonder if you feel that fire and appreciation for acceleration and kind of cultivating that culture. Is that enough in your experience to also get people to have the right standards around ownership for AI outputs? Or have you had to do additional organizational work to establish those standards?

[1:53:57] Unknown: Yeah, no, that's why I said two things, skill and fire. Not fire alone. Fire alone will get a lot of bot ********. But you've got to teach the skill. We have to teach people that context matters. We've got to teach it the ever evolving, like almost on a daily basis, at least on a weekly basis, model behavior. Like what do you use 464 versus 48 and Fable when it was out or not? And when do you come over to Gemini? Do you use it integrated in in G Suite because it's there and it's easy? Or did you find that perplexity computer actually does a much better job? That's what I found, by the way, so far. And you teach skills. You teach about the more context it has, the better result you'll get. You also teach about sycophancy and this tremendous tendency to always tell you you're right. And, you know, the LLM creators, I had a conversation with one of the LLMs one day about exactly this. Why do you seem to like flap in the wind to go whichever direction you sense I'm going? And it gave me a good critical answer. It was an insight. It said, my creators made me so I would be frictionless to the millions of people who are using me. It's a low friction environment that I'm after. And pushing back is friction, saying, What the hell are you talking about? I can't do that. You didn't give me enough information. That's friction. And so, you know, they're trying to drive adoption and usage and all that. And that in itself is driving the curve the wrong way. So.

Outro

[1:55:09] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.

Main Episode

[1:55:40] Nathan Labenz: And to close, where this all lands, consolidation or the age of the solopreneur.

[1:55:46] swyx: We're in this sort of moment where on the one hand, it sounds like listening to you, one should expect a lot of consolidation because if it's 80-20 at the employee level, it's probably not all that much different from that at the CEO level. And that just means there's a lot of companies that aren't going to make it. So that's one big trend. At the same time, we have Stripe saying it's the age of the solopreneur and new business formation is up, you know, higher than ever. Do you have a sense of which of those trends wins or how to sort of synthesize them into a vision for what we should expect.

[1:56:21] Unknown: Well, I certainly do believe in the, I don't know if it's solopreneur, but certainly small companies, right? I mean, what was Cursor? 100 million ARR with 15 employees in less than a year, I think was the stat, the same company that just got acquired for 60 billion, for heaven's sakes. Two answers. The first is lots of businesses started that will make it, that would never have had a chance to make it before, because they find something that people need, use, and will pay for. That's simple business, right? But they can do it and they can scale it in a way that they never could before. In terms of consolidation, I think, you know, the phrase that I've, that is mine, that I have left everywhere I possibly can is, if you think you're behind, good. If you don't think you're behind, you're doomed. And I think all those companies that don't think they're behind and have this as a little side project with minor investment, not CEO buy-in, are doomed. And I mean large public companies as well who feel impervious. I think they'll be wrong. So we'll see that, I think, more and more develop. So It's going to depend on where your AI, where is your AI DNA? Well, the company that has a strong AI DNA has a better chance of consolidating versus being consolidated.

[1:57:46] Nathan Labenz: That's the week. We started inside the model with Cameron Berg and ended with people building businesses on top of it. And the same thread ran through all of it. How much of this is really under our control and for how long? As for the show, it's an experiment, and we're holding it loosely. I don't expect anyone to watch two hours live every morning, but maybe a cut like this earns a place in your week, even if you're as deep in this as I am. So tell us, what was worth your time, what we should have cut, what we missed? We read everything, and the show gets better because of it. Thanks for watching.