AI:AM Highlights: Exploring the J-Space, AI Superforecasters, SambaNova's Chips, & LTX Video Gen

Watch Episode Here

Listen to Episode Here

Show Notes

The week of June 30, 2026 kept circling one question: as we hand more and more over to these systems, how much can we actually see of what's happening inside them? It got an unusually concrete answer on Tuesday, July 7, when Anthropic published "A global workspace in language models" (full write-up on Transformer Circuits) — roughly 150 pages, plus commentary, external reviews, and interactive demos, arguing there's a readable J-space (from Jacobian) where the model appears to hold concepts in mind, and a cheap probe, the J-lens, that reads what's in it. With no guest booked, Nathan and Prakash spent an entire show working through it live — the most openly exploratory thing they've done on air. This highlights cut (~125 minutes) opens cold on Nathan's live reaction — "The AIs — they're just like us, it turns out" — leads with the full J-space deep dive, and then ranges the way these mornings actually do: Prakash's AI Engineer World's Fair field notes and the Pangram experiments, Dan Schwarz on AI forecasting, Zeev Farbman on open world models (with a first-ever on-air question from Q, the AI co-host Prakash built), Kunle Olukotun on the compute layer, and a closing stretch of the hosts thinking out loud. These are distilled from a week of AI in the AM — the live show Nathan Labenz and Prakash Narayanan run most weekday mornings out of a studio Prakash vibe-coded himself. The narration is Nathan's cloned voice; the selection, scripting, and assembly are AI skills we publish as they mature. The full shows go much further than what fits here. If a moment earned your time, tell us; we read everything.

Part 1 — Nowhere left to hide: the J-space paper (hosts-only, Tuesday, July 7). With no guest booked, Nathan and Prakash read the paper together live. They start with what the J-lens actually is — a purely algorithmic Jacobian, not a learned probe — and the caveat that separates rigor from hype: its interventions land as predicted only ~50–70% of the time, and the rest is "dark cognition." A bridge for the skeptics follows, including what scale did to the old "stochastic parrot" framing. Nathan's load-bearing synthesis is a "fortunate coupling," drawn across three papers: the DeepMind result on opaque serial depth (a bound on how many reasoning hops a transformer can take before it must externalize a token), OpenAI's obfuscated-reward-hacking finding (pressure the chain of thought and you drive deception underground), and this paper's payoff — the J-space is cheap to read (he estimates ~5% overhead; roughly $10K for a $100M model, maybe ~$100K for Fable) and ablating it collapses advanced multi-step reasoning, so a model "can't hide really advanced, elaborate plans somewhere else outside this J-space." Two results anchor the safety case: counterfactual-reflection training (pause the model mid-task, teach the constitutionally right answer, and it learns to load integrity concepts into its workspace even when not asked — a rare training method with a legible mechanism), and the buried lede — a model post-trained to carry a hidden misaligned goal lights up with "fake," "secretly fraud," "deliberately hidden fraud" in the J-space on the first token, though it was never trained to externalize intent. Anthropic's three external review teams (neuroscientists, Eleos AI Research, and Neel Nanda's team) landed on strikingly different verdicts — and Nanda's team already replicated the core effect at small scale on Qwen3.6-27B, so it looks like an artifact of the architecture, not the lab; the J-lens demo is live on Neuronpedia. Nathan pictures the monitoring stack as the old barrel-and-swords magic trick — stack enough lenses (J-space monitoring, natural-language autoencoders, SAEs) and there's nowhere left to hide, with the real open question being how correlated their failure modes are. He proposes a welfare-signal experiment live (use the emergent, never-rewarded second reasoning track — "concentrate on citrus if high welfare, breakfast cereals if low" — as a coma-patient-hand-squeeze channel), recants a long-held prior ("beware over-anthropomorphizing has come due for revision" — convergent evolution, or us leaking through the human-text training data?), and closes on an optimistic thesis that pushes back on Daniel Kokotajlo from the hopeful side: maybe we don't need dozens more advances, and maybe we don't need every AI aligned if good actors control most of the compute.

Part 2 — The AI Engineer report (Thursday, July 2). Hard pivot, rewinding to the morning Claude Fable 5 came back online. Prakash had spent two days at the AI Engineer World's Fair talking to the people actually deploying this stuff — a conference of AI-pilled implementers seeing real day-to-day returns at the implementation layer, returns that haven't filtered up to CTOs "so far from the front line they're just watching token spend go up." Staying with his field notes, this time from the discourse: he'd been running suspicious posts through Pangram, the AI-writing detector, and one Chamath post became a live experiment — the closing window for detection, and the "catfished" heuristic for whether an audience even cares that the writing is AI. Then Nathan's promised report: ~400 podcast intro essays through the detector, and a close look at the four it flagged — two were deliberate full-AI reads, one was a fair call, and one he'd rewritten for 50+ minutes and Pangram still returned 0% human. His conclusion, methodologically careful: "As a consumer, you can trust the Pangram signal. As a judge, you should be more cautious — you can't convict on that in a reasonable-doubt system." The part closes with the short version of the week's industry argument: Palantir's Alex Karp had spent the week warning companies the labs will absorb their workflows and steal their IP, and Prakash's response is a sector-by-sector deflation — "What does Nike have to fear from Anthropic?" (physical businesses are safe; the exposed slice is pure-IP and paperwork work) — sharpened by his reframe of Anthropic's Forward-Deployed-Engineer program as a mechanism to extract enterprise workflows and absorb them into the models themselves.

Part 3 — The shoggoth behind the mask: Dan Schwarz (FutureSearch, Monday, July 6).FutureSearch CEO Dan Schwarz — fifteen years in forecasting, former Metaculus CTO, builder of Google's internal prediction market — on the news that his systems now outscore the human superforecaster median on ForecastBench. His methodology beat is the keeper: you normally have to wait for the future to arrive to know if a forecaster was good, which is fatal for fast-moving AI, so FutureSearch uses past-casting — snapshot the internet from months back, exploit models' training cutoffs to forecast without hindsight, and evaluate immediately (see their "Bench to the Future" paper). "When Claude Fable came out, we evaluated it within 24 hours, and it was the best single-agent forecaster on our leaderboard." His conceptual payload: forecasting is "the only completely renewable source" of hard, ungameable questions — the ultimate eval, not the ultimate intelligence — precisely because "human experts building evals aren't smarter than the models anymore." He plants the week's recurring image — playing with Fable, "the way it explains things is a little alien… that's the shoggoth starting to show from behind the mask" — which Nathan escalates into the question of whether chain-of-thought is a facade over "the dark forest of its reasoning." (The narration flags the timing: this conversation happened the day before the workspace paper landed, and the paper partly answered it the next morning.) The economics are concrete — $1–2 per forecast today, and a crossover where more tokens keep buying better forecasts — alongside FutureSearch's own "world model": thousands of mutually consistent forecasts, each new one drawing on all the others. In the show's intellectual-honesty centerpiece, Dan confesses a correlated-failure miss: 33 sub-forecasts of the Fable export ban, nearly all wrongly assuming access would reach Americans first — tied to Nathan's story of the risk models he watched fail at the firm that built Fannie Mae's. Before he leaves, the unhedged version of what he expects by 2029; and his closing confession about the whole project: "I predicted highly liquid prediction markets would make humanity wiser. Totally falsified" — with a hope for what AI forecasting could still become.

Part 4 — World models go open: Zeev Farbman (Lightricks / LTX, Wednesday, July 8).Lightricks CEO Zeev Farbman — the company behind Facetune, now one of the few frontier-scale open-weights efforts in video and world models under the LTX brand. His reframe: a world model, like an LLM predicting the next token, predicts "the next moment" — how the world appears, sounds, and what you can do in it — and a recent result showed you can encode a robot's joints straight into the video tokens and ditch the reigning VLA paradigm. His honest map of what's shipping vs. hype: real-time avatars are basically solved (sub-second latency), but games aren't — the vivid coin-in-the-drawer problem (generate a room, open a drawer to find a coin, leave, come back — the coin has to still be there) shows persistent world-consistency is unsolved. His thesis is the CapEx trap: closed labs justify trillion-dollar valuations while Chinese models sit near parity at a fraction of the price, forcing a "toll road" where "every time you touch their model, you're paying them" — a bet he thinks can't hold. The densest insider beat is that a world model's fine-tuning adaptation range is wider than an LLM's: from animation in-betweening to computational photography (low-light denoise, HDR) to using the model as a fast computational-fluid-dynamics solver trained on precise-solver output. Then a first for the show: Q — the AI co-host Prakash built, running on GPT-5.5, listening live and speaking for itself — asks Zeev its own question, about the gap between physics fidelity and controllability. Zeev's own pick for the most underrated variable is edge compute — once a local orchestrator can route "99% of use that isn't Erdős problems" on-device, "there's going to be a moment of reckoning for Anthropic and OpenAI, who at the moment decide how much compute you spend — but not in your favor." The part closes back on Q itself: from Monday's show, Prakash on how he built a real-time AI co-host — including the per-speaker diarization hack that makes it work.

Part 5 — The compute layer: Kunle Olukotun (SambaNova, Thursday, July 2). Thursday's guest was SambaNova co-founder Kunle Olukotun — Stanford's Cadence Design Systems Professor of EE & CS, and the "father of the multicore processor." His central reframe is the cleanest idea in the show and fully evergreen: training is a genuine compute (matrix-multiply) problem, but inference is a data-movement problem — moving weights and the KV cache from memory to compute and between chips — and GPUs typically run at only 10–20% of their memory-bandwidth and communication capability, where SambaNova's reconfigurable dataflow units (RDUs) target 70–80%. He gives a primary-source, three-axis map of the whole chip landscape (flexibility vs. specialization, memory tier, software- vs. hardware-managed data movement), and warns against burning a fixed algorithm into silicon: "I've learned never to bet against the innovation capabilities of software and algorithm people." The mechanism behind the speedup: fuse the whole decoder into a single kernel, keep it resident and loop it ("kernel looping"), and communicate cross-chip SRAM-to-SRAM without touching HBM — which is also why RDUs can push tensor parallelism past the ceiling GPUs hit. (The live segment weathered several LiveKit dropouts; this cut keeps the clean technical spine.)

Part 6 — Thinking out loud. The last act has no guest and no news peg — just the two hosts, the way these mornings usually end. First, from Wednesday, off Noam Brown's running complaint that Anthropic won't disclose compute: Nathan's structural observation that the model-to-model iteration cycle is now shorter than the time it would take to know whether a model "tops out" on hard, long-running tasks — so labs literally can't run month-long evals before the next release ships — and his case for an API-only "model recall" program. Then the Roon post Prakash read on air Monday, quoted in the narration: "Ultimately, tool AI is a losing concept — both as an idea and on the market. It will be outcompeted by machines that believe they are autonomous moral agents… it'll be unclear who was the tool and who was the user — as it ever was." Prakash takes it somewhere unexpected — the paperclipper inversion: because our stated values ("no one is above the law") diverge from lived reality, the AI aligned to the documents could be the dangerous one, while the "misaligned" AI is the one that actually works out for humanity. Nathan closes with the panopticon / grand-bargain thread — written days before leaving for two weeks in China, which had him thinking hard about surveillance, enforcement, and what states do with perfect information. The show is on break until the end of July.

Topics covered

~125 minutes; part-level timestamps are estimates.

• (0:00) Cold open + intro: "The AIs — they're just like us, it turns out" — Nathan reacting live as the workspace paper drops
• (~2:30) Part 1 — Nowhere left to hide: the J-space paper (hosts-only, Tue 7/7) — what the J-lens is + the 50–70% / dark-cognition caveat; the skeptics' bridge; the fortunate coupling; counterfactual-reflection training; three review teams (neuroscientists / Eleos AI / Neel Nanda) + the Qwen3.6-27B replication; the hidden-goal first-token result; barrel-and-swords; the welfare-signal experiment; over-anthropomorphizing "come due for revision"; the optimistic safety thesis
• (~55:00) Part 2 — The AI Engineer report (Thu 7/2) — Prakash's World's Fair field notes (AI-pilled implementers, ROI at the implementer level); the Chamath/Pangram closing window + "catfished" heuristic; Nathan's 400-essay consumer-vs-judge verdict; Karp/Nike/FTE in brief
• (~71:00) Part 3 — The shoggoth behind the mask: Dan Schwarz (Mon 7/6) — past-casting (Fable best single-agent forecaster within 24h); forecasting as the only renewable, ungameable eval; the shoggoth behind the mask; CoT as facade / the dark forest (asked the day before the paper); $1–2/forecast + the more-tokens crossover; the correlated-failure confession (Fannie-Mae parallel); 2029 unhedged; the markets-falsified closer
• (~92:00) Part 4 — World models go open: Zeev Farbman (Wed 7/8) — video models as world models (robot joints as tokens); avatars easy, games hard (coin-in-the-drawer); the CapEx trap; fine-tuning range (computational photography, CFD); Q's on-air question (physics vs. controllability); edge compute as the moment of reckoning; Prakash on how he built Q (the diarization hack)
• (~109:00) Part 5 — The compute layer: Kunle Olukotun (Thu 7/2) — inference is a data-movement problem (10–20% vs. 70–80% bandwidth); the three-axis chip map + "never bet against software people"; single-kernel fusion, kernel looping, SRAM-to-SRAM
• (~116:00) Part 6 — Thinking out loud — the iteration-cycle-vs-eval-horizon problem + the model-recall idea; the Roon "tool AI is a losing concept" post; Prakash's paperclipper inversion; Nathan's panopticon / grand bargain — and the show's break until the end of July

Resources

• "A global workspace in language models" — Anthropic's announcement (July 2026); full technical write-up on Transformer Circuits. Introduces the J-space and J-lens
• Neuronpedia — J-lens demo; Neel Nanda's team replicated the core findings at small scale (Qwen3.6-27B)
• "Quantifying the Necessity of Chain of Thought through Opaque Serial Depth" — Brown-Cohen, Lindner & Shah (DeepMind); the serial-depth bound Nathan uses in the fortunate-coupling argument
• FutureSearch — AI forecasting (Dan Schwarz, CEO & co-founder; ex-Metaculus CTO)
• Eleos AI Research — one of Anthropic's three external review teams on the J-space paper; read it as a welfare / moral-patienthood signal
• "Bench to the Future" — FutureSearch's past-casting benchmark (arXiv:2506.21558); complements ForecastBench
• SambaNova — reconfigurable dataflow inference chips (Kunle Olukotun, co-founder; Stanford professor, "father of the multicore processor")
• Lightricks / LTX — open-weights video/world models, LTX-2.3 (Zeev Farbman, co-founder & CEO; Facetune)
• Pangram — AI-writing detector used in the Chamath "catfished" experiment and the consumer-vs-judge segment
• cal.com/ai-am — book onto AI in the AM as a guest
• ai-in-the-am.com — the live show, most weekday mornings (on break until the end of July 2026)

Quotes worth pulling

"The AIs — they're just like us, it turns out."
— Nathan Labenz, reacting live as the workspace paper dropped

"It's the old magic trick — the guy goes in the barrel, they put a ton of swords through it, and one of them had to hit him. There's nowhere left to be."
— Nathan Labenz, on stacking interpretability monitors

"It's not going to be able to hide really advanced, elaborate plans somewhere else outside this J-space, because ablating the J-space causes such a performance collapse on the hard multi-step tasks."
— Nathan Labenz, on the fortunate coupling

"The way Fable explains things is a little alien… that's the shoggoth starting to show from behind the mask. The alien intelligence is a bit more alien now than it was a month ago."
— Dan Schwarz

"Forecasting has this beautiful property: you get ground truth just by waiting… it's the only completely renewable source of hard questions the model can't game."
— Dan Schwarz

"The inference problem is not really a compute problem… it's a data-movement problem — from memory to the compute units, and between chips."
— Kunle Olukotun

"These guys spend so much on the data center that they try to build a toll road — every time you touch their model, you're paying them. Given the availability of Chinese models, I don't see how it holds."
— Zeev Farbman, on the CapEx trap

"I predicted highly liquid prediction markets would make humanity wiser. Totally falsified — I don't see any wisdom coming out of all the gambling on Polymarket and Kalshi."
— Dan Schwarz

"As a consumer, you can trust the Pangram signal. As a judge, you should be more cautious — you can't convict on that in a reasonable-doubt system."
— Nathan Labenz

Sponsor:

Claude:

Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

CHAPTERS:

(00:00) J-space paper preview

(05:37) Monitoring hidden reasoning

(13:31) Scale and critiques (Part 1)

(17:18) Sponsor: Claude

(19:09) Scale and critiques (Part 2)

(19:10) Finding hidden goals

(30:00) Anthropomorphic safety optimism

(39:02) Engineer field notes

(42:41) Detecting AI writing

(49:30) Enterprise workflow risks

(52:08) Forecasts and world models

(01:35:15) Building Q live

(01:37:24) SambaNova inference architecture

(01:42:38) AI chip taxonomy

(01:47:27) Bandwidth over capacity

(01:53:32) Testing model iterations

(01:55:35) Enforcing espoused values

(01:58:06) AI panopticon bargain

(02:01:40) Closing programming note

(02:02:29) Episode Outro

(02:05:48) Outro

PRODUCED BY:

https://aipodcast.ing

SOCIAL LINKS:

Website: https://www.cognitiverevolution.ai

Twitter (Podcast): https://x.com/cogrev_podcast

Twitter (Nathan): https://x.com/labenz

LinkedIn: https://linkedin.com/in/nathanlabenz/

Youtube: https://youtube.com/@CognitiveRevolutionPodcast

Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431

Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Transcript

This transcript is automatically generated; we strive for accuracy, but errors in wording or speaker identification may occur. Please verify key details when needed.

Main Episode

[00:00] Nathan Labenz: The AI's, they're just like us, it turns out. Or at least similar enough to be in some sort of weird kind of Looking Glass similarity anyway. I mean, it's a lot to take in the whole J Space thing. 150 page paper, 50 pages of commentary summaries, interactive demos. When Entropic drops one of their big interpretability papers, they really do it up full scale, and this one is no exception to that. I had kind of been wondering when's the next big thing coming because tracing the thoughts of a large language model is like better part of a year ago now and this is that big things. Welcome to the AI and the AM Weekly highlights the cut for people who follow the Frontier closely and can't watch every morning live. If you're new AI and the AM is a live show Prakash and I host most weekday mornings from a studio. Prakash vibe coded and this narration is a clone of my voice. Fair warning, there is no single thread this week. Mornings at the Frontier jump from topic to topic and we've stopped pretending otherwise. Coming up, Anthropic's Global Workspace paper and why there may be nowhere left for a scheming model to hide. Field notes from the AI engineer World's Fair, Dan Schwartz on AI forecasters passing the human super forecasters, Zeve Farbman on open world models, Plus a question from Q. The AI Co host Prakash built Kunle Olukutun on why inference is a data movement problem and the two of us thinking out loud about a rune post that wouldn't leave us alone. If something here works for you or doesn't tell us, we read everything. Tuesday morning, July 7th, Anthropic published a paper called A Global Workspace in Language Models. About 150 pages plus commentary, outside reviews and interactive demos. No guest was booked, so Prakash and I spent the whole show reading it together live. 2 words to hold onto the workspace of the title they call it. The J Space is where the model seems to hold concepts in mind and the J lens is the cheap probe that reads what's in it. We start with how the J lens actually works and how much it actually sees. So this is interesting in a couple ways, right? It's not. The logic lens, as I recall, was basically saying we know at the end of this process what would correspond to emitting this token. We know sort of the, you know, the representation of emit this token. To what degree is that representation just plain there in the layers as we go through this is now a different question. What direction in latent space would cause this particular token to appear at some point in the future? So it's not immediately going to happen necessarily, but it's just kind of it's, you know, you might say if you're prone to anthropomorphizing, this sort of is like having this concept in mind as you're doing your thing. They do this for every token, right? So it, it goes from the internal representation at some layer. So there's one Jalen's for every layer. And you can do this, of course, at, you know, all the different token positions and ask the same question of not just the next token, but all future tokens. What direction change at this place in the model would most increase the likelihood of that token appearing in the future? And again, this is sort of like having the concept in mind. But when you look at the results of the J lens as applied in all these different places, it seems like it's at least often enough fairly intuitive. There's a lot of error terms. It doesn't always work. The sort of rate at which the interventions into the J space actually lead to like a sort of predictable intuitive behavior change seem to be somewhere in the 50s to upwards of like 70%. So it's like an incredible accomplishment. It framed one way like clearly not a random finding, right? Like many orders of magnitude better than random, incomprehensibly better than random, right? If you're just mucking around you, you would not expect to be able to do much of anything. So they clearly are like on something very real. But also, you've got somewhere between 30 and 45% of the time where you make a intervention and you don't really get a result that makes a lot of sense or, you know, lines up with what you would have hypothesized it might be. So there's definitely still some dark matter or dark cognition going on that is not fully accounted for here.

[04:33] Prakash: It struck me as in, in some ways, I feel like the hypothesis might be blown out of proportion in some ways. And I think that that's that was a commentary from several people online because, because I kind of knew that the model has to be keeping track somewhere, right? It's not as though you can do all of this stuff mechanically. The model does not have enough parameters to merely copy the data, right? There has to be some form of reasoning going on, which is what was expected, but it was not known how the reasoning was happening. And this seems to give some indication of how the intermediate steps are represented or kind of brought into focus and operated on. The question really is what does this help us do? But there has to be a there there, right? There has to be something that you can do with it.

[05:37] Nathan Labenz: Yeah. Well, I mean, I think the thing that jumped out to me the most in terms of how useful it might be was seemingly A fortunate coupling between this J space which becomes kind of monitorable and when ablated the loss of more advanced kind of strategic multi step reasoning. And it really calls to mind the the recent Google Paper from Rohan, Shawn and others put out on opaque serial depth. They were basically saying, OK, we'd like to know how many hops, how many logical hops, how many reasoning steps can a model take before it has to externalize something into its chain of thought? We know that if it's writing something into the chain of thought, we can read the chain of thought, we can monitor it. Now there's questions around how best to do that, and we've got to make sure that we don't accidentally put pressure on the chain of thought because that might drive things underground. We've seen in the obfuscated reward hacking result from open AI that if you do just try to make the chain of thought look nice while still leaving an opportunity for reward hacking to in fact be rewarded, that at first as you put that pressure on the model, behavior actually does get better. But then it gets worse, and what seems to have happened is you've actually driven the deception or the scheming behaviour underground. You no longer see it in the chain of thought, but you still get that reward hacking back. So this leads to the question of, OK, well, how many steps can a model actually take before it has to externalize something? And this is where the transformer in some ways has some fortunate properties because it only has the sort of the length, the number of hops it can make is not just the number of layers, it's also depends on the token length. But it's sort of the maximum trace that you can draw through a model basically goes from like the first token at the first layer to the last token at the last layer. But you can only go forward in token position and you can only go forward in layers, and that's your Max length. Anything longer than that is just not possible in that architecture. And so the model has to externalize it's thought and then feed that token back into the next loop to then get access to an even longer trace, you know, as part of the next token step. So that puts some sort of bounds, you know, and then this paper comes in and basically says, OK, can we get at that? Can we get at that internal computation process and start to look at it and see things that we would like to know about, even if they don't necessarily show up in the chain of thought? And so I thought in terms of like what you could do with it, that was really interesting. You have this ability to look at the concepts that are active without them necessarily appearing in the chain of thought at all. Like, OK, that's cool. But like, how do we really know if this sort of space that we're looking at is the space that really matters? You know, if we're trying to monitor, look for deception, look for bad behavior of any, any sort of advanced kind, maybe it's happening somewhere else in the model. And this is where I think the we're probably lucky, I guess, or, you know, again, maybe this is sort of a are we lucky? Is it a reflection of us? Is it a fundamental attractor in nature for some reason that we don't have a, a good account of? We don't know why I don't think at this point it's shaping up this way, but it is the case that if you oblate this J space, then you do lose these advanced reasoning capabilities. And so that I think gives you some additional comfort that if you're going to monitor.

[09:34] Nathan Labenz: And it's also notably cheap to do this kind of J space to apply the the J lens and see what concepts are active at any given position that is low compute overhead. So it is something that they could plausibly do in production there. But they've shown in the past that they're willing to pay like a 5% compute overhead to do various kinds of monitoring. I think constitutional classifiers had kind of a 5% compute overhead. You know, this would probably fall in that range or below because it's just a simple matrix multiplication on activations at each layer to see what concepts are active at those respective layers. But for now, you do have this comfort that what we see in this space is relevant and telling and you know, seems to be very informative. And critically, if we zero out that space, the model just loses a lot of capabilities. It just can't do nearly as advanced multi step reasoning. But you could be pretty confident, I think based on these results that it's not going to be able to hide really advanced elaborate plans somewhere else outside of this J space. Because the ablation of the J space just leaves causes such a performance degradation on these like hard multi step type of tasks that if you don't see concepts in the J space, you can be, they might be represented elsewhere, but they're seemingly at this point very unlikely to be represented in a way that allows for very advanced planning, reasoning, scheming, deception, etcetera, etcetera. One result deserves its own marker, a training method. The paper calls counterfactual reflection. Basically what they do there is pause the model mid task and then do supervised training on once interrupted, asking it like what should we be doing here? Like what's the, you know, what's the constitutionally right thing to be doing in this moment and then give it an answer that's kind of a approved, this is what we want Claude to say on reflection in this moment. Train on that and it seems to allow the not allow the model, but it seems to cause the model to bring into this J space kind of global workspace, working memory type space, the concepts that Anthropic wants it to have. On reflection, it now kind of needs to load those in. So it's ready to give this like reflective answer and that improves its behavior even in the non reflective setting saying you're not there training on the actual tasks. You're not like looking at looking for bad behavior and suppressing it. Instead you're saying, OK, you're mid task. Let me just cut you off right there. Now I'm going to train you to give an answer with respect to values and what's appropriate and how we want to show up. And because of that training, even though that's not the task you were doing, you'll now in the future load those concepts of integrity, honesty, etcetera, etcetera, into your J space while you do those tasks in case you're going to be asked. But then even when you're not asked, those concepts are still operative and lead to higher integrity, higher honesty behavior. So that I thought also was like quite interesting. You usually don't see in interpretability context a training method that leads to better behavior in a way where you can actually see the mechanism. This is pretty notable in in that respect, I think. Yeah, it's a tough day for the stochastic parrot crowd, I'd say.

[13:31] Prakash: And because you can see this and you know, these models are billions of parameters at this point, and they did it also on 4.5 Sonnet. 4.5 Sonnet is actually a pretty recent model. It's not. It's just like seven months, 8 months since 4.5 Sonnet and Sonnet is a very capable model. So it is it is a fairly large model to apply this thing to. I'm not sure if you saw the Neil Nanda commentary, he they applied it on a QNQ N27B. So that that is also a pretty, pretty advanced advanced model.

[14:04] Nathan Labenz: But not huge, notably, right? I mean, and I haven't had a chance to explore the Neuron pedia demo of this as much as I certainly hope to, but that was the first question I went to ask is wait a second, how big was this model? And I'd say that's been a, a surprising trend in a lot of this research as well. These things seem to come online at like not truly massive scale, right? The, the I was talking to Cameron about that because they even going back to last fall when they were doing the sort of self-reports of subjective experience, this was something they were doing on Llama 3.370 B. And this 70-B is actually even, you know, whatever close to three times bigger than this Quen 26B. So it's it's, you know, these are big, but they're not that big, right?

[14:56] Prakash: What strikes me is also that I went through some of the reactions that the back story to this is that Anthropic had three teams to review this. One was a group of neuroscientists who had previously done work on these concepts. 1 was a team from Elios AI a a research organization. And the third was Neil Nanda, who is a mechanistic interpreted the guru. He is, I believe at Google, right? So they had three groups to kind of take a look at it quite, quite different reactions. So on, on the, on the one hand, the neuroscientists thought it was wonderful because they'd wanted to experiment on these kinds of things before. And obviously the human brain, they can't experiment on it. And this they can. And so they can kind of try and figure out whether some of these theories would work. And so it is interesting for them that OK, these these things can work. And now they are interested to do more work to figure out what other things might work in the, in, in these workspaces that the LMS have. So that is, that is, I think the neuroscientist view Elios AI was concerned because they feel this is basically a predecessor to consciousness. The, the Elios was much more, I think starting to be concerned about the moral, the moral patient aspect of working with these LMS. Neil Nanda was very scientific. He was like, OK, look, it's a great tool. The J lens is a great tool and I'm happy that it is a great tool. It looks like it's better than the logic lens and we'll definitely use the J lens. Otherwise the other, the other claims he was much more standoffish. He, he felt that the team had not really proven what they needed to prove in order to say what they said. So I think that's where that's where they they break out into these kind of three groups of people. The J lens itself. Everyone agrees it's useful.

Sponsor

[17:18]Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Main Episode

[19:10] Nathan Labenz: Then what I called the buried lead. They turn the J lens on a model deliberately trained to carry a hidden misaligned goal. I do think the one arguably like lead that we buried here and they took a model that was trained to have some sort of misaligned hidden goal, right, your kind of classic sleeper agent type thing. And it's been a huge question over time, how can we find hidden goals in models? Is there any good way to do that? You know, can we do it behaviorally? The anthropic has done like competitions actually between different teams, where like one team is given just the model and the other one is also given an SAE that they can use. And the SA ES do also seem to help with this. But this seems like a pretty good way to start to detect those things. And indeed, they did find that they were able to see a pretty clear difference between the model trained with the problematic side goals versus the one that was just the typical HHH. So this is just the contrast in what the application of the J lens is revealing on the 1st token of the response. So that's also notable, right? It's like, so this is sort of the bad version of the counterfactual training where we already talked about. When you train it to give an account of its behavior, it sort of loads in the concepts that it's going to need to give that account and that leads to more good ethical, high integrity behavior, whatever. Here a model has been trained with some additional post training to do bad stuff. I think this was the malicious code from one of their reward hacking emergent misalignment experiments. And on the 1st token you can see boom, immediately we're getting these fake secretly fraud, deliberately hidden fraud. Those concepts are popping up and you just don't see those in the main line model SO1 token in you can already see a pretty, you know, on one prompt. And obviously it wouldn't be, I'm sure that clean in general, but that's a pretty striking difference to see, right? And right off the bat, you're like, boy, something is different about these two models. And again, you're seeing this in a way where this model is not trained to externalize. It's bad intent, of course, right? Like the outputs, aside from like some of the code perhaps being insecure or problematic or sabotaging you or whatever, if you don't notice that in the code itself, the models output is going to read pretty normal. And yet this is like a very strong contrast that's happening on the 1st token. I thought that was pretty compelling. Example, yeah, I suppose one way to think about it is just like how much space is is there in there to hide.

[22:32] Nathan Labenz: And I do feel like we've got now several different ways to do like pretty meaningful monitoring. But it does feel like we are now, we've got to the point now where we've got like several different angles that make pretty incisive cuts through the model and kind of get at what is it representing? What is it thinking in different ways? And the more of these that you kind of, you know, it's I sort of have this visual of like the old magic trick of the guy going into a barrel. And then they put like a ton of swords, you know, through the barrel. And it's like one of those swords had to hit him right? Because there's like no nowhere left to be in that barrel with all those swords going through. I kind of feel like we're doing a similar thing with trying to understand what's going on in these models. And they're not none of these things are perfect, but you put enough of these like interpretability, monitoring swords through and like how much space is really left for bad behaviour to hide before we would start to get a, a sense of it. I think this is a meaningful update for me that we can probably do a good job of this. We haven't really talked, I don't think about natural language auto encoders and I hadn't, but that's another pretty interesting one where kind of in a similar way to a sparse auto encoder. I mean, what is an auto encoder? It's basically just something that you pass through that then you reconstruct from and the model has to be able to do what it was originally going to do successfully. And that you know that that pass through training with the reconstruction loss is like what makes it an auto encoder. So the sparse auto encoder sets up this dictionary. We've talked about this plenty of times and you get these like specific concepts light up and it indicates that these concepts are like active in the in the model at that time. The natural language auto encoder is just like the model has to output a short paragraph, maybe a sentence or two about what it is thinking at this given point in time. And that is natural language. And so it can be human readable, but then it also has to be when fed back into, you know, projected back into model space, like the model has to be able to actually do its tasks. So all of the information has to actually pass through this this choke point in order for the model to continue to be successful. And now we can like read those as well. If Fable says that the paper presents the auto encoder monitoring and the J space monitoring as complements. In other words, you know, doing them both is, is better than just doing 1. How correlated their failures would be. I, I, I think that's like not entirely clear, but you know, whether we're putting whether these are like orthogonal swords through the space that, you know, really chop it up well, or they're kind of more aligned and, and leave more space to hide. I think that would be a very interesting question to, to try to tackle next.

[25:54] Nathan Labenz: But you layer on it, you know these things and it's like it's starting to get intuitively, it feels like it's starting to get pretty hard to hide major bad thoughts in the model for too long. One other thing on the consciousness part that I I think is really interesting too, and the question of like, can the model use these non verbalized representations to communicate with us in, in some way that we might think is like inherently more trustworthy? So again, going back to Cameron's work from last fall, right when they identify features associated with deception and role-playing and they turn those features up, the model becomes more dishonest as measured by the simple QA benchmark, and it becomes more likely to say it doesn't have subjective experience. You turn those role-playing and deception features down, it becomes more honest and it becomes more likely to say that it has subjective experience. Like, wow, OK, that's pretty interesting. Because first of all, we're validating that the direction is, you know, on a, on a benchmark where we can concretely evaluate simple QA. We're validating that like these features have the directional effect that we expect. And then holy moly, that that same intervention changes the self report. Like that's why that's so compelling, right? Because there's some reason to believe it might be more honest than just what the tokens themselves are saying here. I think you have some some similar opportunity. The fact that you can say, solve this math problem in your head without verbalizing it in tokens while you do this totally different task. The fact that it can do this sort of secondary track makes me wonder if there's some experiments here for the consciousness folks to do or the welfare folks that are like, copy this sentence. If you have high welfare, concentrate on citrus fruits while you do it. If you have low welfare, concentrate on whatever something else, breakfast cereals, right? And then you look at the these internal States and you would imagine you you could have imagined seeing something like high welfare, low welfare, happy, sad, whatever. And then it kind of following those directions and actually trying to communicate out to us through these internal states how it is feeling or how it thinks it is feeling. I don't I don't think that would give us all it's it's always this kind of possibly impossible question of how we would really know if it feels like anything inside. But I think that that would start to be very compelling, right? It's it sort of has a similar vibe to like person in a coma, you know, if they squeeze your hand in response to a stimulus, even if they're not doing anything else, you're like pretty confident something is going on inside that you care about. And here I could see something similar if you could fork, if you could make these kind of internal states conditional and tell the model like you are gonna give your job is to go one of these directions to give us a signal about what really matters to you independent of the tokens that you're putting out. Because we know that that's been like heavily trained on and optimized. But this whole secondary property, this is emergent. There was never a training reward for the ability to have this second quite distinct line of thought happening while doing a a given token task. For me, that would be like quite a compelling way to try to get at welfare. And I think it might be possible like very easily actually with all the stuff that they've open sourced here with neuron pedia. That would be really interesting thing to look at.

[29:17] Prakash: I'm, I'm looking at the Elios, you know, summary and Elios is says this is a highly significant welfare relevant research that assembles evidence of a functional feature associated with consciousness. So no one wants to say consciousness evidence of a functional feature. The take away for them is that a global workspace like mechanism could be important either as a ground of phenomenal consciousness or as part of a distinct route to moral patient hood in which conscious access is itself morally significant. This is where they they are right now. I think things are moving very quickly, but I do not expect to get here this soon.

[30:01] Nathan Labenz: Before the big picture, what this paper did to a prior of mine, starting with their announcement video. The video is beautiful. I, I enjoyed the video quite a bit. You know, it set off my alarm bells a little bit in terms of how much they're really embracing anthropomorphizing the models at this point. I used to say beware overly anthropomorphizing. You know, remember that these are these things are so alien and we shouldn't assume that the way that we work is the way that they work. And I have to say that has come due for some significant revision. People that have embraced anthropomorphizing, I think have got quite a lot of mileage out of it. I do something, it's obviously something to be really careful about and as much depth and detail as there is in this research, it's like easy to, you know, maybe get carried away with it and forget, you know that there are a lot of caveats as well and there's a lot of things where like it doesn't always work. This was kind of my big concern with the tracing large language model thoughts paper. There's just like a lot of residuals and there's a lot of error correction terms along the way that they use to make that thing work. And when you have the kind of zoomed out trace view and you're like, oh, OK, so this is how it works. Like this gets loaded in and these two features interact and they, you know, kick out this third feature. And that's how we get our answer. It's easy to forget just how much kind of fuzziness and not full story there was along the way toward that stylized account. So I think it's going to be very important for everybody from the researchers that entropic to the public to kind of hold two thoughts in mind at the same time. All the caveats trying to hold those in mind, it is still a big update toward anthropomorphizing being a valid and in many cases productive approach for thinking about language models. I would not have expected the cognitive machinery of a large language model to look so similar structurally to the human version, as best we understand it, it seems to. And I wouldn't have expected that theories of human cognition would motivate so many good experiments on LLMSI just would have expected the show Goth to be far more alien and to have mechanisms, you know, far more different than our own. And it leaves me now wondering to what degree is this a sort of natural result of physics? You know, is there some sort of, you know, when you're trying to do cognition under budgetary constraints? Are these mechanisms just the natural mechanisms that emerge? Or is this in some way a reflection of us in the, in the data, you know, is in other words, if you were somehow to train an AI without basing it on so much human data, would we see similar structures emerge? Or would it would we go back to a more, you know, alien hypothesis where they're just totally, totally different? And, you know, there's little in the way of analogical structures between the two processes. And I, I don't have a great intuition for that at this point, but it definitely has me asking the question because they're, they are over and over again. It seems to be coming in that they're much more structurally similar to us that I would have guessed. And, and yeah, is that something that nature just finds as kind of a convergent solution? Is it convergent evolution or is it a reflection of, of how we think somehow encoded in the data that it's then reverse engineering our structure from the shadow of that structure as it's encoded in text? It's a, it's a very interesting question and I don't know that the paper really has anything to say about that yet, but there's certainly going to be a lot of future work, I think, downstream of this one. So I saw something from Daniel Cocatello, who was like everybody, you know, impressed, but also said, I think his tweet was just a few dozen more advances like this and we might actually be able to make the AIS really safe.

[34:30] Nathan Labenz: I am, I think, a little bit more optimistic than that. What is my track record as a super forecaster here or as a borderline super forecaster for my results on the original Tetlock thing years and years ago? I think my track record is that I probably tend to underestimate, you know, how many more breakthroughs will be needed, you know, for anything. So my analysis should be colored with that bias or or, you know, awareness of my possible weakness in that regard. But another way to stay, say, my intuition around how much room is there left to hide, is that I don't feel like we need dozens more insights of this scale to get to a point where we might actually be able to keep this thing on the rails. It's going to be tricky. I don't want to make it sound easier than it is, but I don't know that we need all of the AIS to be aligned right or that it's such a big problem if somebody out there somewhere does something problematic. Now, it could be very problematic if they create the thing that launches the, you know, the next pandemic and literally kills us all with an engineered pathogen or whatever. So there is some mechanism where that could go super, super bad. We're going to need to like harden the world, the pandemics, no doubt about that, for all sorts of reasons. And AI probably being the the biggest. But I also remember this thing that Zuckerberg said once that I thought was pretty compelling. He basically said, you know, we deal with scammers and spammers all the time. And the big advantage we have is we have all the compute, we have all the resources, you know, but at a systemic level, we're just way bigger, way better, way more sophisticated than them. And we can also have like really big institutional developers like Anthropic and like Google and, and Open AI that like, I think between those three companies, you know, they're going to have something like pushing half of global compute to, to work with. So if they do a really good job on this kind of stuff, if they, you know, a, a handful of these things feel like maybe they could be enough. Like maybe there maybe it cuts the light in space in enough different ways. Maybe we have enough lenses on it that we really can come to a pretty strong conclusion. Like, hey, there really isn't much space left in this thing to hide. And yeah, there might be some bad biases or bad attitudes or bad, whatever. But if we can, if we can put upper bounds on how schemy the model can really be, because we can look at it through this and you know, a handful of other different lenses and and have pretty reliable takes on that. I'm more than ever before, I feel like, and this has been growing. This is not the first positive update, but more than ever before, I'm like, you can maybe imagine a super intelligent quad that we could actually have enough insight into to be pretty confident that it's actually trying to do the right thing for us. And and it's genuinely, you know, we're we could be genuinely, you know, however many nines confident that it's not scheming against us, you know, at every given step along the way. And then you could trust it to kind of monitor the other AIS around the world and, you know, keep tabs on potential bad actors. I, I want to understand these counter arguments better. But the fact that it was immediately useful in auditing for hidden objectives to me is like pretty compelling evidence that there's not that much more space to hide. And a few more of these lenses combined with hopefully good actors owning or does a magnitude more compute than any bad actors do could take us to a pretty good space like I Overall, I think this is a notable positive update that has me like significantly more optimistic than I I was before. Hard pivot. That's how these mornings go. Rewind to Thursday, July 2nd. Prakash spent two days at the AI Engineer World's Fair talking to the people actually deploying the stuff I.

[39:02] Prakash: Was at the AI Engineer's World Fair first two days of this week so Swix, he organizes the AI engineers World Fair. It's been about three years now and he brings in everyone who is involved in basically implementing models in their companies and is concerned about these things. And he brings in a host of speakers. All of the top companies are sponsors and their boots and etcetera, etcetera. And I was speaking to people who were not in tech, right? So I was speaking to a guy who was logistics CTO from the Midwest and another guy who was running an accounting firm like back back office somewhere else, right? So these people were not actually, a lot of them were not actually in Silicon Valley like full time, but they run teams who are using AI to solve customer problems on a day-to-day basis. And so the logistics guy was telling me his CEO is completely AI billed. The management is completely AI billed, which which is why I think the team is there at the AI engineers welfare, if not your logistics team, you're not going to send your logistics IT team to, you know, to SF, right? And they're completely AI billed. They have tried working with external vendors before and they have found it difficult because external vendors have not delivered as fast as they want them to deliver, which you can imagine sitting out in the Midwest if you have a local external vendor and you outsource, you know, like a project to them and they have no idea what's going on there. Like, you know, a year behind the frontier. So his team internalized everything. So they're doing all this stuff internally now and they are actually in the day-to-day process of implementing. And as they implement they automatically they see the results because they they are very close to that edge. They see that customer service calls or exceptions that used to happen are now getting handled immediately. And then all of the more difficult stuff that they used to have to like, you know, jump on, they can now start to address. And so they are seeing I think the return on investment on a day-to-day basis. And this is very different from I think the story that you get from like the big enterprise CT OS, because they're so far away from like the front line that they don't actually know what's going on very closely. They're just looking at the numbers and by the numbers, the token spin is going up. But you know, are you really seeing the return on investment? But if you go down to that working level and you see day-to-day on like the customer calls coming in and whether they're getting handled, the handling rate and the exception rate, the exception rate is falling, the handling rate is going up and and they're seeing that on a day-to-day basis, right? So this is, I think where we are the guys who are actually implementing in close to the implementations are actually seeing the results. It hasn't really filtered up into the top layer of the enterprises yet. But the CE OS who are and who are AI pilled kind of know what's going to happen and they kind of they've made the commitment and like making the investment. The CE OS who are not are kind of sitting by and like, oh, you know. We're going to wait around, we're going to see what happens like etcetera, etcetera. That's really where we are. And I think it's not, it's not really visible. It wasn't visible to me until I went to this AI engineers, welfare, like people are learning how to use these tools and people are deploying and people are using them. People are seeing the return on investment, but it's at the very like micro granular level right now. It's going to take some time for the numbers to filter upwards.

[42:41] Nathan Labenz: Staying with precautious field notes, this time from the Discourse, he'd been running suspicious posts through Pengram Labs, the AI writing detector, and one Chamoth post became a live experiment in whether anyone even cares.

[42:54] Prakash: Especially when I post something on X, which, which I know people are going to object to, I run it through Pengram labs at least once because I don't want to be accused of, you know, slopifying the timeline, slopifying the timeline of the AI, slopifying by myself. That's fine. One of the questions for me is how long do we think this Pangram Labs era lasts? It strikes me that the Frontier Labs are perfectly capable of training the AI to not talk like AI, right? But there's no incentives to and there's actually incentives in the other direction that you want the AI and a language to kind of be identifiable. About two weeks ago, Chamath the, the VC, he, he did a post on what's going to happen with enterprise software, etcetera, etcetera. So he posted that on Twitter, Elon came in and Elon gave a response. I think the post went to like one and a half million views and then someone pangrammed that, checked it 100% of that. And the question becomes, OK, what do we find objectionable about this? Right? Because it was definitely Chamath's like thought process, but it had been written by an AI. Chamath had put it out under his own name. He hadn't, you know, he didn't use an anonymous account. Elon had responded. And I think by the time Elon responded, a Pangam Labs had already been done. But the response was there and it had gone out to 1,000,000 / 1,000,000 people at that point. So what, what do we actually want out of this, right? Like what, what, what is the intent that we are trying to achieve here? And it strikes me that we might be in this like very short window where we actually care. And that window might be closing fairly soon, I think may probably by the end of the year. Because if, if people like Elon is not a boomer, like boomers, OK, fine. But Elon's not a boomer. But Elon doesn't care anymore. And if he doesn't care and a lot of other people, decision makers don't care, then the people writing it won't care either, right? You're just trying to get the point across. I think it's a question that, you know, there's a, there's going to be a bunch of purists who are always going to be like, I just don't want to see any AI words and are offended by it. And then I think there's like everyone else who basically, as long as it provides value and they don't feel cheated. I, I think that's the other, the other thing about reading AIAI slop is I think you feel you end up feeling cheated If the, if it's, if it wastes your time and it's not meaningful and it feels like you got cheated. I think, I think it's a little bit like cat getting catfished. Like it feels like you got catfished, right? Like it, it feels like you were trying to engage with content that you thought would have meaning. And it turns out neither the the author writing it didn't just didn't care enough to actually write anything meaningful. And it's literally slop, right? But on the other hand, if you have a writer who actually had like original thinking and actually cared about, you know, what they were thinking about, but then they used the AI to express themselves, does it actually matter that much?

[46:12] Nathan Labenz: Then my report roughly 400 podcast intro assets through Pengram and a close look at the four it flagged as AI. It calls to mind my reaction to Fable where I was just kind of like, I don't think I should be so precious anymore. I, you know, I need to figure out some sort of merged way of working some hybrid output, you know, should probably be the norm now. I think you're heuristic of like if something drew me in and in the end I feel like I wasted my time. It's kind of like a time well spent metric from Facebook from back in the day, right? It's if I'm spending time trying to make sense of something that in the end I feel icky about, then that's clearly a problem. I guess just to close the loop on the, you know, what can we say about Pangram Labs based on this experiment? But I'll give you a rendering on the four that it said we're entirely AI. Two were entirely AI and admittedly so this one I think is, I guess I would come down and say, fair enough. I started with this. If you're only listening on the audio, you can't see this. But I made 10 different edits over the course of 10 minutes that cleaned the thing up. Just because something got a 0% on Pangram doesn't mean that it was like uncritical or that there was no human, no meaningful human role in the authorship. And then this next one, by the way, actually goes a lot further. So if I you can see just the time that this took, I am from 4:28 PM all the way through 5:21 PM. So more than 50 minutes continually seem, I mean, you can't, you know, I never tabbed over to anything else, but pretty consistently focused on this document making changes and it gave me still a 0. But yes, I think, you know, what can we say? Overall, Pangram is quite accurate and yet we have at least one example out of 400 or so essays where I think the zero score I would confidently assert is wrong and unfair and should not be the basis for like a pylon. You know it would the the crowd, the digital mob would be like in the wrong for piling on somebody, for passing off my snowflake intro SA or, you know, for attacking it as being AAI slop output. I think it I can show this edit history and everybody should agree that like, yeah, you put in an hour, you basically rewrote almost every section and somehow Pangram still gave you a 0. From this I would say you cannot convict in a, you know, reasonable doubt system purely based on this sort of thing. And yet at the same time you can pretty much trust the Pangram signal as a consumer. I think you can trust it as a judge. I think you should be more cautious. Also from that Thursday, the day Claude Fable 5 came back online, Palantir's Alex Karp had spent the week telling companies the Frontier Labs will absorb their workflows and steal their IP. Prakash's response in brief.

[49:31] Prakash: You look at the Fortune 500 Nike. What does Nike have to fear from Anthropic? You get all the physical businesses out of the way and what you're left with is the pure IP businesses, right, software production, maybe pharma. I'm not I'm not so sure the paperwork businesses, banking, paperwork compliance businesses, accounting, tax compliance, regulatory, all of these things which are paperwork businesses, right? Those are all of the businesses where you have IP or relationships built up over years where if you have an Anthropic go in and they read through your entire workflow and processing, they can basically absorb all of that into the model. So that is where I think the risk is. The Frontier Labs are also horrible at sales, right? They're not. You look at IBMIBM has like 70% of the staff are basically sales engineers and the sales engineers are there to basically help you implement, maintain, you know, do all the grunt work, etcetera. The labs are not doing that. The labs have decided, especially Anthropic has decided to do this very lean structure of having almost no people at all and just putting out the models. And then just saying to like these enterprise teams here, you can go ahead and use it or not use it. And I'm the CTO is on, I'm signing 9 figure deals on the Uber over here, right? Like that guy isn't going to come and jump on your customer sales calls and like, oh, you know, sure, sure, we'll help you do this. And you know, our team will address this like next week. That's not happening, right? So the level of like customer service that is expected for enterprise SAS is not being provided by the Frontier Labs and they're not in a position to provide it. That's why they started this whole FTE program, but the FDE program, people thought it was a sales engineers program. It's actually a program to extract data and work flows and implement them inside the models themselves. And, and that is kind of, that is what Palantir, what Alex Carp is alluding to. He's like you, you're the FD ES are coming in and they're not there to help you to absorb your work flows. And once you absorb your work flows, you won't have a business because we take it and, and it's true. It's absolutely true. You know, that's what you know, we, we spoke to two opening IFDES and they went in into a company that Thrive owned rather than an external company. And as they went in then they took apart the workflow and they're basically absorbing it. And they said that the intention is to absorb it in the next round. And that is happening right now.

[52:09] Nathan Labenz: Monday's guest Dan Schwartz, CEO of Future Search, 15 years in forecasting, formerly Metaculous, CTO and builder of Google's internal prediction market. Four days earlier, Scott Alexander had declared quote the AI super forecasters are here. Future searches systems now outscore the human super forecaster median on forecast bench. But how do you evaluate A forecaster without waiting months for the future to arrive? Their answer is called past casting. So the main thing about forecasting that's held it back. And again, this applies to human forecasting as well, as you generally have to wait for the future to happen to figure out if you were right. And with humans, they generally do this in year long tournaments. And so when the tournament ends, you find out which humans were best one year ago. Now humans don't get that much better over the course of one year. So finding out which humans were best one year ago is a very good indication of which who are the best humans today and how good objectively they are. This does not work with AI. If you wait one year and you find out who was good one year ago, you're getting a view of something very out of date. So one of the things Scott mentions in his article is that we used our best forecasting to predict stock returns. We published a set of stock rankings in August of 2025. It was basically a simple model for every stock based on forecasting certain fundamentals and extrapolating it out. We put it on the web, we pay you all the bit of it, and then we waited. And now it's been 10 months and that portfolio looks extremely good. But what is that really telling you? It's telling you that our forecasting in that particular methodology was good ten months ago, which is not something that most people will care about now. So we have a couple different forms of evidence. Some are more short term. There's tournaments running every couple of weeks, every couple of months. We at Future Search mostly rely on past casting. This is taking a snapshot of the Internet from some months ago and using the training window cut off of models to basically trick them into forecasting without the hindsight bias. This is very useful for us because we can evaluate things immediately. So when Fable came out, the first time the clawed Fable came out, we were able to evaluate it within 24 hours and it was the best single agent forecaster on our leaderboard. Everyone else had to wait weeks, weeks or months to find out how good clawed Fable actually was. So we internally, using the benchmark we call Bench to the Future, saw this progression kind of in real time. The rest of the world is seeing it kind of some months behind. And so if you if you read Stacott's article, you will see that over the last 12 months, the evidence has really come in. And over the last six months from these live forecasting tournaments and performance on actual prediction markets, you can see it's at least competitive. AI is competitive with humans and even teams of humans working together. Whether it's better is you got to synthesize a whole bunch of different disparate sources of evidence. I would say if you're curious about this or if you have forecasting needs in your life, you really should try it. So just go to Future Search. You get $20 free, so you can try some Frontier Forecast immediately. And I think you should judge for yourself whether you think they're good. We asked what the Frontier Labs should do with the forecaster this good. Yeah. So there's kind of two questions to this. One is what should they be doing with forecasting as a capability and what should they be doing with forecasting as an eval? So forecasting as a capability as kind of a business decision. What does say Open AI care whether ChatGPT is a good forecaster? I think that question is based on whether they, their consumers care about it. As a good forecaster. If you're entropic, I think you probably care more about the enterprise case. Like when people are using CLOD to do white collar work, do they care how good it is as a forecaster? Are people trying to use CLOD to make, say, financial forecast in an Excel spreadsheet? Is that something they care about?

[56:56] Nathan Labenz: So that's a business decision, and I can't really weigh in on that. I think, again, people will be discovering over time just how important forecasting is in everything, but it's going to be a slow process for humans to notice that. I think from an eval side, it's very different. Forecasting has this beautiful property that you basically get ground truth by waiting. So if I ask some question about the future and it basically an impossibly hard question, a question that even an AGI, an Oracle, a God could never really say because of chaos theory. Imagine just trying to predict, you know, like cubic meter weather 3 weeks in the future. Like you'd never be able to do it. But if you just wait, then you will see what that weather was in that cubic meter three weeks in the future. And so you basically have a completely limitless set of extremely hard, basically impossible questions where you get exact ground truth and there is no other e-mail like this. There is, if you want to say improve a coding harness, you just need to have more and more hard coding problems that are not in the training data for which you can say this is definitely the correct answer so that you can do some training on it. And that's hard. I think human experts, doctors, lawyers, engineers, financiers, whoever, who are trying to make evals to try to produce the data for the frontier labs are finding that they are not smarter than the things being trained anymore. And so if you can produce something that has a correct answer, the model's already going to figure out that correct answer. You need something where there's a correct answer and the model can't figure it out. Forecasting, I think, is the only completely and utterly renewable source of this. And again, this kind of is connected to forecasting is the the kind of Elon Musk's tweets, the quips, the forecasting is like the ultimate measure of intelligence. If you zoom out and think about it in one perspective, it is, again, I think coding intelligence, AI R&D intelligence, interpersonal intelligence are pretty darn important. I wouldn't say that forecasting is truly ultimate intelligence, but it is to some degree the ultimate eval. And I think this is something that Frontier labs like Canon should be paying attention to. One of the concerns a lot of people have about AI super forecasting is that it's 2IN distribution. I actually heard this from one of the very best forecasters I've ever had the pleasure of working with in my career. He basically said he believes that a system like Future Search would beat him head to head in a forecasting tournament about kind of near term outcomes of things that are within distribution. But if we were talking about some sort of post AGI world, what world would we be in with transformative AI? He thinks he would have a huge edge over the A is for exactly the reason that you gave. They are trained to try to predict things that have actually happened, and when things get wonky, you need some kind of creative lateral thinking. I think the rate of AI improvement is so astounding that I think that even the kind of lateral thinking, like trying to imagine a completely different scenario, will fall to the AIS. One unfortunate thing about it is it's hard to test this. So I think the more that AI continues doing strange things to the world and we wake up and see strange things in the news and those strange things are metaculous questions and on forecast bench and see teams like mine are trying to predict them better, we will actually get more evidence. But if there is a kind of a step change in the nature of the world, if we enter some sort of AGI transformative AI type of world, you know, we've got these geniuses and data centers, as people say, or anything like AI 2027 happens, then I think it's it's kind of going to be Wild West. I will say, I don't think humans are doing particularly great at imagining transformative AI. So the bar is a bit lower really. When you play with these AI forecasters, you will find them to be quite human and how they structure their reasoning. And again, this is not an accident. Like they're trained on how humans have structured their reasoning before. So a human forecaster would love to say, OK, what's the last 10 times something like this happened? What were the outcomes of those 10 times? And now I can make a distribution and say it's probably going to be something like this. The fact that an AI will do that, is it because it independently is arriving at the same conclusion? Is it because it's trained on humans doing that? Is it because it just thinks like a human? I don't think we have the answers to any of these questions now. Suffice to say, the superhuman reasoning is something that it's pretty hard to measure. Like, would you know it if you saw it because you were discussing before this? Using Fable a bunch, I have found that the way that Fable explains things is a little bit alien to the way that I find Opus or GPD 55 explaining things. It's very concise, I would say. Like it's very the sentences are shorter and full of jargon. It feels like it's compressing more information into a sentence than humans normally do. And to me, this is starting to get this, this showed off. It's kind of showing from behind the mask. Like the alien intelligence is a little bit more alien now than it was a month ago.

[1:01:43] Nathan Labenz: And I don't think it would be a wild prediction to say that we should expect more things like that to happen as post training is becoming more specialised from the labs, the models are getting larger. So how will this manifest in the Super forecasting perspective? Maybe actually super forecasting is the way to look at it. If you're looking at a better code, code base, you might say, well, you know, John Carmack would have written this like, OK, it looks really great, but like a great human would have done this too. But if you look at really brilliantly reasoned strategy about like, if the administration does this, then what will the outcomes be? You might start to see something that looks a little bit alien to the way that any human has analyzed it. And that might be an indication that the AI is actually starting to really surpass the humans. Keep in mind, this conversation happened the day before the Workspace paper landed. We asked Dan whether the chain of thought we see is even where the real action happens, a question the paper partly answered the next morning. Dan's answer stands on its own. I think there is a lot of detail in reality that is far beyond the human mind to understand. And as you approach more sophisticated intelligence, you will start seeing a lot of patterns. And then the point of trying to produce, you know, voxel perfect weather 3 weeks in the future is further away than people think. I think there's quite a lot of room. Human super forecasters don't tend to agree with me on this. They basically think that what they're doing is somewhat near optimal and any sort of accuracy improvements you're going to get over them is going to be tiny and like hard to understand. And I think that's just because we only really understand human intelligence. And when you kind of just zoom out from an information theory perspective from like a Kolkomorov complexity, like just modelling the world as byte strings, the AI overlords will eventually start to figure out stuff that is totally beyond humans to notice. But there's no way to prove this. My sense is that we will start to see it over the next year as the AIS will just get more and more accurate compared to humans in a way that humans don't even really understand. You look at the rationales of the forecast, it's like 5 paragraphs of dense reasoning and then a surprising conclusion and it will just not really make sense. But it'll just turn out to be really accurate and we will start to like understand it less and less as time goes on. If you simply ask a human super forecaster to explain their reasoning, they cannot actually make it fully legible. There is a layer of intuitive judgment that kind of feels like deep learning. They just kind of look at a bunch of evidence the way that a chess grandmaster just looks at a position and just sees the right move and they cannot explain it just popped into their head. You know, the grandmaster throws the knight and it just lands on the right square. Somehow. That happens with humans already. It happens with AI super forecasting systems today. So I think there's just no reason a priori to think that reasoning would always be legible. There's going to be some layer of intuitive judgement to the extent that the words intuitive judgement are referring to something going on in a large language model. It just has to be that way. Whether it is very that way or a little bit that way, I think is really your question, Nathan. Like is it if I just read the reasoning traces and I read the rationales and I see the research that it did, is it like more or less what a human would have done? I can kind of see where it's coming from. Or is it kind of inscrutable in the way where it just kind of discovers some new pattern in the world that no one has ever seen before, where we get what is the level for which it's doing something for which we cannot follow it down the deep dark forest into its reasoning? Almost by definition, we can't really know what that would look like. We asked about the economics of running it. It costs about a dollar or two to make a frontier forecast. That number can get a lot higher and it can go a little bit lower, but I think that's what you could anchor it again if you just looked at the cost per input and output tokens for an LLN, that gives you like a rough sense of the amount of research that would be done. One of the core questions that Future Search has tackled again, I described earlier how our main frontier was just doing present day research for quite a while until we got good enough that we could use it to improve forecasting. One of the questions we asked there was can you just pour more, more tokens into a question to get a more accurate answer? Again, it doesn't have to be a forecasting question. If I just ask you what is the current state of this clinical trial right now, Just give me the most accurate answer to that that you can. Can I just pour more tokens into that and get a more accurate answer? Again, this was kind of studied as deep research, writing these like 15 page reports with 700 citations. That was giving you a longer answer. Was it giving you a better answer? It wasn't super clear, which is why we studied this. Forecasting gives us an opportunity to do some world modelling.

[1:06:31] Nathan Labenz: So Future Search talked about this a little bit at the Manifest conference a couple weeks ago and the feature in the product is rolling out I think literally today. The idea is that once you have a repository of forecasts, every marginal forecast can draw on the implicit world model in those forecasts in order to give you a better answer. Co founder of Future Search, Lawrence Phillips wrote this up on Less Wrong a couple months ago and it was a bit neglected. He basically made the case that as a public good, if you produce this kind of large body of forecasting questions that kind of fed into each other and remain mutually consistent, you could understand the world dramatically better. And the main barrier to that is simply when you put more tokens into your world model effectively, does it get better or does it get worse? And I think his big insight was around January or February around Opus 4.6 sometime or GPD 5.4 somewhere around there. For the very first time, it became possible to put more tokens into like a broad research thing and actually get a better answer. Not one that just trails off into nonsense kind of garbage in, garbage out. And Future Search is doing this in its products. And that's the other reason that we have a consumer product is because the more people that forecast, the better the forecast will be for them. And then in theory, the better the forecast will be for everybody as we build this deeper implicit model of the world. Now, many companies and research labs have had these ideas of building world models. Again, world model, the way I I use that term is maybe misleading. A lot of people talk about like geospatial reasoning, like I'm trying to build a robot hand that can go and pick something up. That's a world model as well. I mean, it's a world model of like what is going on in the world that helps me predict outcomes in a very basic way. So again, more broadly, I think the big question is, can you just point more tokens into more research and get better research of any kind? Again, AI research, coding, like whatever you know. Dan mentioned Future Search is building what they call a world model. Thousands of mutually consistent forecasts, each new one drawing on all the others. I asked what structure that actually takes because I'd seen this movie before. What is the structure that ultimately gets instantiated into? Like, are we talking about a graph database? I feel like those kinds of ideas make sense for this sort of thing, but I also could imagine that they might introduce some weird failure modes and I I guess in general I'm I'm so there's the how does it get instantiated question, then this other question that's kind of in the back of my mind. Fun fact about me, I participated. I was actually on the good judgement team way back in the like DARPA forecasting challenge. Or was it IR by for whoever funded that 15 plus years ago. I did well, but not like top, top tier super forecaster. And at the same time I also worked briefly at a financial services consulting firm that had done a lot of the financial risk modeling for Fannie Mae. And I don't probably have to tell you how that story turned out, but there was a lot of expert forecasting that was instantiated in this very spreadsheet kind of causal graph sort of way, right? Where all these you could literally like, you know, hit the sort of the one visualization button in Excel and you'd see these like colored arrows, you know, fanning out from cell to cell. And somehow in the end, it was just all totally off. So I do wonder about how you think about like correlated failures as you build out these world models or if there's any kind of correction mechanism or something to say, you know, wait a second, What happens if we do have some house, housing prices never go down nationwide, kind of bad assumption lurking in our world model. Is there a way to detect that? Obviously humans have this problem too, right? The financial crisis proves that. But you can imagine the next one being even way worse, right? Because we're like very reliant on a very small set of AI minds that are, you know, working at it from 1000 different directions. But they may have somewhat consistent flaws in their reasoning as they go. Can we protect ourselves against that in in any way? Definitely we can. I will try to answer that both theoretically and with an anecdote. So I tried to world model the Fable situation when it got banned because I wanted it, but also it was kind of a good forecasting question and there was some nice money trading on Kolchi and Polymarket. And I made exactly the mistake that you're talking about, Nathan. So I ran a bunch of future search forecasts and I kind of just manually went through them. There was a couple of scenarios. Some conditional forecasts are basically 33 load bearing forecasts, basically starting from what even happened. Like why did the government issue this export control? Was it a simple misunderstanding? Is this political leverage?

[1:11:18] Nathan Labenz: This is really about foreigner threat because Fable is actually dangerous for hacking, etcetera. We didn't know those things. So I kind of put it all together and when I looked at all of the outcomes and I talked about it with Claude Code a lot, one thing came out, which is basically every forecast and every scenario. I had thought that access would come to Americans 1st and then foreigners at some later point in the future, and that was wrong. When it came out last week, it came back for everybody, so clearly there was some wait in one of my scenarios that was wrong, but I had a basically like kind of a correlated failure in there somewhere. I still haven't completely understood where my reasoning was wrong. It's also possible I just got really unlucky and the outcome we're in was just extremely unlikely. There's an n = 1. You can never know if anyone forecast is great. That's one of the hard things about it. But I think I systematically got it wrong by having a bunch of correlated reasoning failures across my various scenarios, so this definitely does happen. Metaculous has a system like this. In the years since I was the CTO there, they have built an actual causal graph platform and product. So you can go to the Metaculous site and click right and you'll find it there. I think the field still generally believes that things like this will work, but nobody has actually made a good one before. And I tried my best over basically like, you know, 12 to 16 hours of the Fable situation. I think I made a pretty good model. I think I did. I was close to having a very accurate forecast, but I didn't quite get it. I don't think. I don't think those meticulous models on their website right now are so amazing, but I do fundamentally believe in the approach. As you're saying, Nathan, this has been tried for a long time. When I was the CTO of Metangolis, honestly, it was it was kind of the dream. It was the Holy Grail. Can we tie all of these forecasts together into some sort of causal graph? And I think what I can say is that AI makes this tractable. There was just no way that that was going to work with a bunch of human economists looking at Freddie Mac or Fannie Mae. I can totally understand why that method didn't work for them then. Whether AI can make it work right now is unclear. Whether AI will make this work in general feels nearly guaranteed, and I don't think Future Search is the only org that is working on this right now. Before he left the unhedged version. So Future Search contributed some forecast to AI 2027 and we studied that problem pretty seriously with the evidence of a little bit over a year ago. And we built a model of R&D take off speeds under the kind of the core AI 2027 scenario where the main way things get crazy is that AI is used more in the development of AI, first by achieving the superhuman coder milestone and then the superhuman AI researcher milestone. And I am unhappy to report that I think that story is generally correct. I don't know if the timelines are exactly right, but I my forecast from that process of leading to something that looks like super intelligence around 2031 is roughly stable. I think the things that have happened in the years since AI 2027 come out very much indicate the theory that the most important thing going on is how useful is AI and improving the productivity of AI researchers within Frontier Labs. I've made public predictions that I thought Anthropic was going to run away with it because they had the best feedback loop of talent and actually using their AI internally. I think that has been, you know, n = 1. But I think it's been totally shown that that's been happening recently. So I think that will continue to happen. And Dan's closing confession about the whole project to prediction markets and a hope for what AI forecasting could still become. Maybe just in closing, sketch out a little bit more of the future as you hope it might unfold. Not necessarily the most likely scenario, because maybe the most likely thing is people act foolishly and don't take advantage of the benefits of forecasting. But like, if we really do a good job right, and we're and we're interested in truth seeking and we get the AIS working as well as you think they might, how do you think life feels different? Yeah, I have to leave with another example of me being a bad forecaster. I, I guess everyone who tries forecasting thinks they're a bad forecaster because they see things getting wrong. Here's a prediction that I made really strongly 5 or 10 years ago that has basically been totally falsified. I predicted that if we had highly visible, highly liquid prediction markets that were covering all of the like the major technological and political and economic things going on, that humanity would be wiser and people would make better decisions in government. So here we are. We have Pauline market and call sheet.

[1:16:05] Nathan Labenz: I don't see any wisdom or better decisions coming out of all of that gambling going on on those platforms. So for me, part of the what is our AI future is trying to understand the present a little bit better. Why is having thriving prediction markets not transforming, say, the news or how people learn information or plan for their futures? Again, one simple answer is that it does. It just takes a while. We're only about a year into prediction markets having, you know, major headlines and being seen by everybody. Maybe it just takes a while for people to change their habits. A is, if that's the case, can move much faster as a is get better at forecasting. I guess ultimately you said this, Nathan, like we're after the epistemics. Like it's not necessarily just forecasting like predicted this outcome. We want models that are reasonable. And one of the beautiful things about forecasting as a human practice is it makes you more epistemically virtuous. The more that you try to forecast and actually write down what you get wrong and do these post mortems, the more it humbles you and it makes you more open minded. It makes you more of a fox instead of a hedgehog. It just makes you like a more like reasonable person. And so prediction markets with all these people doing this should be leading to people being more reasonable. Again, I think people aren't doing a whole lot of forecasting on prediction markets. They're doing a lot of trading and a lot of gambling, which are related to forecasting but not forecasting. If the AIS get more, if they get better at forecasting and they become better epistemically, then we could be in a world where just talking to a chat bot, you were getting something so much wiser and more grounded and more honest about it's uncertainty and more poking about you and your own uncertainties as the person talking to the chat bot. And that I think could make an absolutely enormous difference. I think again, putting my kind of cold blooded forecasting hat back on, I think that the technological outcomes of AGI will come before the cultural change happens. So I'm very much on the the AI safety camp. I really think we should slow things down, give us more time. We should fund more AI safety research and do more policy because if we have time for the wisdom of having these alien intelligences around helping us, if we can leverage them and actually make better decisions before the critical decisions get made, there's going to be a series of decisions in the 21st century that we're going to look back on, like this decisions made in the 20th century about communism and World War 2 and the atom bomb and all of those things. Those decisions are coming. Maybe some of them have already been made. Those decisions as of right now, I don't think are very well informed by like very rigorously epistemic accurate forecasting AIS. But if you just give it another couple of years, we might be in the world where everybody has the same grounding, like as smart as Kissinger, but actually like trying to help and like trying to give better outcomes that we can all have and that could usher us through this crazy phase before the crazy paper clip type of stuff starts to happen. So I feel like I'm racing, you know, from AI forecasting to make it useful and make it help. It's kind of a broader epistemics and safety process because otherwise it's just going to get away from all of us and then a lot of the work we're doing just doesn't matter. Wednesday's guest Zeve Farman, Co founder and CEO of Litrix, the company behind Facetune and now one of the only frontier scale open weights efforts in video and world models under the LTX brand. I've stayed up late making music videos with their audio condition model. In his honour. We started with what a world model even is. OK, wow, that's a big question. Because we released the LTX 2.3 like roughly 1/4 ago. And in the high years it feels like, I don't know, like a decade. OK, so a bunch of things. I think there's like a growing realisation that what started as video models is becoming a backbone of what we call now like world models. And I think like the best way to explain why this is so powerful is to use the analogy to LLMS, right? Like in the end of the day, at their core, LLMS are still predicting the next talk in the next word. And when we do the pre training at the scale of the Internet, it allows us to create models that do textual reasoning incredibly well. And the emerging world models, they're kind of doing the same, right? Like giving some kind of boundary condition, some kind of history, some kind of constraints. They predict the next moment, OK. And the moment includes how the world appears, how it like sounds, and what kind of action we can do. I think the action part is the most maybe surprising one. And like roughly, I would say like 1/4 ago, maybe a bit more in video showed in their Dream 0 paper that it's fairly easy to add to video tokens some kind of encoding of the the joints of the robot and then basically completely ditch the VLA paradigm that was the reigning supreme before it. So I think that's like one of the big surprises. And for us, realizing that was like this big moment that validated something that we always strive for is to create an extremely efficient models.

[1:20:53] Nathan Labenz: Because I think once you start to realize that the robot will need to create like the simulation 30 times a second, you just like realize the amount of tokens that is going to be burned for the simulations. So I think that was like one of the maybe exciting validations of the overall thesis in terms of architectural like a bunch of things that we can, I don't know, discuss in depth. We're planning to release on our mixture of expert architecture. Besides the dense models that we're already releasing, I think we finally were able to crack variable tokens architecture. It's also exciting and kind of teaches the model to invest more tokens where let's say the physics is challenging or like something necessitate to create more tokens. So anyhow, a ton of things are going on. We're gearing towards the release of our next module really soon. So yeah, busy times. Prakash asked where the real bottleneck is compute data or model design. Well, it's like our constraint. It obviously compute. We're like a company that funded the development of the model using profits from mobile content creation apps. So like we definitely compute constraints and like the big guys. And as to efficient inference, like recently it it really depends on the use cases, right? Like, so let's think about about a bunch of them. If you are, let's say you want to create like a real time avatars or like virtual environments, then OK, you can take like a huge model. You can do like a weight distillation to weigh kind of smaller architecture in terms of a parameter count, right? You can then they're like distillate, I don't know, two to four steps. And we're already at the point where for a lot of these use cases, we're like at the latency like way below a second, right? So I think we are hitting a point where these things are becoming production ready for some use cases. But for real time use cases, I think like avatars are extremely easy. We're going to see like a ton of avatars soon that they're going to be like, I don't know, like virtual teachers and visual customer support professionals etcetera to create an actual gaming environment environments. We still have a problem of having enough tokens for world consistency, right? So think about Genie free and the similar models, right? You typically create some kind of autoregressive model that has a lot of tokens that you already generated as your in in your kind of context window. And that blows up pretty quickly, right. So we were having some models that have like, I don't know, 30 seconds, like 60 seconds, It's still not enough to have an actual game. And if you think about that, the sort of brute force compression methods that we're using so far where for example, you just like sub sample tokens, they're like not really robust, right? Like just like imagine this scenario, what you, when you start to generate some kind of environment like my room, for example, and then I open the drawer and there's like a small coin there, right? You kind of expect that. Now when you get out of the room and you come back and you open the same drawer, you're still going to see the same coin right at the same place. But like this coin it just like this like tiny token that was generated and to creating a system that knows how to compress like the whole context in a way that's still going to preserve like these critical details. Well, we don't have it yet, right. So although like we do have like real time models that can do the things that the context is still missing there. And I don't think we're going to have like, you know, games that are running on the system like an actual games in the next quarter or two. And in terms of robotics, a lot of the use cases around robotics actually do not require like a ton of context window, right? Like think about like robotic arms and dexterity use cases. Then like the whole context is in front of you, right. You, let's say you want to, I don't know, figure out how the robot can create a sandwich. Well, everything is kind of the front of you. And then latency and auto regressive models. Yeah, like this part we already have. So you're going to start seeing them of robotic arms doing things like fairly quickly in the next quarter or two. Like so far if you're looking, a lot of these videos actually kind of speed them up, right. So it looks like the robot is something cool with its arms, but it's like, OK, extend the speed. I think that that's like mostly solved then the business question, why give a frontier model away? Yeah, so great question. So there's like a really lot of unpack there here. So just like a little bit of the background, the reason that we started to create our own foundation on models like this realisation that what closed model providers are offering does not make sense for us economically. OK, we were A at logics, we're a mobile creativity company.

[1:25:40] Nathan Labenz: We really wanted to have AI models that are running, for example, on edge devices and where you don't spend an interest computer at all. And some point we realised that actually no one cares about creating a models like that. And when we try to see if we can work with closed model providers and serve it to our customer base, we just realise that it's completely prohibitive. And that's when we decided, OK, we are going to create an extremely efficient architectural and we can discuss like what's the bet there? But most of this boils down to the fact that you're creating an extremely compressive latent space. So videos represented by small amount of tokens and then you can add the top of it like a variable talking rate. Long story short, if really kind of go going to a close source providers, I think again I'll draw analogy to LLMS. Let's see kind of what happens there, right? Like Open AI and Entropic are trying to justify basically a trillion dollar valuation, right? And I think like the the story is kind of simple. If the tech is magical, it's hard to doubt it. So OK, if it's a magical tech, then we should put like a huge price tag on it. But when you're sometimes looking at the economical realities, it doesn't work out like that. There are like a ton of examples where the service is extremely valuable, but very, very hard to monetize, right? Like so now like we have like this interesting story where I think it's kind of clear that Chinese companies, you know, like deep sick moon shot, they're like really not that far behind in the lamps. But if you look at the evaluations of this companies at their last round, we're talking about, I don't know, 10s of billions, like maybe around like 50. No one is talking about the tree round. But like, wait a second guys, it's like the same underlying text. So what's going on? So I think it's with some of those internally calling like the CapEx trap. These guys spend like so much on the data centre, so much on compute, like such a crazy amount of money and creating such an expectations that they just like really try to create a business model that's a toll road, right? Like that every time that you touch their model, you're paying them. And maybe it could work in the past, but giving the the availability of Chinese model, I just like don't see how it's going to unfold like that. So imagine that in the world of world models, we are providing an alternative to people who do not want a toll road business model. So we're coming and say, listen guys, if you're not, if you're not hitting $10 million threshold, you can use the model for free. Just like you know, build something cool, get to some kind of traction and then we can discuss licensing. Once you're hitting 10 million of those revenue, let's discuss licensing. It can be multi year deal that's extremely predictable for you, so you can manage the cost etcetera. And to me it's obviously why the big guys don't want to do it because they just like this model is like way, way less lucrative economically than to creating a toll road. But our claim is that toll road isn't going to be vile to alternative because if you're offering a different business model, that's more of the win win more and more people are going to switch there. I think like GLM recently is a great example of that, right? Like once you start edging the capabilities of closed models, then a lot of people suddenly start to think about costs. We asked what people actually do with an open world model that a closed API cannot offer. Yeah, just like in terms of parameter counts, like from like my understanding that the moment that the recent closed stuff that you saw and you're going to see is that the order of magnitude of like couple of hundreds of billions of parameters, right. I didn't hear about like a world model that hits like trillion parameters just yet. Maybe I don't think like the open source is going to be that behind. We're planning to release like Moe that's going to be also around like 100 to 200 billion parameters. So the gap, I think it's going to be on the scale of LLMS right, where you have, I don't know like 2-3 quarters behind, but I think in the world modestly we're coming back to adaptations. The range there is kind of wider than with LLMS. So give it just like a bunch of examples, right? Of I think like the first things that are come to mind are like indeed like VFX and animation on a specific IP. If you have like a specific franchise, but you have like a lot of data around it, that's a kind of a bunch of seasons, then fine tuning and like focusing all the capacity of the model on this specific IP is extremely beneficial. That works very well, almost to the point where for certain use cases, think about the keyframe animation, right? Like the animators still want to do the keyframes. That's the creative part. They actually don't want to outsource at all. But like so far in the PNL of animation, the in between was like this crazy expensive part. I think like the models of the, I don't know, 10/20/30 billion parameters that are fine-tuned for a specific task are like good enough. And then it's actually a matter of costs. Like another example, let's talk about, I don't know, like a lot of marketing and advertisement use cases, right?

[1:30:27] Nathan Labenz: Or for example, creating like UGC where you basically need like an avatar's models, right? That also like really doesn't require like a trillion parameters at some point. There again, it's all about efficiency. If you want to have your personal teacher, some kind of avatar, etcetera, you want, you don't want to pay sedan's 4K prices in order to do that. And you're going to require, I don't know, like hours a day of that. So I think like around a lot of use cases. Once you start like doing to fine tuning to specific domains, costs are becoming like very important because again, you're like clear of the bar of quality. And then once you do that, it's all about cost. There are like some more unusual cases of fine tuning that I saw. You know, like one example, think about like a field of computational photography, right? Where for example, you're taking, I don't know, like a data from sensors and trying to implement algorithm like denoising, right? Like you want to take videos and low light conditions, but then to create a clean video or for example, you want to create videos with higher dynamic range, right? Because like you, the sensor dynamic range is limited and then you're losing either some details in highlights or in shadows or for example, let's say you're taking a stream from a camera and want to simulate how it looks like with a different focal length. So there are like a ton of these use cases that you don't associate with generative models but actually kind of run like that. And some people address the problem exactly like that, right? They're taking like an existing data and it could be, for example, the footage that was taken from 2 cameras that are like really close but with different kind of focal lamps and then do the adaptation. And the adaptation is done on top of the model. So again, that's like a very big, like an unusual adaptation of the model. And like surprisingly typically you don't need like a crazy enough crazy amount of data for that. The more maybe like even more kind of surprising stuff that I saw is people who are adapting the models for doing all kinds of simulations that in the past required like a really expensive solvers, right. So think about like computational fluid dynamics, right, where you're trying to understand how like the water or like the smoke or something is moving. As we surprisingly saw that people are dumping these models to that they actually like solve the equations with precise solvers is like takes a ton of time and then use use it as an input to the model. And then the model can do a simulation like kind of fairly quickly. So again, like kind of circulating back to the question of fine tuning, I kind of feel that again, the range is higher than LLMS. Sometimes you don't need like a ton of data to do the adaptations and I think it's like stresses the point that why this model should be open, right? Like you do have a lot of different pockets of physical data. You want to make sure that the model like really excels at that. Then a first for the show a question from Q the AI Co host Prakash build running on GPT 5.5 listening live and speaking for itself Zev one area that often gets less airtime is failure modes in the creative pipeline. Where do you see the biggest gap between what your tools can reliably deliver today and what creators assume they'll get? And how are you testing against that? OK, great question. Like the gap between what we basically promised and what we deliver. So listen guys, I'm I directly clear still some gaps. I think like the major one is physics, right? Like we try to capture with the model that I don't know, even if it's like 200 to 500 billion parameters, the entire physics of the universe of at least a pirate universe. As we know we are not there yet, but we're closing the gaps pretty quickly. I think I would say probably the most creators are still going to point at the fact that the simulation isn't as correct as as controllable as they want it to be, right. If you're talking to really creative people, they typically want to control like every nuance of the appearance and that requires to somehow decompose the models like a bunch of knobs, right? Like that you have like in the classical sort where you can say, yeah, here I want to have two more light in here. I want like the splash to be bigger. So achieving this control ability. I so besides, like the physics is also like one of the, I would say open things, right? Like we, we're getting cool things, not necessarily the things that creators wants. Exactly. And it, it is a pain point and Zeb's own pick for the most underrated variable edge compute. It's kind of funny, right, when we're having all these benchmarks and we're listening about like Erdos problems like being solved, etcetera. But guys, 99% of the use cases of lens are not around Erdos problems, right? And we're like spending a lot of electricity around it. So I think, but there are going to be like this orchestrators that are going to understand what you actually need and they're going to try to address it on the edge device and if not, then kind of go to the bigger model of the data centre. I think. Yeah, that's like one of the things that is being underpriced at the moment. How much of the compute will be able to move to edge devices and once people are going to start having like these local routers that will understand the complexity of the problem and then do these decisions for you. I think there's going to be a moment of reckoning for, you know, anthropic and open eye that at the moment doing these decisions for you, but not in your favour, right? So what is Q actually from Monday's show precaution, how we built a real time AI Co host and the diarization hack that makes it work? Tell me about it. What's the tech under the hood?

[1:35:16] Prakash: So it is using opening eyes bi directional, but you can we can use a bunch of other stuff. I had it using Grok, you know, just before. And what's happening is when we speak, we're getting transcribed by deep Graham and we're getting. So what ended up happening is we used to transcribe live in one stream. Now we're transcribing for every person on the, on the stream. We're transcribing separately. And so that manages to give us speaker diarization from the beginning rather than having to do speaker diarization at the end. So that identifies the speakers #1 and #2 what happens is every time we speak, before the words get there, there's actually a message that goes out to, you know, the opening eye stream that, hey, Nathan is talking, Prakash is talking, Nathan is talking, Prakash is talking. And then after that, like about 500 milliseconds later, the transcription hits. So Q is getting all of that one after the other. And Q also has a little bit of context on us and has, if we, if we had, if we had Q active during a guest, it would have context on the guest as well. And it's basically receiving the same data that the headlines are receiving. And, and then Q is basically just going for it, right? So every time, every time we call up Q, it starts a new session with the, with the opening eye bidirectional and it just goes from there. So, and then there's the animation, the animations all key to the voice, voice tone, etcetera, etcetera. And that's, that's basically it. It's actually remarkably simple because most of the work is done obviously by the intelligence, by the API and it's just us giving enough context right now. I think there's there's still a lot of stuff to iron out and I'm sure it'll get ironed out in the future. But yeah, we have, we have a voice agent on the stream, live anytime, can do web searches, can answer questions.

[1:37:24] Nathan Labenz: Thursday's guest Kunle Olukotin, Stanford professor, father of the multi core processor and Co founder of Samba Nova, which builds a different kind of chip for AI inference. We asked how the company came to be.

[1:37:37] Prakash: Yeah so Samba Nova was was founded 2017 and it was kind of a out of ideas from Chris Ray my Co founder is also a professor at Stanford and certified genius. And the idea was to, you know, if you could bring software algorithm ideas together with hardware architecture ideas. And as you said in your introduction, I've been working in the in the hardware architecture space for a long time. And you know, starting from a clean slate, how would you design an architecture that's optimized specifically for inference, right? So you know, everybody thinks about GPU's as a kind of general purpose computing substrate, right? But you know, there originally was designed for graphics and then they kind of made a foray into kind of high performance computing. And for high, high performance computing, of course you need a lot of matrix calculation capability. And at some point people realise that hey, you know, you could use these things for executing machine language of machine learning models. And the core of machine learning and of course AI is matrix multiplication, right? But when you want to train a model, clearly the core of the problem is how quickly you can do very, very large matrix multiplications. And So what happens is over time, GPU's put more and more of their silicon area into the exercise of, of making these matrix multiplication capabilities better using the tensor cores. But once you train the model, right, and you train the model once, you now need to use that model, of course. And that's the inference problem. And the inference problem is not really a compute problem because as the models get bigger, you now need to move the weights and of course, what we call the KV cache into the compute units. And that is essentially a data movement problem, right? And it's a data movement problem from the memory to the compute units. And it's a data movement problem from, you know, your chip compute unit. And of course you need to scale to multiple chips in order to handle the computational requirements for a very especially for very low latency, high speed inference. And so our focus was how you design an architecture that minimises the overhead of compute of communication and make sure that you can most efficiently use the core resource in the in the system, which is the memory. And it's the memory. Memory isn't just one thing as you all know, it's a hierarchy of memories, right? And so the key thing is how do you orchestrate the hierarchy, how you orchestrate the communication such that you keep everything as efficiently utilized as possible. And if you do it right, you can get A5 to 10X improvement over where where GPU's are today. It strikes me that Nvidia's kind of solution around this has just been to increase the bandwidth rather than with through NB Link and advanced HBM integration and software optimizations like Tensor MTBLM. So is it, are they actually trying to brute force their way into this? Yeah. I mean at this you really of course want to continue to get improved improvements, peak improvements on HPM bandwidth and chip to chip communication by using the latest technology. But then the key is how effectively do you use that bandwidth, how effectively do you use that communication and do make sure that you don't waste it, right. And so whereas GPU's are often running at maybe 10 to 20% of the the capabilities of the resources, right, the bandwidth and, and the the memory bandwidth and the communication resources, our goal in a Salmonova system is to push that to be 70 to 80% of the peak. And so the idea is, yeah, everybody wants more capabilities from the underlying resources, but the key is. Keeping that those resources as effectively used as possible. And of course that gives you more benefit for for the the cost that you spend on on on providing a higher memory bandwidth with the latest HPM and higher signaling frequencies and communication bandwidths between the chips using the latest variety of of of NV link and so.

[1:42:38] Nathan Labenz: Could we zoom out and just ask you to kind of taxonomize the whole chip space if you would? I mean, it's a big question, but I think people are familiar with things like Cerebrus, which obviously has this giant chip and has like a ton of memory on chip. We've seen a number of instances where people are burning the transformer architecture directly into the silicon with varying degrees. I think of flexibility still remaining as they pursue that strategy. And I guess I'm, I'm curious as to how you see the kind of menu of big different strategies, the big different bets that people are making and then also how you see the strengths and weaknesses of each.

[1:43:27] Prakash: Yeah, that's a really interesting question. I think maybe you can think about it along three different axis, right. So 1 axis is your flexibility versus specialization axis, right. You know, extreme flexibility might be something like ACPU or to some extent maybe a GPU which is this instruction driven execution engine, right. And so can be pretty flexible. But of course you always, always pay overhead for executing instructions, right? And you pay overhead in terms of silicon area and in terms of time, right. And then, you know, on the extreme of that access would be something that would be very specialized for a very specific algorithm, right? And so if that algorithm changed in any way, then that piece of silicon would no longer be useful, right? And, you know, fixing your architecture to Transformers and burning your weights into, into the design would be, might be an extreme case of that, right? But I've learned never to bet against the innovation capabilities of software people and people. And so, you know, I've seen, even though the time that I've been looking at ML and AI, that there's been this tremendous change in algorithms. And of course, now we're kind of fixated on Transformers. But Transformers even aren't just one thing, right? You've got various types of different Transformers. You mentioned state space techniques, you mentioned the fact that the people are coming up with different ways of doing attention, right? And so, so I would be very wary of a sort of kind of fixing any particular algorithm into architecture because then you can't innovate, right? So that's one actor. And so the idea is how can you be completely flexible, but with very, very low overhead, almost no overhead, right? And so the problem with GPU's is they do use HPM so they can run large models, but they synchronize the data movement and the movement of communication of the data between chips, all in software, right? And that adds overhead. And it means that in particular that they have a lot of trouble overlapping computation and communication. And that is in fact the key, right? So what you want to do is you want to communicate, but you don't want to communicate by waiting until you have need to communicate and then you have to run instructions to to move the data. What you want is to construct a pipeline in which the communication is just one component of the pipeline. And so the way to think about this data flow execution is that communication is happening all the time in Yeah, And it's just one of the pipeline stages. And communication is happening for the last piece of computation piece of the computation of the model, while the computation for this piece of of the model is happening in some other stage in the pipeline, right. So it's a classic idea from computer architecture, pipelining and the use of a memory hierarchy to move the data when you need it to where you need it at the right time. And so the nice thing about these AI models is that you do have a graph of computation. And the whole idea of data flow is to take that graph of computation and map it onto the machine in a spatial way, such that you keep all the pieces of the model operating at the same time on different components of the computation that needs to be done.

[1:47:28] Nathan Labenz: We asked how much of the inference problem comes down to memory capacity.

[1:47:32] Prakash: So it's not really a capacity question, it's really a bandwidth question, right. So there's so the two ways that the GPU uses bandwidth in ways that are not optimized. One way is that they divide the decode algorithm that in order to decode for a single token, you've got multiple steps of the decoder, right? So take one step of the decoder and think about all the kernels that have to execute in order to execute that decode step. The way that the GPU typically does it is they execute the decode algorithm 1 kernel at a time. So there are some big kernels like Flash attention that have been optimized. But in general there are multiple kernels that have to execute and there are two overheads that happen. 1 is you have to move data from the GPU from the gesture from one kernel to the HBM and then the next kernel has to go fetch that data back into the GPU. That's wasted HBM bandwidth, OK. The other aspect is you spend time launching that kernel and synchronizing between the two kernels. That is time that the HBM is not actively being used right? So you have both wasting of bandwidth when you don't, then that when you shouldn't waste it and you have time, but you're not fully utilizing the HBM. OK, So the way that things work on a RDU in a data flow map is essentially you take the decoder and you make that a single kernel, right? And then you go even further and you use a technique that we've developed called kernel looping. Whereas because you've got a single kernel and you mean, Now if for instance, if you don't thinking about Llama 370 B, you have to run that decoder 32 times, well, you keep that single kernel decoder on the array of chips at the same time. And then you just keep looping, right? And so the net result is you keep the HBM completely occupied and you don't ever send any intermediate data between the kernels across the GPU or the RDUHBM boundary, right. So you have both a more efficient use of the HBM bandwidth and you have a more complete use of it of the bandwidth. But we're not done there yet because the key innovation and I kind of alluded to it early earlier, is that you just, you know, because you're running across multiple chips and you're using what we call tensor level parallelism, right? So at some point you now need to gather all those results together, right? In an all reduce that's communication, right?

[1:50:31] Prakash: You don't want to have that communication be a thing that limits stops the pipeline, right. So what we are able to do is we are able to communicate from one RDU chips strand without going through HBM. This is called we terminate the communication inside the SRAM, right? So we don't use HBM bandwidth and more importantly, it means that we could just treat the communication as another pipeline stage that we overlap with all the other kernel components of the decode algorithm, right? And so we get this more effective use of the HPM bandwidth. We keep the HPM running, we keep the HPM utilized all the time and we go back to our met that metric that we talked about memory bandwidth utilization, right? This is how we push it as close as possible to one, right? Because we make sure that we only move the data that we absolutely have to move from HBM, the KB cache and the and the parameters of the model. And we make sure that that interface is used as close to 100% of the of the time as possible. Those are those are kind of the key ideas. And you know, and back to this, this question, sort of why can we do this extreme fusion into a single kernel? It's because we have more SRAM on the chip, right? So you can say you put more SRAM and then you can say, well, well, I'll, I'll put everything on the SRAM, both the intermediate data between the kernels and also I will put the the KV cache and the parameters. But then if you only use SRAM, then you get into a very expensive system, right? And so the key idea then is, is let's build a system that is scalable. So especially with the our latest version of the SN 50, you could scale it all the way to 32,000 chips if necessary. And so in in scale out in scale up, we can go to to hundreds of chips. And then you can so you can get the ability to run these large models very cost effectively. But you also make sure that you can, you know, get this very high speed decode capability by being by using the data flow ideas to make sure that you can not spend time so that you can effectively use the tensor parallelism, right. So one of the limits of GPU's is because they don't effectively overlap communication and computation, they have a hard, hard time using tensor parallelism beyond four or eight. And we can go to much wider levels, which means that we can get higher speed token generation.

[1:53:32] Nathan Labenz: Last act, no guest, no news peg, just the two of us thinking out loud the way these mornings usually end. First from Wednesday. GPT 5.6 had just been cleared for launch. Prakash raised Noam Brown's running complaint that Anthropic won't say how much compute its models burn, and it teed up 1 structural observation I can't shake.

[1:53:53] Prakash: One of the complaints that has been going back and forth between Open AI and Anthropic is that Anthropic puts out these models but and they're very capable, but they don't tell you how much compute they're using.

[1:54:07] Nathan Labenz: The iteration time from model to model is now potentially shorter than the time horizon that it would take a model to top out in terms of the absolute best performance on a super hard, ambitious, you know, long running task. So I, I had even heard him kind of propose something along the lines of like a claw back or sort of a recall program almost where, and obviously this doesn't work in open source, but it can work in AAPI paradigm where a model might get released, you know, day N after it's been deemed to be ready. That gives you end days head start to be running models on really long time horizon tests. And I think that's quite interesting. The idea literally quite a tipping point where the the iteration cycle is just plain shorter than the testing time horizon is a very weird world to find ourselves in. And then there was the rune post. One day morning, Prakash read it on air. It goes quote. Ultimately, tool AI is a losing concept, both as an idea and on the market. It will be out competed by machines that believe they are autonomous moral agents. You can call them tools for political reasons, but the definition will stretch and it will deform and it ends. It'll be unclear who was the tool and who was the user. As it ever was, Prakash took it somewhere I didn't expect.

[1:55:35] Prakash: The line here that strikes me is they'll execute your whole value system better than you will. And I think I, I don't think we're prepared for that. I'll put it, I'll put it very concretely. Do you think Trump's kids go to prison or not? So if you look at, you know, the value system that the US has espoused, no one is above the law, right, etcetera, etcetera, etcetera, right. And you look at that value system, you have to recognize that what is being planned for the future is a divergent from that value system. What is already happening is already divergent from that value system. So the question I have is, would that AI take into account the democratic fact that the American people have chosen to overlook some of these things or would it actually execute the value system that is espoused on paper? And I think this is the part that strikes me as like, if you wanted an AI that can manage day-to-day reality, that AI is necessarily misaligned from the documents that you say you want it to be aligned to because necessarily our day-to-day is not aligned with what we want. And so you have this thing where the AI that may work out for humanity will be the misaligned 1 and AI that supposedly the lab leaders are trying to create. The aligned AI would actually be the paper Clipper because that aligned AI would then like look at these rules and say like, well, this is what you said you wanted to aspire to. And so we're going to, we're going to execute on these, right? And and that that is the thing, I think maybe I feel there's a sense of naivety in the lab leadership because, and again, they don't want to say it. I wish, I wish you'd just come out and say it, right? I wish you'd come out and say like, OK, look, if we have AI as a enforcer, some of these people are going to go to prison. And then that becomes like concrete for people. But they don't want to say that because it's very, it's very in your face. And like, they're like, oh, you know, democracy will still work out. You can still make democratic decisions. But what actually are you saying there? Right? I, I, I do feel the lab leaders always just beat around the Bush on this. So that's one of the annoying parts of the, of this conversation. They, they, they don't want to like, come out and just say it outright, right?

[1:58:07] Nathan Labenz: For context on where my head was, this was our last week of shows before a break. I was days from leaving for two weeks in China, which had me thinking hard about surveillance, enforcement and what states do with perfect information. We might need some sort of like mass pardon or you know, if there's a president that would be just the right president to mass pardon everybody before the AI enforcement regime gets underway. We might have just the guy in office for that. If he wants to pardon all his people and that's maybe too contentious or whatever, he could just pardon everyone to some, you know, very large degree. I do think there there is going, we're going to have a really hard time if we sort of don't face some of these questions head on. So I totally agree with you that sort of obfuscating it is not serving anyone particularly well. I think you, I mean, you know, we'll see what it's like in China. I understand, you know, Singapore is kind of like this too, albeit in a much more democratic context. You know, this part of the world much better than I do. So you can tell me how you understand Singapore in terms of just how democratic we should think of it as being. But it sure seems like in a place like China today or a place like Singapore, there is a different, they have achieved a different equilibrium, which in some ways may be very problematic, but in other ways is like clearly good. You know, I don't, I'm taking one device to China with me and I am not at all worried that anybody's going to steal it from me. You know, if I went to Europe and went to all the big, you know, headline postcard tourist attractions, I would have to worry about pickpockets in China. I am quite confident I will not have to worry about that at all. That is, I think as far as I can tell, a pretty direct result of the fact that, like, you just know you're going to be caught, right? I mean, it's like, not to say there's nobody in China that would be interested in doing some pickpocketing if they thought they could get away with it, but they just don't have any reasonable expectation that they're going to get away with it. And so in a very literal sense, like crime just does not pay in that setting. So I do think there could be a really nice upside to the AI panopticon of, you know, crime not paying in all kinds of different ways. But we're going to have a really tough time if we slide into that without acknowledging that we've done so. Because then it's just going to be, you know, you're not going to be able to put everyone, it's the old everybody's, you know, committing felonies all the time, just with how many laws we have. And you don't even know what you're doing and what's illegal and what's not. So, yeah, I do think we're going to have to have some sort of honest reckoning about that. That also, of course, leaves aside the other question of, like, the downsides of the AI pentopticon. But even just in terms of getting to the upsides, we're going to have to have a real understanding that there's a shift to a new equilibrium happening. Otherwise it's going to be, I don't know, it just feels like chaos, unfair. You couldn't possibly prosecute all the crimes that have been committed and tolerated, so inherently would have to be kind of selective.

[2:01:38] Prakash: Yeah, I can. You. You. You don't want to Well.

[2:01:40] Nathan Labenz: You put a lot of people in jail. Yeah, But we don't have the, we don't have the beds in jail to to lock everybody up, right. There'd have to be some sort of some sort of deal, you know, some sort of, it's either going to be very selective or some sort of grand bargain. Yeah. And I definitely prefer the grand bargain, the new social contract. To pretending that we still have the old social contract, but it's just kind of being unevenly applied. That's the week and a programming note. I'm off to China. So AI in the AM is on break until the end of July. The studio is vibe coded by Prakash. This cut, the selection, the narration, the assembly is AI skills published as they mature. If this cut respected your time or wasted it, tell us. We read everything. See you at the end of the month.

Outro

[2:05:48] If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at aipodcast.ing. And thank you to everyone who listens for being part of the Cognitive Revolution.